President Barack Obama and Iran's new leader made very tentative moves to end decades of hostility between their countries but could not break the deadlock enough to organize a meeting.
Obama said an accord was possible if Iran ends concerns over its nuclear program.
New Iranian president Hassan Rowhani called on Obama to ignore "war-mongering pressure groups" to make a deal. He said he was ready for "time-bound and results oriented" talks.
The longstanding gap between the United States and Iran was evident in Obama and Rowhani's speeches to the UN General Assembly and the failure to orchestrate at least a symbolic handshake between them.
Obama said mistrust between the United States and Iran has "deep roots" and that Tehran's "pursuit of nuclear weapons" would remain a US foreign policy priority.
But a "meaningful agreement" between the arch-rivals is possible, he added.
"I don't believe this difficult history can be overcome overnight -- the suspicion runs too deep," said Obama.
"But I do believe that if we can resolve the issue of Iran's nuclear program, that can serve as a major step down a long road towards a different relationship -- one based on mutual interests and mutual respect."
"To succeed, conciliatory words will have to be matched by actions that are transparent and verifiable" on the nuclear program.
"The roadblocks may prove to be too great, but I firmly believe the diplomatic path must be tested," Obama said, adding that he had ordered US Secretary of State John Kerry to pursue diplomatic efforts.
Kerry and Iran's Foreign Minister Mohammad Javad Zarif will meet for the first time at international talks on Iran's nuclear program on Thursday.
Rowhani said Iran poses "absolutely no threat to the world," in his speech and condemned international sanctions against Iran.
If Obama rejects "the short-sighted interest of warmongering pressure groups, we can arrive at a framework to manage our differences," the Iranian leader said, insisting that his country's nuclear drive is "exclusively peaceful."
"Nuclear weapons and other weapons of mass destruction have no place in Iran's security and defense doctrine, and contradict our fundamental religious and ethical convictions," Rowhani said.
He added that the international community had to accept Iran's nuclear activity, which Western nations say hides an attempt to reach a nuclear bomb capacity.
Signup to our newsletter and follow us on Facebook and Twitter!
The UN Security Council has imposed four rounds of sanctions against Iran over its uranium enrichment.
But Rowhani said it is "an illusion, and extremely unrealistic, to presume that the peaceful nature of the nuclear program of Iran could be ensured through impeding the program via illegitimate pressures."
In a later television interview, Rowhani also condemned the holocaust, in another sign of change by the Iranian leadership.
"Any crime that happens in history against humanity, including the crime the Nazis created toward the Jews, is reprehensible and condemnable," Rowhani told CNN. His predecessor, Mahmoud Ahmadinejad, was a fierce critic of Israel who frequently questioned the holocaust.
Israel's Prime Minister Benjamin Netanyahu still blasted Rowhani for making what he described as a "cynical speech that was full of hypocrisy." Netanyahu will speak at the UN next Tuesday.
There had been speculation that Obama and Rowhani could meet at a lunch organized by UN leader Ban Ki-moon. Rowhani was not present however.
A senior US official said that an encounter had proved "too complicated" for the Iranian side.
Rowhani did meet French President Francois Hollande who also said in his UN speech that Iran must take "concrete" measures on its nuclear program to reassure the international community.
The United States ended diplomatic relations with Iran in 1980 in the stormy aftermath of the Islamic Revolution. Rivalry has heightened since then, with the United States leading the sanctions drive.
The speeches overshadowed the growing alarm expressed by world leaders over the intensifying Syrian war and the use of chemical weapons.
Obama insisted the United States remains ready to use military force to protect its "core interests" in the Middle East as he demanded that the UN Security Council pass a "strong" resolution to back a plan to destroy Syria's chemical weapons.
Hollande said a resolution being negotiated by the Security Council must allow for eventual "coercive" measures against President Bashar al-Assad if he does not keep to a disarmament plan.
Russia has resisted Western attempts to make the disarmament plan mandatory for Assad under Chapter VII of the UN Charter.
Kerry and Russian Foreign Minister Sergei Lavrov held talks on a proposed UN resolution, but no signals emerged that a Security Council vote would be held soon.
The United States threatened a military strike against Assad's forces over a August 21 chemical attack near Damascus, which the United States says killed some 1,400 people. This was suspended after Kerry and Lavrov agreed a plan to put Syrian chemical arms under international control.