The demonstration was in response to French satirical magazine Charlie Hebdo's use of the cartoon in an edition published a week after 12 people were killed by Islamist gunmen at its Paris offices.
The image has angered many Muslims as depictions of Mohammed are widely considered forbidden in Islam, and has triggered protests in Asia, Africa and the Middle East, some of which turned deadly.
Iran denounced the Paris massacre but it also condemned the magazine's new cartoon, where the prophet holds a "Je suis Charlie" sign under the heading "All is forgiven".
Plans for Monday's protest led the French ambassador to announce that the embassy, located in busy downtown Tehran, would be closed all day.
The gathering, organised by students but attended by all age groups, was given a heavy security detail of around 150 Iranian police, and although noisy it passed off peacefully after two hours.
Signup to our newsletter and follow us on Facebook and Twitter!
One speaker said the demonstration was to "condemn the insult of Charlie Hebdo," but also to denounce that "the embassy forces women to remove their veils to get a visa".
In Iran, women, regardless of their religious denomination, must have their hair and neck covered by a veil under a female dress code in force since the Islamic revolution in 1979.
"The least I can do is protest against this insult, we condemn insults of the prophet," a woman in her 50s wearing a traditional head-to-toe black chador told AFP.
As in customary in Iran, men and women were segregated at the protest.
Those gathered also criticised Iran's Foreign Minister Mohammad Javad Zarif for being in Paris for a meeting with US Secretary of State John Kerry on Friday.
The organisers in a final statement asked Zarif to apologise for making the trip "just two days after this disgraceful cartoon was published," also demanding the French ambassador be expelled.