They said around 400 entered the site in occupied east Jerusalem throughout the day to commemorate the destruction of two ancient temples there.
Police expelled 10 Jews from the compound for trying to pray there and arrested another two.
Security at the controversial site had been boosted with hundreds of extra officers ahead of the event, which began on Saturday night.
Tens of thousands of Jews also prayed at the adjacent Western Wall during the day.
Advocates for prayer rights for Jews at the esplanade, the holiest site in Judaism and referred to as the Temple Mount, have called on Jews to visit the sensitive area.
It is Islam's third holiest site.
An AFP journalist saw several groups of around 30 Jews visiting the site in the morning and the afternoon, many escorted by police.
Muslims entering through different gates showed their identity cards to the police, who retained them for some visitors until the end of their visit.
Signup to our newsletter and follow us on Facebook and Twitter!
Honenu, an advocacy group that provides legal aid to extremist Jews, said three were held for citing verses from prayer, and another for tearing his shirt, a sign of mourning in Judaism.
Jews are allowed to visit the compound but not pray there, and the site has been the scene of regular incidents when Jews try to ignore the rule and Muslims intervene to stop them.
Police said Muslims had gathered around two Jews being expelled by police and began yelling at them.
Police pushed them away and three Muslims were lightly injured in the scuffle.
Jerusalem has been at the heart of a wave of violence since October in which 219 Palestinians, 34 Israelis, two Americans, an Eritrean and a Sudanese have died, according to an AFP count.
Most of the Palestinians killed were carrying out knife, gun or car-ramming attacks, according to Israeli authorities.
While the number of attacks has declined in recent weeks, Palestinian fears of Israeli intentions to undermine Muslim control of the Al-Aqsa mosque compound were a key factor in the violence erupting 10 months ago.
Palestinians argue that Israel is seeking to change the status quo at the compound, a claim that Prime Minister Benjamin Netanyahu has repeatedly denied.
Israel occupied east Jerusalem in 1967 and later annexed the territory in a move never recognised by the international community.