The grand mufti of Jerusalem Muhammad Ahmad Hussein, the city's highest Islamic authority, had earlier gathered in the Old City with others and condemned the closure of the Al-Aqsa mosque for prayers after the attack.
Israeli police said they had no comment.
Hussein's son Jihad Hussein told AFP his father had been taken to a police station near the Old City.
"Until now, we don't know what is going on with my father," he said.
Signup to our newsletter and follow us on Facebook and Twitter!
One of the mufti's bodyguards, Khaled Hamo, said police "entered the crowd and took the mufti."
The incident came after three Arab Israelis opened fire on Israeli police, killing two before fleeing to the ultra-sensitive holy site where they were also shot dead in one of the most serious incidents in the city in recent years.
Security forces locked down the area and the Al-Aqsa mosque was closed to Friday prayers after the attack in a highly unusual move.
Hussein had earlier condemned the closure of the mosque compound for prayers.
"I have very little information about it, but it doesn't mean you should close the mosque for prayers," he told journalists at the Lions Gate entrance to the Old City, near the holy site.
Hussein was detained at the Lions Gate, the bodyguard said.