"Tunisian Jews feel in danger, they are really afraid," Yamina Thabet, the president of the Tunisian Association Supporting Minorities, told a news conference after visiting Djerba.
According to testimonies from witnesses on the island, three "serious incidents" took place against members of the community in the past few weeks, which were marked by important Jewish festivals including Yom Kippur.
Thabet accused the police of forcefully interrupting a festive meal and seizing "a stolen motorbike without even checking the owner's papers."
She said the police then fired tear gas at the gathering and only left when a tourist bus approached, and denounced "harassment" by the security forces.
According to the testimonies, a man calling himself "the new Hitler" broke into the prayer hall of a Jewish school and assaulted one of the adults present, in front of the children.
Signup to our newsletter and follow us on Facebook and Twitter!
Lawsuits have been filed against the man, who also reportedly assaulted two young girls, and who was detained for 24 hours. But there has been no mention of charges linked to his allegedly racist aggression.
Thabet said the Tunisian Jews were "fed up with being used during visits by politicians as evidence for the peaceful coexistence (between Muslims and Jews) and when the law is broken, they are the first victims of negligence" by the authorities.
The NGO has championed the rights of minority communities in Tunisia, since it was established after the revolution in January 2011, including homosexuals.
It has criticised the authorities in the past for their inertia towards anti-semitic behaviour, drawing attention in particular to the judiciary's failure to prosecute an imam who called openly for a "divine genocide" of the Jews in a televised sermon late last year.
The country's Jewish community has seen its numbers fall dramatically in recent decades, from an estimated 100,000 at the time of independence in 1956 to around 1,500 now.
Earlier this year hundreds of worshippers made the annual pilgrimage to Djerba's Ghriba synagogue, thought to be the oldest in Africa, amid tight security after a suicide attack in 2002 that killed 21 people.