Joanne Liu, president of Medecins Sans Frontieres (MSF), published an open letter describing "the horrific situation" for refugees and migrants in Libyan detention centres she visited last week.
Libya's detention of migrants "must be named for what it is: a thriving enterprise of kidnapping, torture and extortion," Liu wrote in the letter addressing European governments.
Italy and the European Union have been financing, training and providing other aid to Libya's coastguard to stop smugglers from taking migrants and refugees in flimsy boats across the Mediterranean to Europe.
Migrants are then sent to detention centres.
The number of migrant arrivals in Italy in July was down dramatically on the same month last year, suggesting efforts to train up and better equip the North African country's coastguard could already be having an impact.
But Liu said describing fewer migrant departures as a success in preventing drownings and smashing smuggler networks amounts to "pure hypocrisy" or worse, "a cynical complicity" in what is happening.
Signup to our newsletter and follow us on Facebook and Twitter!
In the detention centres, migrants "are packed into dark, filthy rooms with no ventilation," she said.
Detainees told MSF how men are forced to run naked in the courtyard until they collapse from exhaustion, while women are raped and made to call family back home for money to free them.
"All the people I met had tears in their eyes, asking again and again to get out," Liu said.
She said the migrants instead need access to protection, asylum, increased voluntary repatriation procedures as well as safe passage across borders.
Catherine Ray, spokeswoman for the EU's diplomatic service, told reporters that Brussels is trying to tackle the abuses.
"We are completely aware of the unacceptable, often scandalous, even inhumane conditions in which migrants are treated in reception camps in Libya," Ray said.
"We are trying to support the organisations which have access to these camps so that they can help the migrants."
She said more than 7,000 people have been helped to return voluntarily to their home countries while the EU is looking to open up "legal avenues" for those deemed to need international protection.