Libya has been riven by violence and lawlessness since Kadhafi was toppled and killed in a NATO-backed revolt in 2011, and his son Seif al-Islam and 36 former regime officials are on trial for crimes including human rights violations.
The report by the UN's Libya mission and Human Rights Office highlighted a series of flaws in the trial.
"Holding perpetrators responsible for violations is vitally important but accountability should be the result of due process and a fair trial," said UN High Commissioner for Human Rights Zeid Ra'ad Al Hussein.
Signup to our newsletter and follow us on Facebook and Twitter!
"This trial was a missed opportunity for justice and for the Libyan people to have the chance to confront and reflect on the conduct of the former regime."
The report detailed "serious violations" of due process, including "prolonged periods of incommunicado detention for the defendants amid allegations of torture which were not properly investigated".
It said that defence lawyers reported repeated difficulties in meeting those on trial and reported the lack of a full official trial record.
The report recommended a review of Libya's penal code, guaranteed right of access to lawyers during interrogation and the strengthening of other fair trial safeguards.
It also urged Libya to comply with its international obligations by transferring Seif al-Islam, who was captured by rebels in November 2011, to the International Criminal Court.