The New York-based watchdog said IS, also known as ISIS, had inflicted "severe hardship" on the people of Sirte by diverting food, medicine, fuel and cash from the port city since February 2015.
At least 49 people had been killed by methods including decapitation and shooting for alleged crimes including blasphemy, sorcery and spying, HRW said in a report based on a series of interviews.
"The murder of civilians, or wounded or captive fighters, by members of a party to an armed conflict is a war crime, as is executing people without a fair trial by a regular court," said HRW.
"The nature and scale of ISIS's unlawful executions and other acts in Libya also may amount to crimes against humanity."
"The Sirte residents described scenes of horror -- public beheadings, corpses in orange jumpsuits hanging from scaffolding in what they termed 'crucifixions' and masked fighters snatching men from their beds in the night," it said.
"They said morality police aided by informants patrolled the streets, threatening, fining or flogging men for smoking, listening to music, or failing to ensure their wives and sisters were covered in loose black abayas (robes)."
Signup to our newsletter and follow us on Facebook and Twitter!
The 49 executions followed "largely secret proceedings that negate the most basic fair-trial standards".
"ISIS also has kidnapped and disappeared dozens of Libyan militia fighters, many of whom are presumed dead," said HRW, citing exiled councillors and fighters from groups opposed to it.
The 41-page report said IS jihadists had looted and destroyed the homes of those seen as enemies, as well as forced the closure of shops specialising in lingerie or Western clothing.
One resident, a 30-year-old named Ahlam, described life in Sirte as "unbearable" with everyone "living in fear" and innocent people being killed.
"There are no groceries, the hospital has no doctors or nurses, there is no medicine... There are spies on every street. Most people have left but we are trapped. We don't have enough money to leave," she was quoted as saying.
IS seized Sirte almost a year ago, exploiting the chaos in Libya as rival militias and governments battled for power in the wake of the 2011 revolt that ousted and killed dictator Moamer Kadhafi.
It has been seeking to expand westwards from the city, and Europe fears the jihadists could use its port and airport as a springboard to attack the continent.