It is the first time the London-based watchdog has issued allegations of human rights abuses specifically about a 2022 World Cup venue.
The treatment of workers is one of the major issues that has dogged the Gulf state since the controversial decision to award it the right to host football's flagship tournament.
Amnesty said labourers at the Khalifa International Stadium were lied to over their salaries, went unpaid for months and were housed in squalid accommodation.
Seven workers were even prevented from returning home to help their families after Nepal's devastating 2015 earthquake.
"This is a World Cup based on labour exploitation," said Amnesty's Mustafa Qadri.
The 51-page report, "The Ugly Side of the Beautiful Game", is likely to increase international criticism of Qatar and put fresh pressure on FIFA and its new leadership which has pledged to overhaul policies.
The criticisms also extend to landscaping at the nearby Aspire Zone, a world-renowned sports complex where Paris Saint-Germain and Bayern Munich trained this year.
Amnesty said it carried out its research for a year until February and interviewed 234 men, mostly from Bangladesh, India and Nepal.
It claimed 228 men said the wages they received were lower than promised.
Many had no choice but to accept reduced salaries as they had paid debt-inducing recruitment fees of up to $4,300 to enter the Gulf, even though such fees are illegal under Qatari law.
Qatar was uncharacteristically quick to respond to the report, pointing out steps it has taken to improve practices.
It said Amnesty's claims painted "a misleading picture" and were aimed at only four of the 40 companies working at the Khalifa Stadium -- which will also host the 2019 World Athletics Championships.
'Like a prison'
One group of men also told of how one labour supply company, Seven Hills, did not pay staff for months.
Workers at the 40,000-seater stadium and Aspire Zone were also initially put in sub-standard and overcrowded accommodation, said Amnesty.
Signup to our newsletter and follow us on Facebook and Twitter!
"In one case, the main entrance road to the camp was flooded due to inadequate drainage and smelled of raw sewage," the report said.
Amnesty alleged most labourers had their passports unlawfully confiscated, and 88 men including Nepalese "were denied the right to leave Qatar".
One worker, Deepak, said life in Qatar was "like a prison".
"When I first complained... the manager said 'If you complain you can but there will be consequences. If you want to stay in Qatar be quiet and keep working'," he said.
Amnesty recognised the tournament's organisers, the Supreme Committee for Delivery and Legacy, had introduced safeguards.
But it called for further reform, including applying workers' welfare standards to companies involved in World Cup projects.
It also urged Qatar to "fundamentally reform" its "kafala" sponsorship system, which restricts job changing or leaving the country, and has scathing words for FIFA's "lack of meaningful action".
Qatar defends record
Supreme Committee chief Hassan al-Thawadi admitted abuses occur, but said its commitment to reform was "clear and steadfast" and reiterated that no workers had died on World Cup projects.
Relevant reforms had been introduced to address concerns raised by the report, even before the committee knew about it.
These included wage protection, improved accommodation, and sanctions against companies named in the report including the termination of Seven Hills' contract.
"We have always maintained this World Cup will act as a catalyst for change -- it will not be built on the back of exploited workers," said the committee in a statement.
"We wholly reject any notion that Qatar is unfit to host the World Cup."
FIFA said in a statement it was "confident that the structures and processes" put in place by the Supreme Committee would help monitor labour conditions.
Qatar currently employs about 5,100 workers on World Cup sites. Their number will peak at 36,000 by 2018.