"I can't allow you to visit this site without prior coordination," an official at the site told the delegation when they stopped for a "surprise visit."
The site is a joint venture being built by French group Vinci and Qatar's Diyar company.
In response, the 18-member team from the Building and Wood Worker's International federation, gave up a visit to the major Lusail site, 70 kilometres (43.5 miles) north of Doha, where the main stadium for the World Cup is being built.
These scheduled visits only serve as "a show case" in Qatar's benefit, a member of the delegation Ampet Yuson, told AFP.
Members of the delegation then headed to the headquarters of the World Cup organising committee, where they were not allowed to meet with a representative of the committee due to the lack of an earlier appointment.
They protested outside the offices chanting: "FIFA, red card." Following mediation, Hassen Dhawadi agreed to receive them later in the evening.
British daily, The Guardian, last month, said 44 Nepalese workers have died on construction sites in Qatar.
The Gulf state has come under mounting pressure to end exploitation of migrant workers, as it embarks on a multi-billion-dollar construction programme to prepare to host the 2022 football World Cup.
Signup to our newsletter and follow us on Facebook and Twitter!
The International Trade Union Confederation (ITUC) had slammed as "weak and disappointing" Qatar's response to media claims that Nepalese workers were treated as slaves.
The head of Qatar's National Human Rights Committee, Ali Al-Marri, admitted last week some problems but denied claims that the Nepalese workers were treated like "slaves".
The labour minister's undersecretary Hussein al-Mullah, told the delegation that companies in Qatar have met "99 percent of the (local) labour code."
"We intervene where a worker files a complaint," he said.
But an Indian worker, who arrived in Qatar four months ago, told AFP that "if I complain to Qatari authorities, my employer sends me back to my country."
The Work Ministry's Ali Ahmad Al-Khalifi had pledged to double the number of work inspectors to 150, a pledge considered by the ITUC as "weak".
ITUC lodged a complaint in March with the labour ministry against six Qatari companies after being contacted by workers.
The union body says Qatar's labour ministry received 6,000 worker complaints in 2012 while the Indian embassy there received 1,500 complaints in the first five months of this year.
Among the complaints are that false promises are made to workers, that employer obligations on wages and working conditions are not met, that contracts are not respected, passports are withheld, workers are indebted to recruiters or moneylenders, and that workers are forced to live in crowded squalid camps.