Cameron was speaking about a visit last week by opposition leader Jeremy Corbyn to migrant camps in Dunkirk and Calais in northern France.
"They met with a bunch of migrants in Calais. They said they could all come to Britain," he said.
Yvette Cooper, who chairs the Labour Party's refugee task force, said that Cameron's words, made during parliament's weekly Prime Minister's Questions (PMQs) session, were inappropriate for a "complex and sensitive" issue.
"Cameron very wrong to talk of 'a bunch of migrants' in . Divisive, not statesmanlike," she tweeted.
Labour MP Imran Hussain said it was "shocking" and a senior Labour source said it was "entirely unacceptable to a humanitarian crisis on our doorstep".
Signup to our newsletter and follow us on Facebook and Twitter!
The quote was also quickly picked up on social media, with many commenters pointing out that Cameron used similarly pejorative language last year when he referred to migrants in Calais as a "swarm".
Many tweets also included poignant images of Europe's migrant crisis alongside Cameron's words and pointed out the irony of his earlier tribute to the victims of Nazi repressions on Holocaust Memorial Day.
"Today it struck an especially discordant note," columnist Jonathan Freedland wrote in The Guardian.
"One of the lessons of the Shoah... is that is all too easy to dehumanise other people, to turn them from human beings with lives and needs and hopes into a problem to be repelled," he said.
"To speak the way he did was beneath the office he holds -- and beneath him," he concluded.
Corbyn, a veteran left-wing campaigner before he became Labour party leader, visited two migrants camps saying he wanted to see the crisis for himself and calling on Britain to do more to address the problem.