It was not immediately clear who was behind the attack.
The refugees were hit by light weapons fire in waters off rebel-held Hodeida, but the boat managed to dock in the city's port, an official there said.
The International Organization for Migration, which has operations in Yemen, said 42 bodies had been recovered.
The port official said dozens of Somalis who survived the attack, as well as three Yemeni traffickers, had been taken to the city's prison.
More than 30 wounded were reported to have been taken to hospital.
The rebel-controlled Saba agency accused the Saudi-led coalition battling the insurgents of attacking the refugees from the air but did not provide further details.
Major General Ahmed Assiri, spokesman for the coalition, dismissed the accusation and told AFP that the force had not been involved in fighting in Hodeida.
"There has been no firing by the coalition in this zone," he said.
The coalition has been backing pro-government forces waging an offensive from the south to retake Yemen's Red Sea coast from Huthi rebel forces.
The United Nations refugee agency UNHCR said 140 passengers were believed to have been aboard the vessel.
CIVILIANS BEAR THE BRUNT
The attack drew condemnation from UN agencies and the International Committee for the Red Cross, with the ICRC also demanding an immediate investigation.
Despite a war that has cost more than 7,000 lives since March 2015 and brought the country to the brink of famine, Yemen continues to attract people fleeing the horn of Africa.
The UNHCR says that Yemen is hosting more than 255,000 Somali refugees.
Signup to our newsletter and follow us on Facebook and Twitter!
Several refugee camps in southern Yemen host Somali refugees, although not in the Hodeida area.
The UNHCR said that as conditions worsen in Yemen, refugees are starting to use areas further to the north as a transit route.
It called on all sides in Yemen's war to protect civilians.
"UNHCR is appalled by this tragic incident, the latest in which civilians continue to disproportionately bear the brunt of conflict in Yemen," it said.
The ICRC's Eric Christopher Wyss, quoting survivors at the scene, said many people on the boat "were refugees from Somalia or Yemen, fleeing the conflict".
"It was a gruesome and heartbreaking scene. I saw many men, women and children either killed or horribly wounded," he said.
The IOM said it believed the boat had been headed for Sudan.
"We strongly condemn this attack and deplore such a tragic loss of life," the ICRC's director of operations, Dominik Stillhart, said in a statement.
"These people were themselves fleeing conflict, in search of safety and a better life. We call on the warring parties to conduct an immediate investigation into what happened."
The coast south of Hodeida has seen violent clashes over the past 24 hours, leaving 32 dead including 10 civilians, according to military and medical officials.
The fighting killed 14 rebels and eight soldiers supporting the UN-backed government of President Abedrabbo Mansour Hadi, military sources said.
That came as pro-Hadi forces sought to advance northwards from the port town of Mokha in an offensive to retake Yemen's western coast from Huthi rebel forces.
Saudi-led coalition aircraft had taken part in the assault, the same sources said.
Seven fishermen were also killed off Hodeida by gunfire from an unidentified source, while a further seven were killed in a car near Mokha, hospital workers said.
The UN's humanitarian chief Stephen O'Brien has called Yemen "the largest humanitarian crisis in the world", with two-thirds of the population in need of aid.
On Friday, a rebel missile attack killed at least 26 members of the pro-government forces in a camp east of the capital Sanaa, officials at a hospital in Marib town said.