Riyadh said Iran's hajj demands were "unacceptable".
The Iranian Hajj Organisation said: "Saudi Arabia is opposing the absolute right of Iranians to go on the hajj and is blocking the path leading to Allah."
The Saudi side had failed to respond to Iranian demands over "the security and respect" of its pilgrims to Mecca, of whom 60,000 took part in last year's hajj, the organisation said.
In the latest dispute between regional rivals Tehran and Riyadh, "after two series of negotiations without any results because of obstacles raised by the Saudis, Iranian pilgrims will unfortunately not be able to take part in the hajj" in September, Iran's Culture Minister Ali Jannati said.
Saudi officials have said an Iranian delegation ended a visit to the kingdom on Friday without reaching final agreement on arrangements for pilgrims from the Islamic republic.
Riyadh's hajj ministry said it had offered "many solutions" to meet a string of demands made by the Iranians in two days of talks.
Agreement had been reached in some areas, including to use electronic visas which could be printed out by Iranian pilgrims, as Saudi diplomatic missions remain shut in Iran, it said.
On Sunday, at a joint press briefing in Jeddah with Britain's visiting Foreign Secretary Philip Hammond, Saudi Foreign Minister Adel al-Jubeir denounced Iran's demands.
"Iran has demanded the right to organise... demonstrations and to have privileges... that would cause chaos during the hajj. This is unacceptable," Jubeir said.
He said Riyadh annually signs a hajj memorandum of understanding with more than 70 countries "to guarantee the security and safety of pilgrims", but "Iran refused to sign the memorandum".
- 'More than our duty' -
Signup to our newsletter and follow us on Facebook and Twitter!
"If it is about measures and procedures, I think we have done more than our duty to meet those needs, but it is the Iranians who have rejected things," Jubeir added.
This year's would be the first hajj in almost three decades to take place without the participation of pilgrims from Iran.
Riyadh-Tehran ties were severed for four years after more than 400 people were killed in Mecca during clashes between Iranian pilgrims and Saudi security forces in 1987.
In January, relations were severed again after Iranian demonstrators torched Saudi Arabia's embassy and a consulate following the kingdom's execution of a prominent Shiite cleric.
Shiite Iran and predominantly Sunni Saudi Arabia are at odds over a raft of regional issues, notably the conflicts in Syria and Yemen in which they support opposing sides.
Earlier this month, Iran accused its regional rival of seeking to "sabotage" the hajj, a pillar of Islam that devout Muslims must perform at least once during their lifetime if they can.
Tehran said Riyadh had insisted that visas for Iranians be issued in a third country and would not allow pilgrims to be flown aboard Iranian aircraft.
But the Saudi hajj ministry said on Friday that Riyadh had agreed to allow Iranians to obtain visas through the Swiss embassy in Tehran, which has looked after Saudi interests since January.
Riyadh also agreed to allow some Iranian carriers to fly pilgrims to the kingdom despite a ban on Iranian airlines following the diplomatic row, the ministry said.
Last week's talks were the second attempt to reach a deal on organising this year's pilgrimage for Iranians after an unsuccessful first round of talks held in April in Saudi Arabia.
The Saudi ministry said at the time that the Iranian Hajj Organisation would be held responsible "in front of God and the people for the inability of its pilgrims to perform hajj this year".
Another contentious issue has been security, after a stampede at last September's hajj killed about 2,300 foreign pilgrims, including 464 Iranians.