Riyadh said the move Wednesday was in response to criticism of Saudi Arabia's human rights record by Swedish Foreign Minister Margot Wallstroem, which it said was "harmful to the kingdom."
Her remarks, which Riyadh did not identify, were "flagrant interference in internal affairs, which is not accepted in international conventions."
The foreign ministry added that because Wallstroem's comments do not "go well with amicable relations between countries, the kingdom of Saudi Arabia has recalled its ambassador from Sweden".
Swedish foreign ministry spokesman Erik Boman said earlier Wednesday that "diplomatic relations are not broken, but Saudi Arabia's ambassador has been recalled."
The move comes amid a deepening rift in which Sweden cancelled a controversial military cooperation deal after Saudi Arabia prevented Wallstroem Monday from making a speech on human rights at the Arab League.
Wallstroem had been invited to the Cairo meeting in praise of her government's decision in October to recognise a Palestinian state.
Her speech, published by the Swedish foreign ministry, mentioned neither Saudi Arabia nor her feminist foreign policy agenda, but stressed women's and human rights.
"Freedom of association, assembly, religion and expression are not only fundamental rights and important tools in the creation of vibrant societies," it read, noting that "women's rights do not only benefit women, but society as a whole."
Saudi Arabia is the only country in the world where women are not allowed to drive.
The sexes are strictly segregated, women are required to dress in black from head to toe when outside the home and also still need permission from a male guardian to work and to marry.
Wallstroem said "the explanation we have been given is that Sweden has highlighted the situation for democracy and human rights and that is why they do not want me to speak."
- 'Nearly medieval methods' -
She has rarely commented on Saudi Arabia but, in January, criticised the kingdom's treatment of blogger Raef Badawi, who had been sentenced to 1,000 lashes and 10 years in prison for insulting Islam.
Signup to our newsletter and follow us on Facebook and Twitter!
"One must protest against what are nearly medieval methods" of punishment, Wallstroem told TT.
On Tuesday, a day after the Cairo incident, Sweden scrapped its 10-year-old military deal with the Saudis in retaliation.
Saudi Arabia is the third-largest non-Western buyer of Swedish arms. In 2014, Riyadh bought equipment worth 338 million kronor (37 million euros, $39 million).
The deal involved exchanges of military products, logistics, technology and training. Signed by a left-wing government in 2005 and renewed in 2010, it sparked a heated debate after Swedish Radio revealed in 2011 that Sweden had secretly helped the Saudis construct a weapons factory.
Social Democrat Prime Minister Stefan Loefven told public broadcaster Swedish Radio that "this is not a game. It's a serious issue that must be treated seriously."
His party had come under intense pressure to abandon the deal from coalition partners the Green Party.
The Social Democrats refused to cite human rights as a motive for the decision but Wallstroem herself said her speech at the Cairo meeting was blocked.
Commenting Wednesday on the military deal, Wallstroem said "it's a good and correct decision we have made.
"And I feel that when I speak about democracy and human rights, I do it with the support of the Swedish people," she added in remarks to the TT news agency.
Swedish Defence Minister Peter Hultqvist said only cooperation in medicine and gender studies would remain on offer.
"In practical terms, there is no military cooperation," he told public broadcaster SVT.
"What we have is an open invitation to partake in medical and gender training, but the Saudi side has not shown any interest," he added.
Commenting on the severed military ties, liberal writer Fredrik Segerfeldt wrote that Sweden's objective was "to become a moral power" on the world stage.
Such policies recall the bold statements of slain Social Democrat Prime Minister Olof Palme, who made radically anti-US remarks on the Vietnam War and slammed the apartheid regime in South Africa as well as the rule of Augusto Pinochet in Chile, before being shot dead on a Stockholm street in 1986.
But taking a stance against Saudi Arabia today risked Sweden's credibility as a business partner, according to some centre-right opposition politicians and the Swedish business community.
"Foreign policy is not only about other countries," right-wing daily Svenska Dagbladet wrote in an editorial, noting that Swedish industry "must be allowed to trade... even with dictatorships".