The visit coincided with a statement from Iranian President Hassan Rouhani expressing hope for improved relations with the kingdom.
Saudi officials greeted Foreign Minister Mohammad Javad Zarif after he landed at a military airport in the capital Riyadh.
Iran and Saudi Arabia, the region's foremost Shiite and Sunni Muslim powers, have had troubled relations in recent years after taking different sides in the Syrian civil war.
Tehran has backed Syrian President Bashar al-Assad while Riyadh supported Sunni rebels trying to topple him.
Zarif has previously declined a Saudi invitation to visit the kingdom, citing continued negotiations between his country and international powers over Iran's disputed nuclear programme.
In August last year Iran's Deputy Foreign Minister Hossein Amir-Abdollahian held talks with Saudi Foreign Minister Prince Saud al-Faisal in the Red Sea city of Jeddah.
That was the first high-level Iranian visit to the kingdom since Rouhani became the Islamic republic's president a year earlier.
Signup to our newsletter and follow us on Facebook and Twitter!
They discussed the fight against Islamic State group jihadists, who both nations oppose.
Rouhani has stated his wish to improve relations with Iran's neighbours, especially Saudi Arabia.
On Saturday he issued a statement congratulating Saudi Arabia's new king Salman on his accession after Abdullah's death on Friday.
With religious and historic bonds between the two nations, Rouhani told the Iranian state broadcaster that he hoped relations can be improved "more than ever."
"I pray to God for your health and success and prosperity for the nation and government of Saudi Arabia," he said.
Since the June 2013 election of Rouhani, regarded as a moderate, Iran has engaged in a diplomatic push with Riyadh, although relations have soured in recent months over the falling global price of oil.
Both countries are members of the Organization of the Petroleum Exporting Countries but Iranian officials have expressed frustration at Saudi Arabia for not taking steps to cut supply.
Iran has been among the oil exporting countries worst hit by the price fall.