Warning Israelis that the "Islamic State is already here," Rivlin told a conference in Tel Aviv that more support was needed for Arab communities.
A number of Arab Israeli citizens have travelled to fight with IS in neighbouring Syria.
The Israeli government in December agreed in principle to increase financing for Arab-Israeli communities, but the funding has not been finalised.
Rivlin, whose position as president is largely symbolic, said that while radicalism was not unique to Israel, "I am concerned that the more the state avoids taking responsibility, the more the state distances itself, the faster the jihadi Salafists will rush in to fill the vacuum."
Recalling a meeting with a teacher in an Arab community, Rivlin said he was told many Arab Israelis feel like second-class citizens in the country.
Arab Israelis say they are discriminated against and face marginalisation in Israeli society.
"If the children are growing up without a dream, without hope and without aspirations, with the feeling that their blood, their lives, are of a lesser value in the state of Israel, then we must think how we can offer them a dream, hope, and faith," he said.
Signup to our newsletter and follow us on Facebook and Twitter!
Rivlin called on the government to push the funding through as quickly as possible.
"The recent resolution by the government on a system-wide plan for the economic integration of the Arab population is a step in the right direction," he said.
"The plan has many detractors among the Jewish population, and it is clear why. But it is the correct and essential step, for it is a decision that represents a systemic change of direction."
Rivlin has often been at odds with the government of Prime Minister Benjamin Netanyahu, though both come from the rightwing Likud party.
Arab Israelis, the descendants of Palestinians who remained on their land after the creation of Israel in 1948 and who are citizens of the Jewish state, make up about 18 percent of its population.
Israeli security forces estimate around 50 of them have travelled to Syria to fight alongside IS.
A shooting spree in Tel Aviv by an Arab Israeli this month that killed three people led to speculation that the gunman may have been inspired by IS, but his motive remains unclear.
He was killed in a shootout with police about a week later.