"We should begin to look seriously at the question of the spread of the terrorists once Sirte... (is) emptied of the terrorists," Le Drian told a defence conference in Paris.
"They don't disappear. There's a new risk that appears," he said, adding: "Indirectly this will pose new risks for Tunisia and Egypt."
He said it was a "shame, perhaps political reasons prevent it, that all the neighbouring states of Libya don't meet" over the issue.
Le Drian's Tunisian counterpart Farhat Horchani also called for effective regional coordination.
"We have a large number of foreign fighters who arrived from Sirte, or from Syria. I can see no strategy, no cooperation between the states," to deal with the problem, said Horchani who attended the same defence conference in Paris.
Backed by weeks of US air strikes, forces loyal to Libya's Government of National Accord (GNA) have recaptured nearly all of what had been the jihadists' main stronghold in north Africa.
Pro-GNA forces launched a new attack on Saturday against IS in Sirte, saying Sunday it could take several days to gain full control of the city.
Signup to our newsletter and follow us on Facebook and Twitter!
The Sunni extremists took advantage of the chaos in oil-rich Libya after the 2011 uprising to seize Sirte in June 2015, hoisting their black flag above the city.
The capture of Sirte by IS last year sparked fears the jihadists would use it as a springboard for attacks on Europe.
National support for the GNA is seen as crucial to restoring stability and to tackling IS, which took advantage of the chaos that has gripped Libya since the 2011 uprising that ousted dictator Moamer Kadhafi to take a foothold in the country.
While the loss of Sirte is a reversal for Islamic State fighters, between 5,000 and 7,000 of the jihadists remain in Libya, according to French and US figures.
Many "have evaporated in the south of the country," a French security source told AFP.
Le Drian warned that the Libyan crisis would only be solved "if the Libyans themselves secure real political and military coherence."
France wants to see the GNA consolidating its power and becoming recognised by everyone, he added.