Air strikes by an international coalition helped rebel forces to overthrow Libyan dictator Moamer Kadhafi in 2011, but they have not been enough to dislodge IS from Syria and Iraq.
And would any NATO country be prepared to send troops into the vast desert country?
"IS has clearly established a foothold in Libya, and it's a matter of growing concern for the United Kingdom and other NATO countries, so it (military intervention) is certainly possible," Malcolm Chalmers of the London-based Royal United Services Institute (RUSI) think-tank said.
"The question is what the scale and nature of an intervention might be, and that will depend very much on how the threat develops over the next month."
IS has around 3,000 fighters in Libya, according to a French diplomatic source. Fighting has raged the past two days between IS fighters and Libyan forces as the jihadists pushed on the country's crucial coastal oil terminals.
The West fears that they could seize control of oil production as they have done in large parts of Iraq and Syria.
- Ground force unlikely -
The other fear is that Libya offers a direct yet dangerous route to Europe for jihadists who could hide among migrants crossing the Mediterranean from the Libyan coast.
Jihadists have also used Libya as a springboard for attacks across the border in Tunisia, including at the capital's National Bardo Museum and a beach resort, killing a total of 60 people, all but one of them foreign tourists.
Patrick Hamzaideh, a French former diplomat who was based in Libya, said Western intervention "would probably take the form of bombing from the air". He thinks action by ground troops is unlikely.
"But both options are a bad idea. They would not be decisive militarily," he said.
Before any military action is taken, Libya needs a political solution to unite its rival factions.
"Without it, another Western military intervention would condemn the country to instability for at least a decade, with a government that would have the appearance of a puppet of the West," he said.
Signup to our newsletter and follow us on Facebook and Twitter!
Analysts say a key issue is how Libya's many armed factions would react to Western military action.
Five years ago, many Libyans warmly welcomed Western air strikes in the euphoria of the revolt that ended Kadhafi's four decades in power.
But in 2016, it is a very different picture. The influence of the Islamists has grown, as has nationalist sentiment among the armed groups.
Italy, the former colonial power which still has substantial economic interests in Libya, said last month it was prepared to lead an international force in the country, providing that a unity government asks for it to be there.
But Nicola Latorre, the head of the Italian Senate's defence committee, ruled out "any military intervention to wage war".
"It would be a fatal mistake to allow Daesh (an Arab name for IS) to say that Libya had been invaded by the West," he told AFP, saying the aim of any action would be to guarantee the security of a new government and train the army.
Speaking after rival Libyan politicians signed a deal on December 17 to form a unity government, British Defence Secretary Michael Fallon said up to 1,000 British special forces could be among approximately 6,000 European and US soldiers who could take part in offensives to halt IS's advance.
But the deal is already under pressure because it does not have unanimous backing from Libya's two rival parliaments, one based in the eastern city of Tobruk and the other in the capital Tripoli.
- Taking root on Tunisia border -
US and British special forces are already in Libya to assess the situation and Washington has said it will offer full support to a unity government, if one can be formed.
French officials meanwhile say reconnaissance flights are helping them build up a host of information about IS targets in Libya.
They say the Islamists are making inroads into villages in neighbouring Chad "and are beginning to take root on both sides of the border with Tunisia," said a French defence official, speaking on condition of anonymity.
"The idea is to create an international force under a UN mandate. We are confident Russia will vote for the resolution," the official said.
But Jean-Vincent Brisset, from the IRIS think-tank in Paris, said that the French army is already "overstretched".