In May, forces loyal to the unity government began an offensive to retake the Mediterranean city and home town of slain dictator Moamer Kadhafi, which IS seized in June last year.
The offensive came amid growing concern that jihadists would use Sirte, where they had set up centres to train militants, as a springboard for attacks on European shores hundreds of kilometres (miles) away.
One June 9, pro-government forces entered Sirte and more than two months later pushed the jihadists from key positions including their headquarters at the Ouagadougou conference centre, a sprawling compound near the city centre.
AFP correspondents who toured the city on Saturday said loyalist forces were pressing their advance, buoyed by US air strikes earlier this month that targeted IS holdouts in the city.
The forces fired machineguns mounted on pick-up trucks and light weapons at IS targets in the distance as they tried to advance towards an area known as "residential district three" facing the Mediterranean, the correspondents said.
On Sunday, the spokesman for the forces, General Mohamad Ghassri, told AFP that both residential districts near the waterfront were "currently battle zones".
Pro-government forces "have entered district number two" which could fall under their control later Sunday, he said.
"IS has only control over one sector, residential district one in the heart of downtown Sirte," he said.
"From a military point of view, the battle (for Sirte) is over," he said, adding that "victory" would be announced soon.
Earlier the forces loyal to the UN-backed Government of National Accord said in a statement that their fighters had seized buildings on the outskirts of district two and were chasing the jihadists.
- Eyes on Rome -
Mustafa al-Faqih, one of the commanders of the loyalist forces, told AFP that the jihadists fled after their headquarters fell on Wednesday.
Signup to our newsletter and follow us on Facebook and Twitter!
The "enemy forces collapsed" in the face of the loyalist advance, he said.
"We will advance on... areas where the enemy is still deployed in the coming days, God willing."
The taking of the Ouagadougou centre, where Kadhafi once hosted Arab and African summits as well as European leaders, was the first in a string of blistering losses sustained by the jihadists this week.
A statement by the loyalist forces on Saturday said they had seized a radio station near the centre which the jihadists used to broadcast propaganda, describing its capture as "important".
They also drove jihadists out of the university of Sirte which they had used as a key defensive position, they said.
On Sunday, pro-government fighters could be seen removing jihadist graffiti from the walls of the Ouagadougou centre.
"The caliphate will endure and expand," read one slogan at the centre's main entrance, before it was removed.
A trail of destruction has hit the once grandiose centre, with windows shattered, ceilings caved in and bullet strikes on its walls.
Outside, a group of pro-government fighters flashed victory signs and pointed to a banner left behind by the jihadists that read: "We are fighting in Libya but our eyes are on Rome."
Libya's former colonial power Italy lies a mere 300 kilometres (less than 200 miles) across the Mediterranean from Sirte.
On Thursday, Italian Prime Minister Matteo Renzi faced criticism at home for reportedly sending special forces to Libya to help the anti-IS fight without approval from parliament.
Renzi's centre-left government has refused to confirm or deny reports that dozens of special forces have been deployed to help with de-mining and training pro-GNA forces.
More than 300 pro-government fighters have been killed and 1,800 wounded since May in the battle for Sirte, medics have said.
The jihadists have not revealed their losses.