The first parliamentary election since Tunisia's 2011 revolution pitted Ennahda against secular opponent Nidaa Tounes, with an array of leftist and Islamist groups also taking part.
Analysts predicted no single group would win the outright majority needed to govern alone.
But Ennahda's head acknowledged late on Monday that his party had finished second.
"#Ennahda president Rachid Ghannouchi congratulates B Sebsi (Beji Caid Essebsi) on his party's win... a few moments ago," his daughter Soumaya said on Twitter.
Party spokesman Zied Laadhari earlier told AFP that "they (Nidaa Tounes) are ahead by around a dozen seats".
"We will have around 70 seats and they will have about 80" out of 217 contested in Sunday's election, he said.
Nidaa Tounes, whose name means "Call of Tunisia", boasted on Facebook that it had bested its long-dominant Islamist rival.
"We won. Long live Tunisia," it said.
Speaking earlier, party leader Essebsi was slightly more cautious.
"We have positive indications that Nidaa Tounes could be leading," the 87-year-old told reporters.
Several media outlets, including Al-Mutawassit television, which is considered close to the Islamists, projected a lead of around 20 seats for Nidaa Tounes.
Ghannouchi stressed the need for consensus going forward.
"Whoever comes out top, Nidaa or Ennahda, the main thing is that Tunisia needs a government of national unity, a political consensus," he told Hannibal television.
"This is the policy that has saved the country from what other Arab Spring countries are going through."
Signup to our newsletter and follow us on Facebook and Twitter!
The election's organisers gave the provisional turnout in Sunday's landmark vote as 61.8 percent.
- 'Important milestone' -
Some 80,000 security forces were deployed to avert extremist attacks on polling day, which passed without major incident, as some three million from an electorate of five million people voted for a 217-seat parliament under a new constitution drafted in January.
US President Barack Obama hailed the vote as "an important milestone in Tunisia's historic political transition".
Tunisia is seen as a beacon of hope compared with other chaos-hit countries like Libya and Egypt, where regimes were also toppled during the Arab Spring three years ago.
But its transition has been tested by militant attacks and social unrest, while poverty and unemployment, which were key factors that sparked the 2011 revolt, remain unresolved.
The whole campaign was fought on the axis of the economy and security.
La Presse de Tunisie, the country's main French-language newspaper, said the election was a major achievement on the path to establishing stable institutions in the former French colony.
"The mission is almost accomplished, which is considerable," it said in a commentary.
The government had warned of possible jihadist attacks after a standoff on Friday between police and suspected militants near Tunis that killed a policeman and six suspects, five of them women.
Giving a stamp of approval, the head of the European Union's election observer mission, Annemie Neyts-Uytterbroeck, said voting had been "more than satisfactory".
An array of parties competed for seats in the new parliament, some fronted by stalwarts of veteran dictator Zine El Abidine Ben Ali who was toppled in 2011.
His ouster ushered in a coalition government and interim president that won international praise. But the birthplace of the Arab Spring protests has flirted with disaster.
A rise in militant activity last year when two opposition lawmakers were assassinated by suspected Islamists -- long suppressed under Ben Ali -- threatened to derail the road to democracy.
Critics accuse Ennahda and its secular allies, which have been running the country, of failing to address people's security needs or shore up the economy.
Ennahda has proposed the formation of a government of national unity and has not put forward a candidate for a November 23 presidential election, keeping its options open over whom it will back.