A fifth suspect, a minor, was in detention in a juvenile centre pending further proceedings, the source told AFP.
Spanish and Moroccan authorities said the suspects were believed to have enlisted 12 people who were then sent to IS-run territories in Syria and Iraq.
Judge Santiago Pedraz at the National Court in Madrid "ordered unconditional detention for membership in a terrorist organisation" for the four adults, the source added.
The five were taken into custody on Tuesday in a series of raids against a suspected recruiting cell that police say contacted women online and radicalised them.
Authorities arrested the suspects in Barcelona as well as in Ceuta and Melilla, two Spanish territories bordering Morocco. Two other suspects were taken into custody in Morocco near the border with Ceuta.
Signup to our newsletter and follow us on Facebook and Twitter!
According to the source, one of the suspects refused to answer the judge's questions while the three others responded.
The Moroccan government said the suspects had links to members of the IS group planning attacks in Morocco.
Ten years after the March 2004 Al-Qaeda-inspired bombings killed 191 people in Madrid, police and security analysts say homegrown jihadists, already familiar in Britain and France, have started to appear in Spain.
Security sources say up to 100 radicals have been sent to Iraq and Syria after being recruited in Spain. Hundreds of others are said to have joined IS from other European countries such as Britain and France.
In August police detained two females -- a 19-year-old from Melilla and a girl of 14 from Ceuta -- who tried to cross from Melilla into Morocco, suspected of heading for Syria.
The 14-year-old is also in detention in a juvenile centre.