According to the suspects' plan, two attackers were to detonate suicide vests while the others were to kill passersby with guns and explosives in the western city of Duesseldorf, prosecutors said in a statement.
The suspects were identified as 27-year-old Hamza C., 25-year-old Mahood B., and Abd Arahman A. K., 31, who were arrested in the states of Brandenburg near Berlin, North Rhine-Westphalia and Baden-Wurttemberg.
A fourth Syrian man, Saleh A., 25, had been in custody in France since turning himself in in February, they said, and Germany had now requested his extradition.
"According to current investigations, the four accused were planning to commit an attack in Germany for the foreign terrorist organisation Islamic State," said the prosecutors.
Saleh A. and Hamza C. had joined IS in early 2014 in Syria, where "they were ordered by the group's leadership to carry out an attack in the old town of Duesseldorf".
"Two suicide attackers were to set off their explosives vests at the Heinrich-Heine-Allee, a major street in Duesseldorf," the prosecutors said in the statement.
"After that, other attackers were to kill as many passersby as possible with guns and other explosives."
Signup to our newsletter and follow us on Facebook and Twitter!
Saleh A. and Hamza C. had crossed from Syria to Turkey in May 2014. From there they travelled separately in March and July last year via Greece to Germany.
Germany last year saw a major influx of 1.1 million migrants and refugees who mostly travelled overland, the majority via Turkey, Greece and Balkans countries, to seek asylum in Europe's biggest economy.
Abd Arahman A. K. had already arrived in Germany in October 2014 "on the orders of the IS leadership", tasked with manufacturing the suicide vests, said the statement.
By January this year, Saleh A. and Hamza C. had convinced Mahood B. to join in the attack.
The prosecutors added that there was no evidence "that the accused had already initiated the concrete implementation of their attack plan".
And they said that "today's arrests are not related to the forthcoming European football championship in France".
Since the November 13, 2015 attacks in Paris that killed 130 people, German authorities have repeatedly said they believe Germany faces the threat of jihadist attacks.
Last August, IS threatened Germany and Austria with attacks in an online execution video, urging jihadists to commit attacks against "unbelievers" there.
Since then Germany has seen two knife attacks against police blamed on Islamists, a 15-year-old girl and a 41-year-old Iraqi man.