Interior Minister Thomas de Maiziere said the men, one of them 17, appeared to have been smuggled to Europe through some of the same channels.
They were detained after a series of pre-dawn raids in the states of Schleswig-Holstein and Lower Saxony.
They are said to have travelled through Turkey and Greece on false passports.
They have been named only as Mahir al-H, 17, Ibrahim M, 18, and Mohamed A, 26.
Investigations so far suggest the three came to Germany in November 2015 with the intention of "carrying out a previously determined order [from IS] or to await further instructions", prosecutors said in a statement.
The Federal Public Prosecutor`s office has said no concrete missions or orders have so far been found, despite the seizure of "extensive material".
Mr de Maiziere said those arrested could have constituted a "sleeper cell" - a unit of people who remain dormant and inconspicuous in a community until activated.
More about:
