The system would facilitate oil companies and businesses to continue trading, without relying on the US-led global market and dollar.
Exactly how the system would work is still being determined.
Earlier this year, President Donald Trump pulled the US from the 2015 Iran nuclear deal and re-imposed sanctions.
The Joint Comprehensive Plan of action, as the nuclear deal is officially known, was negotiated during the US presidency of Barack Obama.
It saw Iran limit its nuclear activities in exchange for sanctions relief.
EU foreign policy chief Federica Mogherini announced the plan after talks at the UN with the remaining members of the accord - Britain, China, France, Germany, Russia.
A statement said they were determined to "protect the freedom of their economic operators to pursue legitimate business with Iran".
Ms Mogherini said the technical experts would discuss the details of the new payment mechanisms.
Analysts though doubt whether such a mechanism is feasible given the US could modify its sanctions to encompass any new system.
Mr Trump has called the deal "one-sided", "disastrous" and the "worst I've ever seen".
He believes renewed economic pressure, which went into force in August, will force Iran to agree to a new deal.
The value of Iran's currency, the rial, has been damaged by the new US policy.
More about: US Iran nuclear-deal
