A handful of economists are already downgrading their outlook for U.S. growth this year after Britons voted to leave the European Union. The vote sparked severe volatility in stock markets and a rally in the dollar. For America, that`s a one-two punch.

Experts at Goldman Sachs (GS), Barclays and Bank of America (BAC) lowered their forecast for U.S. economic growth. Other economists too told CNNMoney they anticipate reducing their outlook later this week but requested not to be named since the forecasts weren`t public yet.

The reductions are minor for now. However, the huge uncertainties unleashed by the vote, along with the stock market selloff, are the key reasons behind the volume being turned down on U.S. growth.

"We expect only a small net drag on U.S. growth from the Brexit vote, with the caveat that the outlook could change significantly if the equity markets keep falling," Jim O`Sullivan, chief U.S. economist at High Frequency Economics, a research firm that slightly lowered its outlook too.

The last thing America needs is another headwind. Growth so far this year has been anemic: the U.S. economy only grew 1.1% between January and March on an annual basis, the Commerce Department reported Tuesday. That`s higher than the initial estimate of 0.5%, but still very low.

Beyond weak economic growth, job gains have started to slow this spring and American employers are very tepid about investing in their own businesses. Better growth was already an uphill battle, before Brexit.

American trade with the U.K. makes up less than 1% of U.S. economic activity. Nor is the U.S. directly affected if U.K. economy sinks into recession as a result of leaving the EU.

However, Brexit could hurt the U.S. economy in two key ways: volatile stock markets and a strong dollar.

When stocks bounce around a lot -- in just the last two days, the Dow lost close to 900 points -- Americans tend to hold off on spending. And U.S. consumers make up about 70% of economic activity in America. When they don`t spend, the economy doesn`t grow. How they feel about spending is often linked to how good or bad U.S. stocks are doing.

Secondly, a strong dollar makes American exports more expensive, and less attractive, to foreign buyers. A stronger dollar usually hurts U.S. exports and trade -- a theme that played out last year and especially hurt American manufacturers.

Brexit "is another in a long string of confidence shocks, hitting an already vulnerable US and global economy," says Ethan Harris, global economist at Bank of America, who cut his growth estimate down by 0.2 percentage points.

Not everyone is dimming their outlook.

UBS economist Samuel Coffin is sticking with his forecast of the U.S. economy, though he adds "the U.K. decision does add some downside risk to our growth forecasts."

So don`t hit the panic button on America`s economy -- Brexit doesn`t appear to be triggering a recession in the States this year. But the historic vote means major uncertainties loom large on both sides of the Atlantic.

More about: