The unrest in the overcrowded prison in Japeri, which is effectively run by a criminal gang, began on Sunday.
Three inmates who took part in the mutiny were shot, but are not thought to have life-threatening injuries. Police seized three guns and a grenade.
The unrest comes just days after the army took control of security in the state of Rio de Janeiro.
Officials at the Milton Dias Moreira prison said the riot could have been in response to the new security measures, which are aimed at tackling worsening gang violence.
They said that eight prison wardens and 10 inmates had been taken hostage by rebelling prisoners.
The rebels gave themselves up after several hours of negotiations.
Rio residents have expressed fears of a surge in violence in the next few months.
President Michel Temer has compared gang-related violence to "a cancer" and said last week that organised criminals had all but seized control of the state.
He appointed an army general to oversee security in the state after the governor made a plea for help after the annual carnival celebrations were marred by violence.
Rio's police budget has been slashed in recent years because of a financial crisis. There have been criticisms that police do not even have the money to pay for the petrol in their patrol cars.
Finances in Rio state have been badly hit by a national recession and a slump in oil prices, as well as high levels of corruption.
The financial problems have only emboldened criminal gangs.
Figures from the Rio state government show an 8% increase in killings last year over 2016 and a 26% jump since 2015.
There were chaotic scenes during the famous Rio carnival this month, with gun fights and looting. Three police officers died in violent clashes.
National TV news bulletins also broadcast footage of gangs surrounding and robbing tourists.
BBC
More about: RiodeJaneiro