Pro-marijuana activists cheered in the public gallery and chanted "Weed are free now" when the Constitutional Court gave its landmark ruling.
In a unanimous ruling, judges also legalised the growing of marijuana for private consumption.
South Africa's government's had opposed its legalisation, arguing the drug was "harmful" to people's the health.
It has not yet commented on the ruling, which is binding.
In April Zimbabwe became the second country in Africa, after Lesotho, to legalise the use of marijuana for medical use.
Three South African cannabis users who had faced prosecution for using the drug brought the case, saying the ban "intrudes unjustifiably into their private spheres".
In his judgement, Deputy Chief Justice Raymond Zondo said: "It will not be a criminal offence for an adult person to use or be in possession of cannabis in private for his or her personal consumption."
It will, however, remain illegal to use cannabis in public, and to sell and supply it.
The Cannabis Development Council of South Africa welcomed the ruling, and called on the government to drop charges against people found in possession of the drug.
Jeremy Acton, the leader of the Dagga Party, which campaigns for the use of cannabis, said the ruling should have gone further to legalise the carrying of marijuana in public.
Cannabis is referred to as "dagga" in South Africa.
The court gave parliament 24 months to change the law to reflect its ruling.
Adults who used marijuana in private would be protected by the ruling until the law was amended.
The court did not specify the quantity of cannabis a person can grow or use in private.
Parliament would have to decide on this, it said.
More about: cannabis
