In Kherson in different years, there were two Jewish cemeteries. At the end of the 18th century, the authorities founded a city necropolis in the Zabalka region. The area received its name due to the fact that it was separated from the city by a ravine (gully in Ukrainian).

The cemetery was named Zabalkovskoe (now - the Korabelny district of the city). There were allocated Jewish and Karaite sections. Jewish section was named the Old Jewish cemetery. It can be seen on the city's master plan dated 1872. In the twentieth century, it fell under construction and has not survived.

At the end of the 19th century, Jews constituted 30% of the population of Kherson. One cemetery was not enough for the community. The city authorities on the northern outskirts of the Zabalka district have set aside two large burial sites for members of the Jewish community. The cemetery was named Novo-Jewish. Its area was 63.4 sq. m. The cemetery had a synagogue and a caretaker's house.

Over time, the cemetery in the city began to be called Jewish. In the summer of 2021, the local community installed a plaque at the cemetery, designating it as "Old Jewish cemetery in Kherson" (translated as Old Jewish cemetery in Kherson).

Already in the first third of the twentieth century, the city authorities stated that the New Jewish cemetery had fallen into decay. A 1926 utility report said the cemetery had fallen into disrepair over the past decade, overgrown with trees and shrubs. Many graves were plundered by local residents, who carried away marble and zinc from the burials.

The cemetery suffered from the actions of the occupiers during the Second World War. Since the second half of the twentieth century, the Jewish population of Kherson declined and in the mid-1960s the city authorities closed the Jewish cemetery for burials. Since then, the cemetery has fallen into disrepair. Its territory has decreased by a third. Private cottages have been erected on some of the cemetery lands.

In 2004, the local Jewish community took the cemetery under protection. The territory was put in order, the monuments were partially restored. In 2012, the cemetery suffered from arson. Burned out according to various sources about 700 square meters of territory.

The most notable of the surviving structures in the cemetery is the red granite crypt of the merchant of the first guild, Moisey Rabinovich. Also, the cemetery has preserved traditional matzevahs and gravestones in the form of a chopped-off tree trunk typical for Eastern Europe.