The Olde Pink House started life as a dwelling, with construction having started in 1771. However, it was not completed until 1789. Its founder was James Habersham, a rice planter and Patriot against the British. The house owes its distinctive coloring not to paint, but to the red brick beneath the white stucco. The red brick bled through, discoloring the stucco with its pink pigment. Despite successive owners trying to paint over the discoloration, one finally made the decision to paint the house pink, making it an instantly recognisable Savannah landmark. It is now regarded as one of the finest restaurants in the city.

The restaurant is haunted by three ghosts. The first apparition is of a man in gray uniform, leading some to suggest that it could be none other than James Habersham himself. A second is also a male in uniform seen sitting at the bar. Finally, the third is a servant girl that has been sighted crying on the second floor. Other phenomena include the sounds of coins, dice being thrown and voices. The sounds of crying have been heard and attributed to the distressed servant girl.