The graves are at the North Fulton Golf Course in Chastain Park in the city's Buckhead district.
The operations director of the Chastain Park Conservancy, Ray Mock, said that he always knew there were graves somewhere in the park, but he didn't know an exact location until he discovered an old map showing the cemetery.
"I really didn't have any expectations," Mock told Channel 2's Richard Elliot. "I was almost positive we had a cemetery, but I wasn't sure what we would find."
Mock said Fulton County ran two almshouses, or poorhouses, in the park from 1911 to the 1960s. He guesses that the people buried under the course lived in those poorhouses.
Len Strozier of Omega Mapping Services used ground mapping sonar to locate the graves and marked each one with a small orange flag.
"As we started the survey, and Len was traveling around with his machine, he kept putting flags in the ground, and flags and more flags, and I felt a little excitement," Mock said. "It's like discovering something for the very first time."
Mock said the city built the golf course in the 1930s and must have known the cemetery was there, though virtually all records vanished over the years.
The Chastain Park Conservancy plans to leave the graves undisturbed, but may plant wildflowers there and add an informational sign explaining the old poorhouse and the people buried there.
"When I come to places like this, this is holy ground," Strozier said. "This is a special place. Think of the families who would come here regularly to attend to their family. It really means a lot to me, it really does."
© 2014 CBS Interactive Inc. All Rights Reserved.