The city of Rock Mart, as it was then called, was chartered on August 26, 1872, with Charles Taylor Parker as its first mayor. The town had been surveyed and laid out, by Colonel Seaborne Jones in 1871, along the railroad right of way. He sold lots to merchants and laid out 6 lots for churches, both Black and White, which he sold for $1 to $5. Colonel Jones also donated land for a park and a cemetery, Rose Hill, where he and his family are buried overlooking the city. He and his father built the Commercial Hotel between the railroad and the Euharlee Creek.
The 1880's saw the most productive period for the local slate mines and many of the buildings in town date from this time and are made of the native stone. The city of Rock Mart officially changed its name to Rockmart on December 1, 1924. The population of the city was 1,400 but by 1930 it had more than doubled to 3,264 people due mainly to the opening of Goodyear mill on December 12, 1929.
In 1989 Rockmart was made a Georgia Hometown Community for its efforts toward preservation and revitalization.