Hodges was born in May 1819 in Princess Anne County and was the son of Charles Augustus Hodges and his second wife, Julia Nelson Willis Hodges, both free African Americans of mixed-race ancestry. His middle name may have been Edward. The Hodges family was one of the more prosperous of the numerous free black families in Princess Anne County (later the city of Virginia Beach). Hodges’s father purchased three farms and his own father’s freedom, and he also arranged for private tutors to teach his children how to read and write. Hodges’s older brother Willis A. Hodges served in the Convention of 1867–1868, his nephew John Q. Hodges served in the House of Delegates, and his older brother William Johnson Hodges had a long career in New York as an antislavery activist and a short career in Virginia as an African American political leader after the Civil War.