Fossett was born in Richmond in November 1780, the son of Mary Hemings and an unknown father, possibly William Fossett, a free carpenter who worked intermittently at Monticello late in the 1770s. Mary Hemings, the oldest child of Elizabeth “Betty” Hemings, was at the time an enslaved domestic servant in the household of Governor Thomas Jefferson. Three months later she and her children were carried off by British troops during Benedict Arnold’s raid on the capital. After Cornwallis’s surrender at Yorktown in October 1781, Fossett and his family were brought back to Monticello, Jefferson’s residence in Albemarle County.
During Jefferson’s absence in France (1784–1789), Fossett lived in Charlottesville with his mother and siblings. She had been hired to a white merchant, Thomas Bell, by whom she had two children. Unable to marry legally, Bell and Mary Hemings, described by her grandson Robert Scott as a “bright mulatto,” lived as husband and wife, an arrangement open