Fossett was born in Richmond in November 1780, the son of Mary Hemings and an unknown father, possibly William Fossett, a free carpenter who worked intermittently at Monticello late in the 1770s. Mary Hemings, the oldest child of Elizabeth “Betty” Hemings, was at the time an enslaved domestic servant in the household of Governor Thomas Jefferson. Three months later she and her children were carried off by British troops during Benedict Arnold’s raid on the capital. After Cornwallis’s surrender at Yorktown in October 1781, Fossett and his family were brought back to Monticello, Jefferson’s residence in Albemarle County.
During Jefferson’s absence in France (1784–1789), Fossett lived in Charlottesville with his mother and siblings. She had been hired to a white merchant, Thomas Bell, by whom she had two children. Unable to marry legally, Bell and Mary Hemings, described by her grandson Robert Scott as a “bright mulatto,” lived as husband and wife, an arrangement openly acknowledged by Bell and recognized by the local community. (She was called Mary Bell in some public records and described as Thomas Bell’s “widow” in an insurance policy.) When, in 1792, Mary Hemings Bell asked to be sold to her husband, Jefferson agreed to sell her “with such of her younger children as she chose.” Thomas Bell purchased Mary and his own children by her, but Jefferson retained her older children, twelve-year-old Joseph and nine-year-old Betsy.