Peter Vivian Daniel was born on April 17, 1818, in Henrico County at the country farm of his parents, Peter V. Daniel, then a member of the Council of State and later an associate justice of the Supreme Court of the United States, and his first wife, Lucy Randolph Daniel. His maternal grandfather was Edmund Randolph, a former governor and attorney general of the United States. While Daniel was growing up, his cousin Raleigh Travers Daniel, who became a member of the Council of State and attorney general of Virginia, lived in the Daniel household and studied law with Daniel’s father.