Berkeley was born in the parish of Saint George’s Hanover Square, London, England, where his parents, John Symes Berkeley and his second wife, Elizabeth Norborne Berkeley, resided while Berkeley attended the House of Commons as a Tory member for Gloucestershire. Their only other child, Elizabeth, married Lord Charles Noel Somerset, who became the fourth duke of Beaufort. From 1756 until 1765 Norborne Berkeley served as guardian of his sister’s son, who in the latter year became the fifth duke. Berkeley never married but provided handsomely and obtained a commission in the Royal Navy for a son who became Vice Admiral Sir Charles Thompson, baronet (ca. 1740–1799).