In low- and middle-income countries, poverty and impaired growth prevent children from meeting their cognitive developmental potential. There are few studies investigating these relationships in high-income settings. Participants were 12,536 children born between 2000 and 2002 in the UK and participating in the Millennium Cohort Study (MCS). Short stature was defined as having a height-for-age 2 or more standard deviations below the median (≤ − 2 SDS) at age 3 years. Standardized British Abilities Scales II (BAS II) language measures, used to assess language development at ages 3, 5, 7 and 11 years, were the main outcome assessed. Children with short stature at age 3 years (4.1%) had language development scores that were consistently lower from ages 3 to 11 years (− 0.26 standard deviations (SD) (95% CI − 0.37, − 0.15)). This effect was attenuated but remained significant after adjustment for covariates. Trajectory analysis produced four distinct patterns of language development scores (low-declining, low-improving, average and high). Multinomial logistic regression models showed that children with short stature had a higher risk of being in the low-declining group, relative to the average group (relative risk ratio (RRR) = 2.11 (95% CI 1.51, 2.95)). They were also less likely to be in the high-scoring group (RRR = 0.65 (0.52, 0.82)). Children with short stature at age 3 years who had ‘caught up’ by age 5 years (height-for-age ≥ 2 SDS) did not have significantly different scores from children with persistent short stature, but had a higher probability of being in the high-performing group than children without catch-up growth (RRR = 1.84 (1.11, 3.07)). Short stature at age 3 years was associated with lower language development scores at ages 3 to 11 years in UK children. These associations remained significant after adjustment for socioeconomic, child and parental factors.