But councillors supported a recommendation by planning officers to approve the plans at a meeting on Tuesday.
Speaking in support, Cllr Roger Harris said: “We have been made aware that there are 700 plus households in Abergavenny that are desperate for affordable housing.
“This site is going to be run by a housing association, 100 per cent affordable.”
Cllr Mat Feakins said there is “a significant need” in Monmouthshire for affordable housing.
“There are not many as sustainable locations as Abergavenny, which this site can provide,” he added.
Six two-bedroom houses and six three-bedroom houses are proposed either side of a new entrance to the site.