Humans have a pretty good sense of
scale, or reasonable ranges of these
numeric attributes, of different objects, but do pre-trained language
representations? Although pre-trained Language Models (LMs) like
BERT have
shown a remarkable ability to learn all kinds of knowledge, including
factual
knowledge,
it remains unclear whether their representations can capture these types
of numeric attributes from text alone without explicit training data.
In our recent
paper,
we measure the amount of scale information that is captured in several
kinds of pre-trained text representations and show that, although
generally a
significant amount of such information is captured, there is