Personality assessment helps us to predict how people behave under various circumstances or how well a person might perform within certain roles. However, there are reasons to question the supposed ‘construct validity’ of tests designed to assess various personality attributes including dispositional traits. To demonstrate this, the paper first discusses a realist account of test validity where validity requires that both the attribute exist and that changes in the attribute are causally related to changes in test scores. The paper demonstrates that the validity for tests of dispositional traits is questionable given conceptual problems with traits existing as within-person attributes capable of causing changes in test scores. The widespread reliance on Likert-style response formats is then discussed in relation to the assumed quantitative structure of personality attributes. Based on a realist view of measurement, the uncritical adoption of a representational theory of measurement within personality research means that the validity of all personality tests claiming to ‘measure’ personality attributes is questionable. Suggestions for addressing test validity in personality assessment are then discussed in terms of paying greater critical attention to personality theory itself and adopting a realist theory of assessment and measurement.