The cosmological constant problem arises at the intersection between general relativity and quantum field theory, and is regarded as a fundamental problem in modern physics. In this paper we describe the historical and conceptual origin of the cosmological constant problem which is intimately connected to the vacuum concept in quantum field theory. We critically discuss how the problem rests on the notion of physically real vacuum energy, and which relations between general relativity and quantum field theory are assumed in order to make the problem well-defined.