Speckle is always present in Optical Coherence Tomography (OCT) measurements. To a first approximation, the speckle size is determined by the OCT resolution length and the point spread function of the focusing optics in the sample arm. But the speckle size is also affected by the tissue microstructure. We demonstrate this phenomena by performing measurements on optical phantoms with a controlled density of scatterers using time-domain OCT. In the very low density limit, the scatterers are easily identified on the OCT cross-section and, in fact, one can hardly speak of a speckle pattern. The corresponding speckle size is the resolution length axially and the point spread function of the focusing optics transversally. As the number of scatterers increases, a true speckle field appears and the measured speckle size decreases. In the high density limit, the speckle size reaches an asymptotic value that is about 70% of its low-density regime values. In addition to experimental results, theoretical estimates of the limiting speckle size values are presented. Our work contributes to a better understanding of speckle in optical coherence tomography.