You said something about using vec4 to combine textures... This sounds odd to me, I'm not sure what you mean by this.

vec4 has four components and a texture typically also has four components. vec4(Red, Green, Blue, Alpha). One vec4 is one texture.

So far as the number of textures you can use.... I would say that 8 textures will probably work on most machines, maybe not well.... There will likely be a few here and there that won't like this at all.

You can also look up 'texture atlas' which is a technique for packing many UV coordinates into just one texture.

You said something about using vec4 to combine textures... This sounds odd to me, I'm not sure what you mean by this.

I assume that he's referring to texture splatting (aka splat mapping), where you have up to four terrain textures then another texture which is effectively 4 alpha channels, one for each of the terrain textures. Most of the time, one channel will be at 1.0 to select a specific terrain texture and the other three at zero; at boundaries between terrain types, two channels will have intermediate values to blend between two textures, with the other two at zero.

It would be possible to have another 4 terrain textures and another alpha texture (OpenGL 3.x guarantees at least 16 texture image units per shader), but getting to be expensive in terms of both memory consumption and bandwidth and processing time. Given that you're unlikely to want to blend more than two textures at any given point, you'd probably be better off with e.g. a GL_RG8UI texture holding the indices of a primary and secondary texture for each texel plus a GL_R8 texture to specify the blend factor between the two. That supports up to 256 terrain textures with less memory consumption for the "map" texture than the 4-alpha-channels approach (which was originally designed for hardware which lacked array textures, so you couldn't choose which textures to blend, only vary their blend factors).

You'd need another alpha map for each set of 4 terrain textures, but the only hard limit is the number of texture units supported by the implementation. OpenGL 3.x guarantees at least 16 texture units, so 1 base texture, 3 alpha maps and 12 terrain textures is possible.

However, it's inefficient to read many textures when you're typically only using 1 or 2 at any given point, so you'd probably want to use a different approach with more than 4 textures (or even just with 4 textures).