How can I compress textures on the fly while using TexSubImage?

Due to the nature of my application, most of the textures being loaded aren't packaged with the application itself and are loaded on the fly. Therefore, they can't be compressed before-hand. When I load new textures, I package everything onto atlases with glTexSubImage2D. I've recently found out the amount of textures I have can take up quite a bit of memory and so I'm trying to get "on the fly" texture compression working.

Now what I've tried doing is switching the atlas' internal format to CompressedRgbaS3tcDxt3Ext instead of Rgba. That works fine, it's only after I try subbing the texture into the atlas is when I get a GL_INVALID_OPERATION error.

I've googled the issue and there doesn't seem to be anything out there explaining this situation and how to properly sub uncompressed images into a compressed format.

Can you be more specific about what you mean by "quite a bit of memory" (and how you're measuring it)? It's normal enough for texture memory usage to be high-ish, and you may be worrying over (and putting significant effort into) something that isn't actually a problem at all.

I've been measuring memory usage with GPU-Z by watching the "Memory Usage (Dynamic)" as highlighted here: http://i.imgur.com/yEA5KhW.png (this image was taken at idle without the application running)

I can fully expect the memory usage to go beyond 2gb as there's no limit to how many textures are loaded (I'd expect no more than 4gb, though). I'd like for this application to be 32bit compatible but I was running into GL_OUT_OF_MEMORY errors whenever it tried allocating past 2gb. I got around that by switching to 64bit but, like I said, I'd like to support 32bit. Sure, I can set the /LARGEADDRESSAWARE flag to keep it 32bit and allow up to 4gb of system memory but I feel like I should be compressing the textures anyway if at all possible.

Also, due to the nature of the application, it's not viable to only load textures that are needed at the time because any of the textures could be used at any point.

Now what I've tried doing is switching the atlas' internal format to CompressedRgbaS3tcDxt3Ext instead of Rgba. That works fine, it's only after I try subbing the texture into the atlas is when I get a GL_INVALID_OPERATION error. I've googled the issue and there doesn't seem to be anything out there explaining this situation and how to properly sub uncompressed images into a compressed format.

Apparently I hadn't read far enough into the spec to find the section detailing that the texture dimensions do indeed need to be multiples of 4, which mine are not. Any ideal solutions for converting my textures to multiples of 4?

EDIT: I'm guessing I should pad the textures with enough opaque black pixels to convert the dimensions to multiples of 4. But then how would that be accomplished? Is there some OpenGL magic or would I be better off padding the byte array myself?

You could probably just initialize the textures dimensions with glTexImage but without uploading any texel data, and then use glTexSubImage to upload the texel data you got. All texels that are not covered by glTexSubImage will remain the default color, which is 0,0,0,0 if I remember correctly.

EDIT: I'm guessing I should pad the textures with enough opaque black pixels to convert the dimensions to multiples of 4. But then how would that be accomplished?

From the extension spec, I think this may answer your question:

Originally Posted by EXT_texture_compression_s3tc

If the internal format of the texture image being modified is [DXT1, DXT3, or DXT5], the
texture is stored using one of the several S3TC compressed texture image
formats. Such images are easily edited along 4x4 texel boundaries, so the
limitations on TexSubImage2D or CopyTexSubImage2D parameters are relaxed.
TexSubImage2D and CopyTexSubImage2D will result in an INVALID_OPERATION
error only if one of the following conditions occurs:

* <width> is not a multiple of four, <width> plus <xoffset> is not
equal to TEXTURE_WIDTH, and either <xoffset> or <yoffset> is
non-zero;

* <height> is not a multiple of four, <height> plus <yoffset> is not
equal to TEXTURE_HEIGHT, and either <xoffset> or <yoffset> is
non-zero; or

* <xoffset> or <yoffset> is not a multiple of four.

So what it sounds like is, while xoffset and yoffset must be a multiple of 4, the width and/or height of the subload can be less than 4 for that last row and/or column of 4x4 texel blocks, if your texture doesn't use all texels in that 4x4 block (e.g. a 6x6 MIP level, or a 2x2 MIP level).

OpenGL still throws an invalid operation if the image dimensions aren't a multiple of 4, even if the image offset is a multiple of 4 (or in my test case: 0,0). It's only after I manually edit the image and extend the dimensions to fit the multiple of 4 that the image loads and draws without any error. It doesn't seem like I can get away without doing some pre-processing to the image before copying it to the atlas.