Uniform Buffer Objects, dynamic sized arrays and lights

Hello, I'm new to this forum as a poster. I tried making the title as descriptive as possible so here goes:

I'm new to UBOs and I've seen many different implementations and I don't think none of them serves my purpose and/or I can't understand UBOs completely. In my application, I have a number of lights that isn't fixed. I'd like to feed them to the GPU so I can use them in the shader code. Bonus: I'd prefer to have lights stored in the GPU much like the VBOs, where I just need to bind the buffer's ID (GLuint) to use it. The trick is that even if this is possible, you'd have to bind several UBOs, one for each light and bind a certain index to them so that it maps to the array on the shader code. Is there any way to do this?

Though I don't need to specify the data at this point, I do need to specify a size... but I don't know how many lights I will need. This is a run-time value but if I move this code to a per-frame function I'll be creating a new UBO every frame and thus defeating the purpose. Also, I'm not sure what this Buffer Index should be.
After the initialization, I can pretty much do this:

Uniform blocks must have an explicit size specific in the shader. Shader storage blocks however do not; they can be unbounded in size, with the size taken dynamically based on the range of the buffer object bound to the SSBO.

SSBOs are only available in GL 4.3 hardware.

Uniform blocks are probably sufficient for your needs. You can set a maximum hard limit on the number of lights, then simply pass fewer than that as a uniform variable:

So your uniform buffer would always have space for 100 lights, but you would also pass a variable that says how many to use.

In any case, you will always have some kind of maximum size in your buffer object, no matter whether you use UBOs, SSBOs, buffer textures, or some other mechanism. You don't want to be constantly allocating buffers of arbitrary size. You allocate the maximum size, then fill it with whatever you need.

My example was simple, but I will actually require lights to have about 20N in size and I will also need an array with Materials which can also be 20N in size. The reason for having several materials is that I'll have the material index per pixel on a second shader pass that will do the light evaluations on a quad with a render-targeted texture. I will also be stress testing so I didn't really want to have a maximum size for the array. But even if I use your Uniform Block suggestion, do I keep a reference (GLuint) for the Buffer Objects so I can just bind them before drawing without passing the data to the shaders unless a light is modified? What I want is to have an array of structs in the shader which I can bound dynamically per frame. For example, binding the C++ struct lights[i].ubo to the shader variable lights[i] on a per-frame basis after having generated every lights[i].ubo on a initialisation stage by buffering the data on lights[i].position, lights[i].cutoff_angle and so on.

EDIT: Oh I think I'm starting to understand what you can actually do with it. Could I, for example, allocate space for 100 lights, buffer the data into the shader, save the UBO id and then later when one of the lights gets updated I could buffer the data of that light into the shader without buffering all of the lights? I don't want to keep pushing things from the CPU to the GPU if the GPU could already have stored data like the lights... I would prefer to change for example light 33 and light 66 without sending the other lights over to the shader. So what I basically want is an array of light structs in the shader that may have specifically indexed lights being updated on demand.

Maybe I'm reading you wrong, but: glBufferSubData updates a part of the buffer-object. So all you have to do is declare a uniform-block in the shader (with a sufficient-once-for-all number of lights) , create an equally-sized buffer in gl, bind it to the uniform block and - whenever you Change a light-setting - update the part that corresponds to it.
I don't exactly know what you mean by 20N lights though. If the N is meant Alfonse's 100 then maybe a uniform buffer is too large as the overall-size of uniforms is limited. Then you'd have to use textures or Images to store and get-into-the-shader your light-Settings.

N is the size of a float. In the OpenGL description of layout std140 it is said that data is stored in blocks of 4N if there is a vec3 or vec4, 2N if there is a vec2 and N if it's just floats.

Each of my lights has 20N, thats 5 * vec4. So if there's 100 lights, that's 500 vec4s or 2000 floats. Same for materials so I could be passing 4000 floats per frame to the GPU and I really wanted to use the BUS as little as possible if I already have the data there.

For materials, I really just need to collect all of my materials and send them once, unless a new material appears at run-time which is highly unlikely. Then when collecting all the objects, I pair them with a material ID so even if the objects have to be updated a lot (due to changes in one of the scenegraph matrices) the materials won't change so all I really need is the material ID to be passed to the shader on a per-primitive basis. For the lights, might be as you are saying: I allocate 2000 floats in the GPU's memory and I feel it the first time, then whenever a light is updated I send the part of it that needs updating. So if light 33 gets updated I do a glBufferSubData(GL_UNIFORM_BUFFER,33*20*sizeof(GLf loat),20*sizeof(GLfloat),data.data()); and that only sends 20 floats through the BUS to the GPU, while the other lights are already stored in the GPU's memory, right?

So if light 33 gets updated I do a glBufferSubData(GL_UNIFORM_BUFFER,33*20*sizeof(GLf loat),20*sizeof(GLfloat),data.data()); and that only sends 20 floats through the BUS to the GPU, while the other lights are already stored in the GPU's memory, right?

That's the theory. I've to say I'm unsure about a limit of uniforms that reside in a buffer. 2000 floats would - if I remember right - have been too much for my old Laptop lying around (but that one is quite a few years old). There is a limit for the upper number of uniform-floats etc., but - as I said - I'm unsure about if those do apply for uniform-blocks at all. But I'm really no expert on the newer features of gl as I'm mainly working with Features present in 2.1. I've never used nor concerned with ShaderStorage-blocks etc. Maybe they're a more adequate way for your needs: If ShaderStorage-blocks are - as Alfonse stated - not explicitely sized in the shader it is highly probable that no such limits on the number of uniform floats apply to them.

For materials, I really just need to collect all of my materials and send them once, unless a new material appears at run-time which is highly unlikely.

I won't worry so much about how much data gets send. If a buffer or texture Needs to be grown by a fex Bytes once every 20 Frames that's nothing. Just do partial updates. If a buffer Needs to be grown that is one allocation, one buffer copy + the update with new data. Sounds like 2 Million clock-cycles at most...

2000 floats would - if I remember right - have been too much for my old Laptop lying around (but that one is quite a few years old).

If it supported UBOs at all, then it is required to allow individual uniform blocks to contain at least 16KB of data. So 2000 floats is merely half of the minimum capacity; AMD supports 64KB buffers even on my old HD 3300.

I won't worry so much about how much data gets send. If a buffer or texture Needs to be grown by a fex Bytes once every 20 Frames that's nothing. Just do partial updates. If a buffer Needs to be grown that is one allocation, one buffer copy + the update with new data. Sounds like 2 Million clock-cycles at most...

Which explains why the ARB just released an extension who's primary purpose is to make growing the size of a buffer object after initial creation impossible.

No, the ARB have made it abundantly clear that expanding the size of a buffer (or texture) in-situ is not a good idea.

Simply trying to grow it was my first idea before I was rasping my formulation
Creating a new buffer, copying the old and filling it up should be no problem if done once or twice a second - but I can just speak from my not alltoo hardware-demanding perspective.

And: you caught me. If I remember right UBOs were one thing my old lappi could not handle so I decided against using them - at least until I do not feel the need to test my stuff on other hardware regularly. That thing gave up compiling shaders containing for-loops with an animosous "Shader to supported by HW:"<Nothing>

If it supported UBOs at all, then it is required to allow individual uniform blocks to contain at least 16KB of data. So 2000 floats is merely half of the minimum capacity; AMD supports 64KB buffers even on my old HD 3300.

Which explains why the ARB just released an extension who's primary purpose is to make growing the size of a buffer object after initial creation impossible.

No, the ARB have made it abundantly clear that expanding the size of a buffer (or texture) in-situ is not a good idea.

So what do you think of what I said? Allocating a UBO of 100 * 5 * 4 floats, binding the UBO ID to a binding point and block index so that you just need to bind it to use it. Then when an individual light is updated is it possible to update that specific location in the UBO? Using BufferSubData like I said?