that reads from a file into the buffer and returns the number of bytes read. I am reading from .MJPEG that has a 5byte value, say "07939", followed by that many bytes for the jpeg.

The problem is that the JPEG byte size could overflow the buffer. I cannot seem to find a neat solution for the allocation. My goal is to not create a new buffer for every image. I tried a direct ByteBuffer so I could use its array() method to get direct access to the underlying buffer. The ByteBuffer does not expand dynamically.

... however toByteArray() still takes a copy of the result which might be undesirable.
–
Peter LawreyMar 1 '11 at 18:07

You can extend ByteArrayOutputStream to get the buffer directly, it's a defined part of the class, just protected. You would have to synchronize yourself on it (if needed) and remember that this is the full array, including the bytes not yet written. But that's something one could handle if the memory-usage/performance matters. You'll get similar restrictions with other approaches too.
–
BorisMar 1 '11 at 18:15

Allocating a byte[] is so much faster than reading from a file or a
socket, I would be surprised it will make much difference, unless you
have a system where micro-seconds cost money.

The time it takes to read a file of 64 KB is about 10 ms (unless the
file is in memory)

The time it takes to allocate a 64 KB byte[] is about 0.001 ms,
possibly faster.

You can use apache IO's IOBuffer, however this expands very expensively.

You can also use ByteBuffer, the position() will tell you how much data was read.

If you don't know how big the buffer will be and you have a 64-bit JVM you can create a large direct buffer. This will only allocate memory (by page) when used. The upshot is that you can allocate a 1 GB but might only ever use 4 KB if that is all you need. Direct buffer doesn't support array() however, you would have to read from the ByteBuffer using its other methods.

Another solution is to use an AtomicReference<byte[]> the called method can increase the size as required, but if its large enough it would reuse the previous buffer.

The usual way of accomplishing this in a high-level API is either let the user provide an OutputStream and fill it with your data (which can be a ByteArrayOutputStream or something completely different), or have an InputStream as return value, that the user can read to get the data (which will dynamically load the correct parts from the file and stop when finished).