With an 8K buffer it only took 0.01 seconds each of user and system time, but using -bufsize 1 took 0.02 and 0.03 seconds respectively, a total 5x increase. The CPU percentage reported value also increased by 20. If that's not an issue for you then by all means use a bufsize of 1. But if it is, then you'll need to benchmark it to find the best size for your application.

unbuffer and similar solutions don't work because the latency is not due to stdio buffering, it's due to the way the openssl program processes its data.

openssl base64, openssl enc and openssl dec have a default buffer size of 8kB, which can be specified with the -bufsize option. When the input or output is base64, the buffer size goes down no further than 80 bytes because openssl wants to be able to process a full base64 line at a time.