Thursday, May 4, 2017

OpenSSL constant hell

Last week, someone opened a bug for opmsg, saying itwont compile with recent versions of gcc.I am happy that I've been noticed about it, since it showspeople are using opmsg. The "bug" however is not withinopmsg, its about the way OpenSSL changes their functiondefinitions (breaking the API!) in between minor versionsof OpenSSL.What exactly happened?In above screendump you see two functions constp() andconstpp(). They will serve as an easy, down-stripped demo.While with constp() the constchar * declarationby the programmer is more like a promise to the callerthat the data at which ptr points to, will not be written to,this is different when the type changes to a doublepointer. You can pass char * variables to constp() withoutany problem, because constp() just promises you to not modifythe pointee data. It would be allowed to do so for char *variables, but it confines itself to this promise, whichis good practise to show the user of the API "Hey, wewont modify your data as you pass it to us." There are no problems if you would change a foo(char *)declaration to a foo(const char *) because the later declaration just promises more to you, and you lose nothing by foo()making additional promises to you.The takeaway is: you can pass a char * variable to afunction that wants a const char *. Both foo()functions are accepting the same type. You can seethis by the compiler accepting the call to constp()for a char * variable.Thats different when a double pointer is involved aswith the constpp() function. Here you have to passa const char ** because otherwise the pointer value itself couldbe assigned a char * by which const objects could be modified.This would violate the const correctness of the programand obsolete the const keyword. Thats why the foo(char **)and foo(const char **) arguments are really different types.In other words, you cant just add a const keyword todouble pointer function arguments as you could do itwith single pointers. You end up having entirely differentfunction signatures.

Now, guess what OpenSSL has made with theDH_get0_key(const DH *dh, BIGNUM **pub_key, BIGNUM **priv_key);andRSA_get0_key(const RSA *r, BIGNUM **n, BIGNUM **e, BIGNUM **d);functions? They "just" added a const to the BIGNUMdouble pointers somewhere after the 1.1.0 versionalready introduced a new API. Thats a warning forC11 programs (one that you should not ignore) but evenworse, as you can see in above demo, its an error forC++11 programs.So, I had to add a wrapper function for the functionsin question which call the right functions, dependingon the OpenSSL version.If you want to read more about the double pointerconst topic, its described here.