What exactly do you mean by creating your own instruction sets? If you want to define extensions, that's possible if you implement your own OpenGL implementation (e.g. add your extensions to an open source implementation like Mesa or if you can add it without changing the existing functions and "inject" the new functions in a LD_PRELOAD kind of style). But this wouldn't be very common.
So please tell us what you have in mind exactly?