I am looking for a Mac OS utility that will allow me to use the Compose key sequences known from Linux. Using the Option key is not an alternative as I need it to be consistent with my existing Linux installation.

The Cocoa text system is one way of implementing the compose key on Mac OS X. The other way is using a dedicated keyboard layout, for instance my U.S. custom keyboard layout (I haven't seen another implementation yet).

When the compose key is implemented with a keyboard layout, it will work all over the OS in any application, not just in Cocoa applications. It can be switched on and off with the UI by selecting another keyboard layout and there is a visual feedback while typing the compose combination.

You don't need a special utility. The Cocoa text system is very configurable, and you can set up key combinations and/or key sequences to perform various actions, including inserting special characters as in compose. See this excellent document.

Of course, you're going to have to spend a bit of time converting the sequences from your Linux compose file to Cocoa dictionary format. A script could probably help here.

DoubleCommand is software for Mac OS X (a kernel extension) that lets you remap keys, in other words change the way your keyboard works. Often used to make a PC keyboard more comfortable with a Mac, swapping the Alt (Option) and Windows (Command or Apple) keys, since they are in swapped positions on Mac and PC keyboards. DoubleCommand is a quick and easy way to fix this if you use a PC keyboard with your Mac.