There are lots. Just head over to http://exotica.org.uk and look at all the custom formats. David Whittaker, Jochen Hipple and David "uncleArt" Lowe to name but a few.

Custom format does not imply that music is directly coded without a music program. AFAIK it usually simply means that the format is rare and/or not meant for wider public. For example a custom format may be designed just for the purposes of one artist or scenegroup or game designing team.