The variables to increase are extra_mem_top, extra_mem_bot
and pool_size. Actually, the ly2dvi script sets these variables
as environment variables which overrides the settings in the
texmf.cnf file, so as long as you use ly2dvi and these settings
are not high enough, I'm afraid you have to edit the settings
in the ly2dvi file instead. Note: for the 1.4 series, this was
introduced in 1.4.13, so if you have 1.4.12 or older it may be
enough to upgrade.

Okay, I start using 1.5 again (manually setting the vertical spacing :-( )

I added an extra zero to each of the three variables (and even tried
also inserting a huge amount of main_memory:

# tex needs lots of memory, more than it gets by default on Debian
non_path_environment = {
'extra_mem_top' : '10000000',
'extra_mem_bottom' : '10000000',
'pool_size' : '2500000',
'main_memory' : '35000000',
}
but it didn't help:
[12]
! TeX capacity exceeded, sorry [main memory size=8350001].
\leftalign #1->\hbox to 0pt{#1\hss }
l.33578 ...tscale height 2.5000 \outputscale }
%
If you really absolutely need more capacity,
you can ask a wizard to enlarge me.
Here is how much of TeX's memory you used:
550 strings out of 20894
7626 string characters out of 2447394
886098 words of memory out of 886099
3530 multiletter control sequences out of 10000+15000
4809 words of font info for 18 fonts, out of 400000 for 1000
14 hyphenation exceptions out of 10000
23i,6n,19p,468b,246s stack positions out of 3000i,100n,1500p,50000b,4000s
Output written on ClaOpII.dvi (12 pages, 1379548 bytes).

Where comes the [main memory size=8350001] from? the number doesn't
change even when I alter the settings in ly2dvi...

Please help!

Btw: The problem occured when trying to typeset the first 85 measures of
the first 20 staves on A4-paper.
I need to typeset 385 measures of 48 staves on A3-paper - so there is
still a LONG way to go!