Now, new problem in CoM frame is:|dw:1356887265577:dw|
Do you agree?
Now, working out KE of system is easy. When spring is fully compressed, both masses will be at rest and KE will be zero. PE will lead you to compression of spring.

Velocities in CoM frame are:
\(v^*_{M}=v_o-v(CoM)=v_o-2v_o/3=+v_o/3\)
\(v^*_{2M}=v_o/2-v(CoM)=v_o/2-2v_o/3=-v_o/6\)
which means that:
M-block is moving to the right with speed \(v_o/3\)
2M-block is moving to the left with speed \(v_o/6\)

I do not know where you are going wrong, because I do not understand what laws you are following or in what frame of reference you are working.
If you understand my drawing above and what I wrote:
"Now, working out KE of system is easy. When spring is fully compressed, both masses will be at rest and KE will be zero. PE will lead you to compression of spring."
You will get:
\(KE\,^*=\Large\frac{1}{2}\normalsize M \Large(\frac{v_o}{3})^2 \normalsize + \Large\frac{1}{2}\normalsize (2M) \Large(\frac{v_o}{6})^2 \normalsize = \Large\frac{1}{12}\normalsize M v_o^2\)
Then \(PE=\Large\frac{1}{12}\normalsize M v_o^2=\Large\frac{1}{2}\normalsize k \;\Delta l ^2\)
will lead to \(\Delta l\) = 5 cm

maybe,,here is where i went wrong,
both blocks during max compression will have some common velocity,
3(1) + 6(1/2) = 3v + 6v
2/3 =v
so consn of energy will modify like this :
1/2(3)(1^2) + 1/2(6) (1/4) = 1/2(3+6)(4/9)+ 1/2(200) x^2
ohh!
i got 5 cms! \m/ !

i'll try to explain my approach now :
first let us consider the inelastic collision between the 2 blocks,
applying COLM ,
3(1) = (3+3)v
so v=1/2
now we are reduced to the question
|dw:1356966700681:dw|
and we need to find max compression in this case