Hello, first your AVI movie needs to be playable in QuickTime Player, thanks to the appropriate codec, if available.
If this step is OK, then you need the free Real Export Plug-in for Mac. QuickTime Pro is not free but you can try to export with iMovie (in the File menu: Share/QuickTime/Expert Settings/Movie to RealMedia).

Yes, not compatible with Tiger, I didn't saw that sorrry.
But AVI is just a container you know, so the video in an AVI file can be compressed in various codecs (like with the MOV container), each with variable compression (quality). Which codec is used in your AVI movie? What is its bitrate (in kbits/s)? Play the movie in QuickTime Player and hit Apple-I to display information.
If the codec is efficient, say DivX or XVID like many AVI files, you won't get a smaller file if you export the movie in RMVB... There th only solution would be to decrease its pixels size (definition). And why RMVB? This is not a leading codec anymore, you can try to export your movie with QuickTime Pro in another codec and/or container : the codec "H.264" in a .mov or .mp4 container offers very good quality and the smallest foot print.