No, this does not work in Mellel, and probably never will. Mellel does not use OS X's text engine or controls, so thsoe features have to be implemented separately and from-scratch to be included in Mellel. As far as the disctionary popup, I hope the Redlers spend their time working on features that users have been asking for/waiting on for YEARS rather than something minor as the dictionary popup.

BTW: There was a request for the feature in the forums dated 29 Jan 2009 that includes a workaround that approximates the floating dictionary window by calling the Dictionary app and opening it with the curretnly highlighted word in Mellel—perhaps that would suffice for your needs.

(I remembered there was a previous discussion, I just didn't realize how long ago it was made: that thread is dated over 7 years ago!)