O.C. 'Housewife' -- Ordered to Pay Ex-BF $40,000

"Real Housewives of O.C." star Gretchen Rossi's ongoing defamation suit against her ex-BF just BACKFIRED royally,TMZ has learned, after a judge ruled SHE has to pay HIS lawyer fees.

According to documents filed in Orange County Superior Court -- the judge ordered Rossi to fork over $40,799.38 in attorney's fees for ex boyfriend Jay Photoglou ... after several of the claims she made in her lawsuit were dismissed.

As we previously reported -- Rossi and Photoglou have been embroiled in a protracted court battle ... after she sued him for defamation, claiming he trashed her in the media.

Jay's attorney, Bryan Freedman, tells us Gretchen has until next week to pay up ... or else.