The truly sad thing about this is that he’s talking about his “girlfriend,” not “wife.” I thought it was only within marriage that sex became a deadening chore for women… I thought only married women preferred sanctioning the infidelity of their partners to frequent sex with said partners. Have the comics led me astray all these years?

It took me a couple of reads after reading the comments, but I think the joke is this:
The girlfriend recommended the swingers club, obviously for both of them, because apparently her needs were not being met – and the stereotypical male decides she must be thinking of only his needs and he goes by himself while she’s at the book club. And while the punchline at the expense of the self-centered male is slightly amusing it still isn’t that ‘funny’.