BOSTON, March 22 (Reuters) - Goldman Sachs Group Inc Chief Executive Lloyd Blankfein said on Thursday a recent media report speculating about his possible retirement this year was wishful thinking, but conceded it would be difficult to decide when to leave his job.

During a public appearance in Boston, Blankfein, one of the most powerful banking executives in the world, said speculation about his retirement was off the mark.

"The Wall Street reported (my retirement), which is more of a wishful thought than anything," Blankfein said at a luncheon hosted by the Boston College Chief Executives Club. "(The story) wasn't right, but it may not be wrong forever."

During a question-and-answer session with State Street Corp CEO Jay Hooley, Blankfein discussed retirement but declined to specify when he would end his tenure at the bank.

"My wife says, 'Don't quit,' and I think part of that is selfish," the 63-year-old Blankfein joked. "...I'm worried about it, but I do know that it is very hard to leave these jobs. They are hard jobs to get. People have been speculating (about) when I'm leaving (since) the day I got the job."

After the Wall Street Journal story was published earlier this month, Goldman Sachs fueled further speculation about Blankfein's retirement with an announcement that narrowed the pool of candidates to succeed him at the top job.

The bank said Harvey Schwartz would retire, leaving David Solomon as sole president and chief operating officer. Solomon is now the most obvious successor to Blankfein, who has held the top job for 12 years.

"We narrowed down the succession candidates, but said nothing about my tenure," Blankfein noted. "When things are going badly, you can't leave. When things are going well, you don't want to leave."

"And so, almost by definition, you have to leave when you don't want to leave," he said.

"...I don't think I'm destined to leave because I'm finding something else attractive," Blankfein said. "I just think I'm going to have the discipline to leave when I want to stay."

Shares were down 3.5 percent at $252.60 in late trading, against the broader Dow Jones Industrial Average, which dropped more than 700 points, or 2.9 percent.