Akihabara has changed a LOT over the years. At one time it was a very nice place for DIY audio-builders, but many of the good shops have since closed down or shifted to other types of products. Most of the district now caters to computers, game computers, digital cameras, and the ever-present cellular phones. But it is still a fun place to visit.

As a one-stop source for semiconductors, I suggest Wakamatsu, because they have a decent, well-rounded selection overall. Other shops may be better (and cheaper) for specific device manufacturers, but Wakamatsu is the most convenient. Wakamatsu also carries audio-grade capacitors (Elna, BlackGate, Sanyo OsCon, Nichicon Muse), and resistors (Kiwames and so on).

Kaijin Musen doesn't carry semiconductors, but their selection of passive components is better than that of Wakamatsu (the Mogami RCA chassis jacks are good and cheap).

For speaker units you could try Hino Musen or Koizumi Musen.

Perhaps it is an idea to pick up the latest issue of M&J (formerly Musen to Jikken) and pore over the advertisements for suitable shops before you leave for Japan.