After graduating from Takigawa Daini High School, Kaji joined Cerezo Osaka in 1998. He made his first league appearance on July 25, 1998 against Avispa Fukuoka. At the club, he found it difficult to break into the first team and was loaned out to J. League Division 2 side Oita Trinita at the start of the 2000 season.

After spending 2 years at Oita, Kaji came back to Division 1 in 2002 to play for F.C. Tokyo. The manager Hiromi Hara made him a regular and assigned a more attacking role. His club won the 2004 J. League Cup by beating Urawa Red Diamonds in the final which ended 0-0 after the extra time. The penalty shoot-out was required and his successful conversion as the last penalty taker for Tokyo finally broke the deadlock.

He gained his first cap for Japan when national coach Zico played him on October 8, 2003 in a friendly against Tunisia. He was rated highly by Zico who made him the first-choice right full-back. He was a member of the Japan team who won the 2004 Asian Cup in China.[2] In a 2005 FIFA Confederations Cup against Brazil, his goal was disallowed by a controversial offside decision.[3] His first goal for the country finally came on August 17, 2005 in a World Cup qualifier against Iran.[4]

He made the final squad for the 2006 FIFA World Cup finals. In a preparation match against Germany just before the tournament, he injured his ankle when tackled by Bastian Schweinsteiger.[5] Zico considered his replacement but decided to keep him. He missed Japan's first game in the tournament against Australia with Yūichi Komano filling in for his place, but he returned to the team and played in the rest of the competition.

After the 2006 World Cup, Zico was replaced by Ivica Osim who also regularly picks him for the national team. He was a member of the Japan team for the 2007 AFC Asian Cup and played all the Japan games in the tournament.