CANBERRA, Australia—Australia’s government came under fresh pressure Wednesday to reject Japanese submarines as the spearhead for a new undersea fleet, with one of the country’s top security think-tanks urging consideration of a homegrown naval deterrent.

As newly appointed Defense Minister Kevin Andrews prepared to inspect government-owned shipbuilder ASC this week ahead of a decision on whether to hold a tender competition for a A$25 billion fleet of new conventional submarines, the Australian Strategic Policy Institute...