Least Privilege for SQLPS via SQLAgent Jobs

Here's a two-liner that aborts when I run it as a SQL Agent job in SSMS:

$hostname=$env:computername
set-location "sqlserver:\sql\$hostname"

Here's the info from the job history:

A job step received an error at line 2 in a PowerShell script.
The corresponding line is 'set-location "sqlserver:\sql\$hostname"'.
Correct the script and reschedule the job.
The error information returned by PowerShell is: 'Cannot find path "SQLSERVER:\sql\C27TWSQLCRAN01v" because it does not exist.'

I've found that if the account used to run the job has admin-level privileges, as is well, but when I remove the account from the admin group, I get the abort.