All other secure keys and strings, including BMC and switch passwords, user and administrator account passwords are still manually updated by the administrator.

Important

Starting with Azure Stack's 1811 release, secret rotation has been separated for internal and external certificates.

In order to maintain the integrity of the Azure Stack infrastructure, operators need the ability to periodically rotate their infrastructure's secrets at frequencies that are consistent with their organization's security requirements.

Rotating Secrets with External Certificates from a new Certificate Authority

Azure Stack supports secret rotation with external certificates from a new Certificate Authority (CA) in the following contexts:

Installed Certificate CA

CA to Rotate To

Supported

Azure Stack Versions Supported

From Self-Signed

To Enterprise

Not Supported

From Self-Signed

To Self-Signed

Not Supported

From Self-Signed

To Public*

Supported

1803 & Later

From Enterprise

To Enterprise

Supported so long as customers use the SAME enterprise CA as used at deployment

Alert remediation

When secrets are within 30 days of expiration, the following alerts are generated in the Administrator Portal:

Pending service account password expiration

Pending internal certificate expiration

Pending external certificate expiration

Running secret rotation using the instructions below will remediate these alerts.

Note

Azure Stack environments on pre-1811 versions may see alerts for pending internal certificate or secret expirations.
These alerts are inaccurate and should be ignored without running internal secret rotation.
Inaccurate internal secret expiration alerts are a known issue that is resolved in 1811– internal secrets will not expire unless the environment has been active for two years.

Pre-steps for secret rotation

Important

If secret rotation has already been performed on your Azure Stack environment then you must update the system to version 1811 or later before you execute secret rotation again.
Secret Rotation must be executed by via the Privileged Endpoint and requires Azure Stack Operator credentials.
If your environment Azure Stack Operator(s) do not know whether secret rotation has been run on your environment, update to 1811 before executing secret rotation again.

It is highly recommended that you update your Azure Stack instance to version 1811.

Note

For pre-1811 versions you do not need to rotate secrets to add extension host certificates. You should follow the instructions in the article Prepare for extension host for Azure Stack to add extension host certificates.

Operators may notice alerts open and automatically close during rotation of Azure Stack secrets. This behavior is expected and the alerts can be ignored. Operators can verify the validity of these alerts by running Test-AzureStack. For operators using SCOM to monitor Azure Stack systems, placing a system in maintenance mode will prevent these alerts from reaching their ITSM systems but will continue to alert if the Azure Stack system becomes unreachable.

Notify your users of any maintenance operations. Schedule normal maintenance windows, as much as possible, during non-business hours. Maintenance operations may affect both user workloads and portal operations.

Note

The next steps only apply when rotating Azure Stack external secrets.

Run Test-AzureStack and confirm all test outputs are healthy prior to rotating secrets.

Store a back up to the certificates used for rotation in a secure backup location. If your rotation runs and then fails, replace the certificates in the file share with the backup copies before you rerun the rotation. Note, keep backup copies in the secure backup location.

Create a fileshare you can access from the ERCS VMs. The file share must be readable and writable for the CloudAdmin identity.

Open a PowerShell ISE console from a computer where you have access to the fileshare. Navigate to your fileshare.

As you can see the error massage would indicate that there is a problem accessing your fileshare but in reality it is the folder structure that is being enforced here.
More information can be found in the Microsoft AzureStack Readiness Checker - PublicCertHelper module

It is also important that your fileshare folder structure begins with Certificates folder otherwise it will also fail on validation.
Fileshare mount should look like \\<IPAddress>\<ShareName>\ and it should contain folder Certificates\AAD or Certificates\ADFS inside.

For example:

Fileshare = \\<IPAddress>\<ShareName>\

CertFolder = Certificates\AAD

FullPath = \\<IPAddress>\<ShareName>\Certificates\AAD

Rotating external secrets

To rotate external secrets:

Within the newly created \Certificates\<IdentityProvider> directory created in the Pre-steps, place the new set of replacement external certificates in the directory structure according to the format outlined in the Mandatory Certificates section of the Azure Stack PKI certificate requirements.

Rotating only internal secrets

Note

Internal Secret Rotation should only be done if you suspect an internal secret has been compromised by a malicious entity, or if you have received an alert (on build 1811 or later) indicating internal certificates are nearing expiration.
Azure Stack environments on pre-1811 versions may see alerts for pending internal certificate or secret expirations.
These alerts are inaccurate and should be ignored without running internal secret rotation.
Inaccurate internal secret expiration alerts are a known issue that is resolved in 1811– internal secrets will not expire unless the environment has been active for two years.

Syntax

For external secret rotation

For internal secret rotation

Start-SecretRotation [-Internal]

For external secret rotation rerun

Start-SecretRotation [-ReRun]

For internal secret rotation rerun

Start-SecretRotation [-ReRun] [-Internal]

Description

The Start-SecretRotation cmdlet rotates the infrastructure secrets of an Azure Stack system. By default it rotates only the certificates of all external network infrastructure endpoints. If used with the -Internal flag internal infrastructure secrets will be rotated. When rotating external network infrastructure endpoints, Start-SecretRotation should be run with an Invoke-Command script block with the Azure Stack environment's privileged endpoint session passed in as the Session parameter.

Parameters

Parameter

Type

Required

Position

Default

Description

PfxFilesPath

String

False

Named

None

The fileshare path to the \Certificates directory containing all external network endpoint certificates. Only required when rotating external secrets. End directory must be \Certificates.

CertificatePassword

SecureString

False

Named

None

The password for all certificates provided in the -PfXFilesPath. Required value if PfxFilesPath is provided when external secrets are rotated.

Internal

String

False

Named

None

Internal flag must be used anytime an Azure Stack operator wishes to rotate internal infrastructure secrets.

PathAccessCredential

PSCredential

False

Named

None

The PowerShell credential for the fileshare of the \Certificates directory containing all external network endpoint certificates. Only required when rotating external secrets.

ReRun

SwitchParameter

False

Named

None

ReRun must be used anytime secret rotation is reattempted after a failed attempt.

This command rotates all of the infrastructure secrets exposed to Azure Stack internal network as well as the TLS certificates used for Azure Stack's external network infrastructure endpoints. Start-SecretRotation rotates all stack-generated secrets, and because there are provided certificates, external endpoint certificates will also be rotated.

Update the baseboard management controller (BMC) credential

The baseboard management controller (BMC) monitors the physical state of your servers. The specifications and instructions on updating the user account name and password of the BMC vary based on your original equipment manufacturer (OEM) hardware vendor. You should update your passwords for Azure Stack components on a regular basis.

Update the BMC on the Azure Stack physical servers by following your OEM instructions. The user account name and password for each BMC in your environment must be the same.

After your PowerShell prompt has changed to [IP address or ERCS VM name]: PS> or to [azs-ercs01]: PS>, depending on the environment, run Set-BmcCredential by running Invoke-Command. Pass your privileged endpoint session variable as a parameter. For example: