(Resolved) Azure Bot Service returning various access errors

Summary of impact

Last week customers reported a variety of Azure Bot Service issues such as:

Internal Server Error dialog on bot load

Inability to connect via FTP

Inability to connect via Kudu

Inability to setup continuous deployment

Files disappeared from the bot editor

Only customers who created a bot with the Azure Bot Service between April 3rd and April 12th, 2017 were affected. Customer experiencing these issues were partially or completely unable to use their bots after creation.

Root cause

Engineers identified as the root cause a change made on April 3rd to the default ARM template that provisions all Azure Bot Services. This change introduced an extra Azure Storage account during provisioning, which, coupled with a misconfiguration, caused the bot to look for the file system in the wrong Storage Account. Two factors contributed to delaying the resolution of the problem:

Other changes were deployed around the same time, causing engineers to investigate in the wrong direction

The ARM change was done by another team, and not all people were notified of the upcoming change

Mitigation

Once the problem was identified, engineers applied a patch on April 12th at approximately 7PM PST to mitigate the issue. With this, the issue did not affect newly created bots anymore. Then on April 13th at around 12PM, the faulty ARM template change was reverted.

Next steps

All customers impacted are encouraged to create a new Azure Service Bot.

We’ve updated our deployment steps to make sure sign-off is obtained from both the ARM team and Bot Framework teams prior to deployments of ARM Templates. This mechanism will prevent issues like this from happening again.