Menu

Simplifying MBrace Deployment

If you’ve used MBrace then you’ll know that it’s a really powerful tool for quickly performing data science and data engineering related tasks. It’s always had a great integration with Azure, however, it’s typically relied on the use of some legacy Azure services, notably in the form of Cloud Services. In recent weeks at Compositional IT, we’ve been looking at whether we can modernise some of the Azure integration by moving the Cloud Services over to the newer App Service, which simplifies many of the challenges you’re likely to have when developing applications by providing simplified stories for deployment, monitoring and development. It’s also fully supported by the Azure Resource Manager tooling which allows us to configure our Azure services in a declarative manner.

After getting MBrace running as a webjob, we also managed to create an Azure Resource Manager template which allows us to create all of the resources required by an MBrace cluster and then deploy the worker onto the cluster. Currently, it relies on a few PowerShell commands to submit the deployment, but there’s plenty of other options including the Azure CLI and the Azure management libraries. However, there’s also the option of creating a Deploy to Azure button allowing us to deploy an MBrace cluster in as little as 3 clicks.

If you now browse to https://github.com/isaacabraham/mbrace-arm you’ll see a shiny Deploy to Azure button. Click on that and you’ll get directed to the Azure login page. Login to your Azure account (if you don’t have one yet, sign up and you’ll get a $200 free credit for your first month). Create a new resource group (it’ll make it a lot easier to kill the cluster when you’re done) and specify the machine size and count you want. Click next and next again and the deployment starts and then you’ll have an MBrace cluster available for use. During my testing it’s taken about 90 seconds on average to spin up a 10 node MBrace cluster which is really fast.

Unfortunately, the Deploy to Azure website doesn’t list out the outputs of an Azure Resource Manager template, which means we have to go into the Azure portal to retrieve connection strings for Azure Storage and Azure Service Bus, fortunately clicking the Manage your resources link takes you directly to the page in the Azure Portal for our Azure Resource Group. Click there and then grab the connection strings for your Service Bus account and Storage account.

Now open up your editor of choice, here’s mine below, VS Code, write up a quick script to do some processing in MBrace, execute it in F# Interactive and it gets shipped up to your shiny new MBrace cluster hosted in the cloud.