Formal Metadata

CC Attribution - NonCommercial - ShareAlike 3.0 Unported:You are free to use, adapt and copy, distribute and transmit the work or content in adapted or unchanged form for any legal and non-commercial purpose as long as the work is attributed to the author in the manner specified by the author or licensor and the work or content is shared also in adapted form only under the conditions of this license.

Content Metadata

Infrastructure as Python Code: Run your Services on Microsoft Azure [EuroPython 2017 - Talk - 2017-07-11 - Anfiteatro 1] [Rimini, Italy] Using Infrastructure-as-Code principles with configuration through machine processable definition files in combination with the adoption of cloud computing provides faster feedback cycles in development/testing and less risk in deployment to production. The Microsoft Azure Cloud (https://azure.microsoft.com/) allows different ways to provision, deploy and run your python service: The Azure Resource Manger Templates (https://azure.microsoft.com/en-us/resources/templates/) allows you to provision your application using a declarative template. With parameters, variables and Azure template functions, the same template can be used to deploy your application in different stages (dev, test, production) and environments for different customers. We open sourced the tropo library (https://pypi.python.org/pypi/tropo/) to create Azure Resource Templates from python. Azure SDK for Python (http://azure-sdk-for-python.readthedocs.io) for a low level access to manage resources in the Azure Cloud. An Azure Ansible Module (https://docs.ansible.com/ansible/guide azure.html) based on the Azure SDK to automate software provisioning, configuration management, and application deployment in a single environment. Each of the alternatives has different strengths and drawbacks. Presenting our learnings from migrating our infrastructure into the Azrue Cloud will help to avoid common pitfalls and show deployment patterns that will ease the live of devops

OK so common instead of 1 and I most of the developed 3 under we develop machine learning algorithms for retail so for example if we calculate optimized price decisions for online retailers and all we calculate future demand for retailers and implemented replenishment on top of

00:25

it so that the source of the retailer's my The goal of stock or have too much waste so that's what thing we do is with

00:33

machine learning and most of us that's Python and you can contact me we to at all on conference on associative and if you have any questions and the slides will be available and get up and our company account uh I show the link to the slides again at the end of the presentation so that you can take your notes

00:54

so maybe before we start with things Microsoft is still really cool company that knows about source embraces Python and it's really fun to repress OK nobody at 1 1 1 hand so probably some that's the same way I thought I think on open source guy for all my life and we have been self-hosting all infrastructure we and of all the time the 1st 3 digit number of cells and sometimes policy dual came to us and said OK we want to move to the cloud and we started at the every rating AWS and everything the final result OK let's go to AWS it's a cool cloud provider every coal companies going to AWS and so on and then some months later he came again and said OK now we're going to the cloud but it will be Microsoft measure so at 1st I thought 0 no no I have to add Mr. window so all I have to learn PowerShell and they have all I lose all my open-source knowledge of all the stuff and put it to a lot of to be quite different from what I've learned a lot in the last half year we have started to migrate all of our infrastructure 2 2 2 as a we following a shift and lived approach so in the 1st place you just using the basic resources of Microsoft as 0 that storage networking and computation of virtual machines and once we have done as the initial immigration but we're moving up the stake and using a much more sophisticated services like manage database services HDFS services or even the diminishes steak so I have quite learned a lot about Microsoft in the last half year I think they've changed a lot from probably they are still not of the coolest open-source company but the really embraced by a treat tiden as a 1st class citizens in the and I'll show you in the next 20 or 30 minutes a little bit how we employed to agile and would be flown from all deployments so what

03:03

is Microsoft as Microsoft of as of cloud infrastructure provider from from MS and it's basically Infrastructure-as-a-Service so you give the mining and you get infrastructure um that's a little bit the same results guys guys who gets the money and they build as the infrastructure of but the tone on cycles all steam was about I don't know to choose 3 months and you the get you have to buy the so as you have to put them in the red you have to put the cables in there and get all the software on it and it so have really long cycles if you just want to try something all want to scale up and infrastructure the service and you probably have an API and you can get so lost this 1 click and get lots of them and to buy and once you are done this testing thing out you control them away so that you are much faster arm and the which have really helps you to grow as a company I you see here the as that sport that's there you why are you can represent as you can click so was little machines all this stuff but I'm once you got beyond the initial phase of this destroying stuff and you don't want to deploy infrastructure we you i and you want to use tools in the best way of decorative declaration of the infrastructure that you always can deployed again and poultry can deploy the same version of the infrastructure in the testing environment and so you want to all to to automate all this stuff so now I'm going to show you how we automated and how we learn what we learned about the tools of Microsoft for whites on before we start

04:49

I'll tell you a little bit about the architecture of Microsoft as a so the basic concept in as a resource and resource providers for example little machines are resources and the resource providers takes care that you can call API and did you get the resources that you request so for each different resources that you have a different resource providers so for the 2 machines computation storage to higher level services and and they all fulfill the resource provider contract that's a standard API you can inject was as a and and on top of the resource manager you have different tools like portal like a CLI Command mall pipelined library or even the plane rest calls of how you can provision your infrastructure in as a

05:48

come mostly if users 3 different deployment options around 1 of the we don't use the much of partial on the next 1 is on thin plates that's a decorative deployment options and then you have the time of the REST API Microsoft ships you of the bicycle and libraries and you can talk to them we have the Python API all we live in a set command line client albeit in the book and really my thing nice thing is that they use us mega API definitions and Jason ski for the content the of the API as well for the payload so some beside and they support of our shot and I think PHP and they all generate the client libraries from the same it's made of sauce definitions so you will always have the same version of the of client libraries in the different programming languages that's really nice because this approach applies is always up to date and on par with the impartial library 1 of the basic concepts

06:59

of Microsoft measure of the resource groups you can you can group use resources into a resource pool and you should do this so that you could of course put your whole infrastructure in 1 resource pool but it's better to put your resources in different resource groups based on the deployment lifecycle because normally will always deploy a whole resource pool playing a complete mode and and you want to note if you just want for example job data storage accounts you don't want to to update the whole infrastructure but just a resource groups that you interested in and on the left side you see a sample a definition of that are so what that's of IPI compatible internal repository repository reuse so we have an availability set in a local and so on then we have some network interfaces again storage a little machines and that defines the also this internally and you could can use this resource school all the definition of the resources in the school to deploy a cell this introduction in development and testing come from what's important and what's really complex topic in as a is there will be a success are control so for each operation you can do we the command line ordered the portal and you can have a world this success control so you come into 1st created the a can't you kind of super use and just usually the best linear fit this account and start adding other accounts and drop privileges and give the deployment accounts just the privileges they need to deploy their resources groups and you could even say some users are only allowed to view the resource troops to see what's what's inside the infrastructures and others are able to deploy it and that you should really take care of this because it helps to prevent errors and that's for example what happened to me I just wanted in the beginning of fall migration project wanted to deploy or a resource group and is that that there was still a social founded so I deleted it and all the data was gone and in principle I should have been able to do this because this daughter con was not my business it was not my service but we want to that far was world basic system and management of and so I could alleviate and even if I haven't I shouldn't have to do it so what we are using them as the primary deployment option is contemplates arm

09:40

contemplates are declarative text based description of the desired deployment state and its adjacent document and to submit it should be as a resource manager and the resource manager and takes care of the appellate provision of the resources the World Bank and the deployment and the but not shown you the simple command line interface or you can deploy a resource group you always have to tell as a sum which resource to be 1 to to deploy the name and the template of the defined all that is the stuff now and can deploy a template to different regions and different services and and you can define multiple resources in 1 template but each has a unique resource only can live and 1 template and you have 2 different deployment options that's always either completed more incremental if you don't incremental and deployment of there's as a resource manager will only add new resources that are in the new template but will not delete stuff and that's OK for trying things awkward normal where you want to to the complete and deployment where as a resource manager also takes care that it did resources that are not defined in new template because you always want to have the decorative status is in your templates to be the 1 that is deployed in Asia so

11:09

what does a minimal template nuclides and you always have to link to adjacent schema of the template and you have to give the content versions and then you can specify resources parameters variables and outputs and Microsoft term open-source oldest Jason schema definitions for the different templates and you can go to be tough and really see what what kinds of values are allowed a new template for which resources so on let's define a simple Storage account that's a single resource or and again you have to tell the resource Manager which type you want to deploy in this in this case it's much of storage starts account and you have to give him the API you want to talk to so for some 1 resource type they always different API Russians you can use with different parame and you have to tell in which region of West Europe US AGR you want to deploy and your resource and then some specifics about resource in this case it's just the type of storage account and want to deploy a locally redundant no from 1 thing we learned to use the woman and use very extensively is taking off resources so for each resource that you can apply for a number of Texas and up to 250 and you can later on use the textual grouping for example in building 1 monitoring and 1 0 4 are you going to have a larger infrastructure was I don't know 3 digit number also was and it really helps to see which so this is responsible for which this which costs or in monitoring and to see that so fails so on my from the beginning to think about it taking scheme and really apply text all your and consensus Chung and they aren't template is not simply a chastened on template or chastened file but you could use this in the chosen file and a value parts and you can't use the on template functions so each time you use the bracket notation you basically call a function and during deployment to dissolve dysfunction will be readily we replaced in the template for the role of art for example from the storage accounts and as are they share 1 big namespace wall customers so probably if you take the name test for a storage account you can't deployed because it's already used by some other guy and you can't use this unique string functions and together with a group by uh resource provide the 100 generate you a unique ID and that only you use and with this you have the unique name in the global namespace and another 1 is you know don't you already specified in the command line client to reach region you want to deploy and was the long arm with the resolution that resource Group locations you can get this value and don't have to type it again and again and now you also can use this template to deployed in different regions without changing the value of some of the location and you have lots of different functions at hand you have a right functions they can use the 1st of the last value of an array you can get an index of an array the length of an array of numeric functions such drew basic calculations and you have some staying functions so that you can use in the templates we on another pattern you can use in your templates is the use of variables so as soon as you need to use 1 of them 1 variable more than 1 single template it can define a variable is a storage account and then you can use its template in this case the defining storage account name and then could can use the variable in the uh in the computations to get to Texas torture counter to a server the and the 3rd thing you can use in in templates on the outputs

15:39

are so for each variable you generate inside the played you can define an outputs so that once you run on template on edges you can see what the actual value once assimilated

15:52

using the template to use some of resource template and in different stages in our example it so we always have the test area than a staging area and then the production area and you want to inject into your template external values and you can do this parameters so you define the parameters you want to use in your India templated you can use it in your resources and once and you want to deploy it and you can specify an additional template empowerment file that you actually provide the real value so you have 1 template and different parameters fire and with this you can deployed in different regions on different as staging areas of for fuel and infrastructure

16:42

Woodruff the fastest some don't put sensitive data into templates on so much of the measure provides a waste of time to inject sensitive data into your templates is out of them having than in your own gets repository and in plain text so they have to secure string and secure option type of objects and they have from time retrievals was in the template function for the acute type objects and you can also reference to key value secrets so you can imagine that generate but if you don't want the secrets and then you can use it into a new templates and in production always turn off debugging and lobbying because it could also done knowledge of your secrets from full

17:31

rise of simple deployments in the as a resource is is pretty OK but for complex once it's really really fast gets out of hand because it's not just adjacent fiber you define your own resources but you have also contain versioning for different resources you have parameters and variables inline template expression language you can also link temporally templates together so uh it's pretty fast it's pretty hard to to edit all the templates by hand so we have a pretty soon come to Visual Studio which supports the resource templates and talks and you have all 2 intelligence as intelligence and highlighting and it makes it much easier to really edit these templates and therefore also tried to put some Python libraries around the template and generate the stuff but that didn't work out that well so at the moment we still edit the templates by hand but is a powerful tool like Visual Studio

18:40

so how do you actually talk to the to the REST API for Microsoft as a as a problem much of what the command line interface and the command line interface version 1 was building in

18:54

yes but some estimate was of guys told us that didn't work that well because it was a command line client that want to target to the Linux arguments and you from the open source community and the node yes clients just didn't behave like the tools like expected in in the cell like Linux so also the community so they developed the class silly version 2 and they were developed it in Python that's a really nice command line interfaces miss onto to completion and the nice documentation and different amount but from and it has some support for searching you as part part and it's also fully generated from this figure definitions also REST API so they always up to date with this on command line client and it really helps you to traverse this arrested PI and the other example

19:51

you can tell the the AC command line client to lose you all the storage accounts inside the resource group and then you get back chasing document is all the information and that's fine if you want to use it to to preprocessors in Python or some language language that understands that Jason or even piping into J Q and then selection is just some values and but you can also use different output formats so for example you can dump it as a table of the command line interface uses the tabulate library that's a common law library and Python to dump activity of formats or you can dump it as a tab-separated values and then you call your walks period of time to to get the data out of it so it really fits well into the command line Chen Linux manifested earlier can also use gems parts of the Korean language so for example here I just want to list all the storage accounts that are stand out the other ideas from from the name and you just want to have the name and end point of the book so that's really nice to inject was used as an API

21:10

muse and the deal exclusively in configuration management to a provision and deploy also want services and and we also from said it was a good way to use the tube original stuff on as a because was designed that somebody you don't have to have topic at and configurations for example we have finetly Granger's almost names and you have to be defined them in the as a template and we also had defined them in our and the was scripts so and there's certain measure around module 4 4 and the blue and we are using this and to inject it was as a whole and the book so there are 3 ways how we use it and use it to deploy down blades is and the book that is the possibility the possibility to generate the resources directly you the the rest API and we use it as a dynamic inventory script and to bridge the cell and services into all other ends of the scripts we as simple as a deployment of is pretty easy it's not that different to the mount command line client so at the bottom you define few resource templates you can also define some pollinators that you want to use in the resource template and detailed as a rich resource group you want to deploy in which the location and you have the same deployment modes as I said earlier the complete and incremental deployment mode so that's pretty in the same than the command line client but what it really helps us using the power tools from formalism and the ball that we can use the same parameters that we use elsewhere and instead of writing a template file you could also use the measure of our mode for and the to specify the resources that in line and the wall and to smokes side OK for all for simple resources like computation and storage and and that hurricane but for the most sophisticated ones there's still no support of from 4 financeable for the measurement units and that's what search us only support for this for for for for services the so we'll still stick to their resource templates the so this is so that little more complex example so really using the resource manager was and the will to deploy a little machine and so again you have some parameters they tell which size of machine you want to deploy which borda can't you want to use uh the initial set out and in which the network you want to deploy the machine and Microsoft of false so quite a range of of online it's distributions which you can use so in this case the just say OK give us the latest been 8 and you always get an up-to-date version the and if you want know if you deploy lots of machines on probably helps so if you take them right and then you can use the and the will of uh in dynamic inventories feature to pull the actual information from you deployed infrastructure and she did in year and the will scripts and reuse of the groups that are the thing that defined their for you are the actions you want to do later in the previous stage the so

24:59

on why we quite use in the will was as an introduction to the UN much at all and there's some points that we are not that happy about it and it doesn't work is the latest client libraries so it's always a little bit behind Microsoft did the initial implementation of the edges and the blue module but does not General Wheeler open source community around so until now I'm not sure this model let's see how this so and walls and using that diff defining resources using and the ballistic evidence simple task of a complex task it's better to switch back to the resource templates and just call the template and the book the dynamic entry was really helpful for us and be pretty much use it and it works and so now we kind of have a hybrid approach that we use adjacent templates to define all infrastructure and inject the parameters of the of the and the book

26:04

so that's a rough overview of the deployment of user Microsoft as a as a set of earlier if you have any questions you can ask them now all just meet me at the conference thanks thank hi how you say you're planning to shift you either in such shifting out in the highlands services I wonder if you've got an idea yet of what the silence of this thing you should do to our if for example Antonov use only the simple blob storage but we want to migrate and to the achieve services that as a poor whites and at the moment we are deploying all was close databases at all in some Microsoft Azure has of on-demand service for proscar services where automatic back up and automatic upgrades and all this stuff so that after 2 examples that we want you to go up in the state and even something like that a full managed to do so this and they don't have to care about and deploying stuff that's also an option but it I a straight on the I was wondering how do not indicate that new tools like all or like any other so if you have a gaze that has and basically you have a good repository or is all you all deployments installed there if requirement teach the fire and that's what everybody uses and once we want to migrate to a new version of the dual branch and then updated versions deployed to the test develop environment and see if every single works and then roll it or to the other stages yeah but you mentioned role based access control and I was wondering like how do you for example the word develop were less privileges in other people for were individual accounts use the the axis he's and now we have of be forcing 211 internal Active Directory hordes of open it up so and referencing true he sings the permissions that on all internal of the groups and the developers to add up and then we have some some labels attached to each you were developed and that's the way we manage to this success is that the i th that usually 1 well sorry ES Heizer iterations services will be there for that's a separate will we just use the API to single users in the inside thank you it OK let's thank you true