DevOps

This is going to be a quick post that shows the use of the PowerShell task in a Pipeline. Nothing in the post is really specific to the Azure DevOps Project we have been using over the last few weeks, but just in case you’re totally new to Azure DevOps and/or this series you can use the following posts to get started.

PowerShell Task

The PowerShell task will allow you to do pretty much anything. If there isn’t an existing DevOps task that fits your needs more than likely you can find a way to use the PowerShell task to accomplish what you need within the context of the computer the task is running on and even external computers that depending on your networking and security setup. The following is a sample task I added to a Pipeline that will output all the environment variables to the logs. This is an inline script, but you can also run scripts from files. Also, note that this works on both Windows and Linux agents.

While this script isn’t super useful for a production Pipeline I often use it when setting up a Pipeline to get a good feel for what is available variable wise. Also, keep in mind that depending on the trigger of the run these variables can be different. For example, if a run was triggered by a pull request you will have a number of pull request related variables. The following is the output of this command on my test project that was triggered via a pull request and therefore contains a bunch of SYSTEM_PULLREQUEST_x variables with information about the pull request. The agent was running Linux.

I don’t know about you but being able to see what paths the built-in path variables actually map to helps me a lot especially when files need to be moved around.

Wrapping Up

As stated above you can do just about anything with the PowerShell task. I have used it for everything from reading a JSON file to building a VM for QA. If you hadn’t used this task before I hope this post helped you get started and opened your eyes to the huge range of things you can do with the PowerShell task.

Last week we covered adding branch policies to a branch in an Azure DevOps Repo and this week we are going to deal with what happens when you need to break the policies you set up for some reason.

Setting Bypass Security

There are a couple of ways to set up bypassing depending on how broadly you want to give someone rights to bypass. We are going to start with the narrower option which is allowing bypass for an individual branch (sadly security options are currently available at the folder level like policies are). Starting from the list of branches for your repo mouse over the branch you want to set security for and click the three dots for the menu and select Branch security.

On the dialog that shows find the user, you want to change for security for, Eric Anderson in this example. After selecting a user their specific setting will be loaded to the right. On Bypass policies when completing pull requests change the option to Allow.

The second option for setting bypass security is at the repo level. From your Project settings under Repos select Repositories and then fine the Branches node under the project you want to set the policy for.

From here it is the same as the branch level. Find the user, you want to change for security for and set Bypass policies when completing pull requests to Allow.

Bypassing Policies on a Pull Request

Now that we have our security setup we are going to walk through what the bypass process looks like. Here we have a pull request that is missing approval by a reviewer.

Now let us say we don’t have a reviewer available for some reason and we need to complete this PR without review. Use the dropdown on the Set auto-complete button and click Complete.

When the PR completion dialog show you will notice a section at the top with a red background that lists out the policies that haven’t been met. With your new-found security, you will also have a section for Policy override options. To proceed and bypass the policies check the Override branch policies and enable merge checkbox, enter your reason for overriding, and click the Override and complete button.

Do note that the fact a PR was overridden is visible on the list of completed PRs and the reason will show when mousing over the bypassed indicator as well as in the details of the PR.

Wrapping Up

Branch policies are great and will help you make sure the code that makes it in your branches are high quality and don’t break your builds. Hopefully, you won’t need to bypass your policies often, but now you know-how without having to temporarily remove the policies or getting people used to blindly approving changes.

Repo Introduction

The repo used here is the same one used in the Pipelines posts linked above and contains two .NET Core 3.1 web applications. The repo also contains three branches (master, releases/1.0, and releases/1.1).

It is worth noting that putting a forward slash in a branch name displays as a folder in the UI as you can see with releases in the screenshot above.

Editing Policies

Mouse over either a specific branch or a folder and it will show the three dots for the menu. Click the dots and then select Branch policies. For this example, we are putting policies on the master branch.

This will bring you to the page that allows you to view and edit the policies on the selected branch or folder.

The descriptions do a good job of explaining what policies do what so I’m not going to bore you with repeating them. The official docs on branch policies also go into a lot more detail. If you are not working along I highly recommend using Require a minimum number of reviews and Check for comment resolution. Build validation I would recommend no matter your team size as it keeps you away from the possibility of having that one magical machine that is the only one your build will work on. The following screenshot is with the first two recommend policies set. The settings shown for the require a minimum number of reviews are based on the fact that my project only has one contributor. After your done make sure and click the Save changes button.

Build Validation Policies

There is a bit more to the build validation policy which is why I’m covering it in a different section. On the Branch policies screen click the Add build policy button.

In the edit build policy screen, the only required change is selecting the Build pipeline to make available when a PR that is targeting the branch that is policy is for. Here we are using the automatic Trigger so any time we push to our remote branch when it has an open pull request it will run the select build pipeline. Policy requirement controls if a successful build is required before the pull request can complete or not. For a since person project Build expiration isn’t a big deal, but if your working with a team it can be helpful. Click Save when you are done.

Back on the branch policies screen, you will see the new requirement listed. You can also add as many build validations as you need.

Wrapping Up

Hopefully this quick little into to branch policies will help your team improve the quality of the code that makes it into your branches. I know to require another person or two to review your code before you can check-in sounds like it will slow you down if you are new to the concept but in reality, it helps catch issues before they make it to QA and production which saves time and money in the long run.

A few weeks ago we covered Conditionals in YAML to show how to conditionally run tasks and jobs as well as how to make a job dependent on another job. This post is going to cover combing conditional and job dependencies. If you are new to this series you can use the following posts to catch up.

Existing Job

As a reminder, our sample pipeline has 4 jobs. The WebApp1 job always runs, the WebApp2 job is run or skipped based on a pipeline variable, the DependentJob depends on WebApp1 and WebApp2, and finally, the TagSources job is dependent on all the previous jobs. We are going to be tweaking the TagSources job in this post. The following is the YAML for the setup of the TagSources job without its tasks.

With this setup WebApp1, WebApp2, and DependentJob all have to report successful or the TagSources job will be skipped. The following screenshot shows a pipeline run with the variable to build WebApp2 set to false.

As you can see the TagSources job was skipped because one of its dependent jobs was skipped.

Dependencies with Conditions

Let us say for our pipeline we want the TagSources job to run as long as all jobs were successful or if WebApp1 was successful and the WebApp2 and DependentJob jobs were skipped. To do this we are going to add a condition element and manually check the results of the dependencies as you can see in the following.

And you can see in the results the TagSources job ran even with the two skipped jobs.

Wrapping Up

If there is a simpler way to accomplish what we did above I would love to hear about it. If I remember correctly I found the above in a GitHub issue, but I don’t have the link. I’m not sure how many of you will have Pipelines that will need this, but hopefully, this will save someone some research time.

In this week’s post, we are going to cover manually tagging instead of using the tagging feature built into Azure DevOps. This post will be using a sample Azure DevOps project built over the last few weeks of posts. If you want to see how this project has gotten to this point see the following posts.

Why?

Tags give you information about the state of your repo when an event happens, a build for our case. This can be super useful especially when you need a place to branch for a hotfix, for example. The automatic tagging covered in my Azure DevOps Pipelines: Naming and Tagging post is the way to go if it works and it has for about 95% of my projects. I have a couple of projects where the automatic tagging would not work for some reason. Since I couldn’t work out why Azure DevOps wouldn’t tag I ended up having to add a job to the Pipeline to perform the tagging.

Permissions

To perform this manual tagging the account used in our Pipeline will need to be giving contribute permission to our repo so that it will be able to push the tag. Use the gear in the lower left to open the Project Setting page.

Select the Repositories option and then click on the specific repo you would like to change the settings for, Playground is the repo we are using in the example. Note that this setting can also be changed using the top-level Git repositories option if you want to change the permission for all of your repos.

On the Security tab under Users looks for the user than has Build Service in the name and select it. When the user is selected it will show their permissions to the right of the users. Find the Contribute option and change its value to Allow.

YAML Changes

As a quick reminder, the YAML for this project currently has 3 jobs. Two web application builds (WebApp1, WebApp2) and the third job is there to show how to use job dependencies (DependentJob). I’m going to skip showing the YAML for these existing jobs, but that if you need the full existing YAML it can be found in posts linked at the top.

Since we already have multiple jobs in our Pipeline we are going to add the tagging code as a new job. This will easily allow us to only tag when all the other jobs have run successfully. The following is the full YAML for the new job.

First off you see that this job depends on our existing three jobs to complete successfully before this new job will run.

dependsOn:
- WebApp1
- WebApp2
- DependentJob

Starting in the Steps section you will see a checkout step that is normally handled automatically by the job, but in this case, we need to use the persistCredential option so the job will still be authed and allow us to push to our git repo. The self option is used to signify the current repo/branch. See the official Checkout docs for more information.

- checkout: self
persistCredentials: true
clean: true
fetchDepth: 1

The final bit of the job is the PowerShell task that performs the actual tagging and push. I found most of this on stackoverflow and/or in a GitHub issue, but it has been a while so I don’t have the links handy. The StdErr stuff was to work around some git output that didn’t affect the tagging but was causing the job to be marked as failed. Other than that it is using the standard git commands to tag and push.

Wrapping Up

I honestly hope none of you have to use this. It was a huge pain to work out. It is also to work around some sort of issue with Azure DevOps built-in tagging support. To be fair I can see where in more complex Pipelines you might need a level of flexibility that the built-in tagging couldn’t provide and this would be your only option.

In this week’s post, we are going to cover changing the naming of Pipeline runs to provide more information as well as tagging our source when a pipeline is run. This post will be using a sample Azure DevOps project built over the last few weeks of posts. If you want to see how this project has progressed check out the following posts.

Naming

By default, Pipeline runs are naming using the current date with a number for how many times the Pipeline has run for the day. For example, the fourth build on March, 17th, 2020 would start with the name 20200317.4 plus the description of the last commit. If you have the need you can change this naming scheme by using a name element in your YAML. As with the rest of the YAML related things you have all the same information available as the rest of the Pipeline to use in building whatever name might be helpful for your situation. For our example, we are going to add the branch name to the front of the run date and count by adding the following name element to the top of our YAML file.

The above would result in master_20200317.4 using the same example as above. The following screenshot shows the actual results from this change in the sample Pipeline.

Tagging

Tagging source code when running a Pipeline is a helpful way to know exactly what was included when a Pipeline is run. Here we are going to walk through using Azure DevOps to automatically tag on successful builds. From the Pipeline, you want to tag click the Edit button as you would if you were going to edit the Pipeline’s YAML. Then click the three dots and select Triggers.

Now click on the YAML tab, then Get sources, under Tag sources we are going to select On success so tags will only happen if the build completes successfully. Also, notice the Tagformat which allows you to change how the tag is named. When done make sure and Save your changes.

After running a build with the above changes head over to the Repos area of the project. From Files click on History and from there you can see the tag on the last commit that was included in the build, which is displayed here as master_20200325.1.

Wrapping Up

Using clear naming for your builds can give you a lot of information at a glance, but it does take some thought to make sure information your including is helpful. Tagging is also super helpful when viewing history to know what went out with what release, and of course, they can also be used for branching. Come back next week for a look at how to manually tag when Azure DevOps automatic tagging doesn’t work for whatever reason.

In this week’s post, we are going to cover some ways to make tasks and jobs run conditionally. This will include options such as Pipeline variables to jobs that are dependent on other jobs. This post will be using a sample Azure DevOps project built over the last few weeks of posts. If you want to see the build-up check out the following posts.

Sample YAML

The following YAML is based on the YAML from the previous posts, see links above, expanded with examples of using some ways of conditionally running some task or job. This is the full file for reference and the rest of the post will call out specific parts of the file as needed.

Job Dependencies

The more complex pipelines get the more likely the pipeline will end up with a job that can’t run until other jobs have completed. The YAML above defines three different jobs, WebApp1, WebApp2, and DependentJob. I’m sure you have guessed by now that the third job is the one that has a dependency. To make a job dependent on other jobs we use the dependsOn element and list the jobs that must complete before the job in question can run. The following is the YAML for the sample DependentJob with the dependsOn section highlighted.

The above condition will cause the WebApp2 job to be skipped if the BuildWebApp2 variable isn’t true. For more details on how to use conditions see the Conditions docs.

Creating a Pipeline Variable

The rest of the post is going to walk through creating a Pipeline variable and then running some sample builds to show how depends on and the conditions defined in the YAML above affect the Pipeline results.

We are starting from an existing pipeline that is already being edited. To add (or edit) variables click the Variables button in the top right of the screen.

The Variables pop out will show. If we had existing variables they show here. Click the New variable button to add a new variable.

We are adding a variable that will control the build of WebApp2 called BuildWebApp2 that defaults to the value of true. Also, make sure and check the Let user override this value when running this pipeline checkbox to allow us to edit this variable when doing a run of the pipeline. Then click the OK button.

Back on the Variables dialog click the Save button.

Edit Variables When Starting a Pipeline

Now that our Pipeline has a variable when running the Pipeline under Advanced options you will see the Variables section showing that our Pipeline has 1 variable defined. Click Variables to view/edit the variables that will be used for this run of the Pipeline.

From the Variables section, you will see a list of the defined variables as well as an option to add new variables that will exist only for this run of the Pipeline. Click on the BuildWebApp2 variable to edit the value that will be used for this run of the Pipeline.

From the Update variable dialog, you can change the value of the variable. When done click the Update button.

Pipeline Results from Sample YAML

The following is what our sample Pipeline looks like when queued with the BuildWebApp2 variable set to false. As you can see the job will be skipped.

Next is the completed results of the Pipeline run. You can see that the Build Dependent Job was skipped as well since both Build WebApp1 and Build WebApp2 must complete successfully before it will run.

Changing the BuildWebApp2 variable back to true and running the Pipeline again results in all the jobs running successfully.

Wrapping Up

Hopefully, this has helped introduce you to some of the ways you can control your Pipelines. As with everything else Azure DevOps related things are changing a lot and new options are popping up all the time. For example, while writing this post the team just announced Runtime Parameters which look like a much better option than variables for values that frequently vary between Pipeline runs.

In last week’s post, we refactored some YAML that was reusable into a new file. This post is going to cover moving that same reusable YAML to a new repo and then using it in our existing sample repo. This post is going to build on the Azure DevOps project created in previous posts if you are just joining this series check out the previous posts to find out how the project has progressed.

Create a New Repository

First, we need to create a new repository that will be used to share the YAML in question. Using the Repos section of Azure DevOps as a starting point you click the dropdown with the currently selected repo name, Playground in this example, and then click New repository.

You will be presented with a dialog where you will need to enter the Repository name and any other of the options you want to configure. In this example, we are naming the repo Shared and adding a Git ignore file for Visual Studio. When done click the Create button.

The following steps should be taken on the new Shared repo. At this point, you could clone the repo and do the rest of the steps locally and then push the changes to the repo or you can use the web interface to make all the change which is the route this post is going to show. Either way, you go it shouldn’t be too hard to adapt the steps. For the web interface to add a new file in the root of the repo click the three-dot menu to the right of the repo name and then select New and then File.

The next prompt will ask for a file name, I’m using buildCoreWebProject.yml. Click Create to continue.

You will land in the file editor. Copy and paste the code out of build.yml that we were using from the previous post into the new file. The following is the full YAML from build.yml for reference.

Once the code is copied in click the Commit button to save the changes to the master branch.

Switching back to our original repo, Playground in this example, we need to add the Shared repo as a resource for in our azure-pipelines.yml file. The following is the resource deliration for using another Azure DevOps repo. The official docs for check out multiple repositories also show examples with GitHub and Bitbucket. I also found this stackoverflow question helpful.

Shared on the repository line is the name we will be using when referencing a file out of the Shared repo. Name is the Azure DevOps project and repo name. Type is the repo type which is Git in our case. Finally, ref is the branch name from the Shared repo that we want to use. Now that we have access to the files from the Shared repo we can use its buildCoreWebProject.yml instead of the local build.yml as a template.

Notice that the only change is on the template line which changed from build.yml to [email protected]. The @Shared on the end of the filename is what tells the pipeline the file’s source is the Shared repo. The following is the full azure-pipeline.yml for reference. The frist job is using the template form the Shared repo and the second one is using a local template.

Wrapping Up

Being able to utilize YAML from different repos can help cut down on duplicated YAML and help keep your pipelines across repos cleaner. As with anything else, this is a useful tool when applied appropriately.

In this post, we are going to refactor our sample Azure DevOps Pipeline to move some of the redundant YAML to a new file and replace the redundant parts of our main YAML file. This post is going to build on the Azure DevOps project created in previous posts. If you are just joining this series check out the previous posts to find out how the project has progressed.

Starting YAML

The following is the YAML for our current pipeline that builds two different web applications using two different jobs. Looking at the two jobs you will notice that they both have the same steps. The only difference in the steps is which project to build (WebApp1.csproj or WebApp2.csproj) and what to call the published artifact (WebApp1 or WebApp2). When developing applications we would never stand for this level of duplication and the same should apply to our pipelines.

Add a New File

To attack the duplication above we need to take the shared steps from above and move them somewhere they can be reused. We will be walking through the steps using the Azure DevOps web site and committing directly to the master branch, but these same steps could be performed locally or on the web on any branch. First, from the Repos section of the site we need to add a new file by clicking the three dots at the level we want the file added. In this case, we are adding to the root of the repo but the same option is available on any folder.

A dialog will show where you can enter the New file name, we are going to use build.yml in this case. Next, click Create to continue.

Shared YAML

Now that we have a new file we can start building the new YAML that will handle the repeated steps from the original jobs. The first thing we are going to do is define a set of parameters that this set of steps can be called with. We are going to use this to pass what project to build, which build configuration to use, and what name the published artifact. The following is the definition of our parameters.

We can then use these parameters in the rest of the file using the ${{ parameterName }} syntax. Note that any pipeline variables are also available using the $(variableName) syntax. The following bit of YAML shows both types in the arguments line.

While you can use pipeline variables I recommend passing all the values you need via parameters for the same reason that we try to avoid global variables when doing general programming. I’m using both here to show the usage of each. The following is the full YAML in our new file.

Using Shared YAML

Not that we have the YAML that is the same between our two build jobs we can switch back over to our main YAML file, azure-pipelines.yml in the sample, and remove the steps we are wanting to replace. While the jobs will both have a steps section the only thing we will have left in them is a template call to our other YAML file, build.yml for the sample, that passes the parameters to run the other file with. The following is the resulting YAML file with the call to the shared file in both jobs highlighted.

Wrapping Up

Being able to remove duplication from your YAML files should help improve the maintainability of your pipelines. I know the samples don’t show it, but the template is just a step and you could have other steps before or after it just like you would with normal tasks.

This post is going to show how to run multiple jobs out of a single YAML file from an Azure DevOps Pipeline. This post is going to build on the Azure DevOps project created in previous posts. If you are just joining this series check out the previous posts to find out how the project has progressed.

This post is going to take this pipeline and split the build and publish of the two web applications and make each application its own job. In Pipelines a job is something that a single agent takes and runs. By splitting into multiple jobs the pipeline can run multiple jobs at the same time if you have enough build agents available. One reason to do this would be to speed up the total Pipeline run if you have parts of your build that are independent. Another example of why you would need jobs is if the different jobs need different agents such as one needing a Windows agent and another a Linux agent.

Creating the Jobs

Having different jobs means we are going to have to move things like what agent pool to use and the steps for the job under a jobs element and then declare a specific job and the details that job needs to run. As you can see in the following example the end goal is the same as the YAML from above (except it is dealing with a specific project), but the details are nested under jobs and defined under a job.

Also notice that you can still define variables that can be used across jobs as is done above with the buildConfiguration variable. The following is the full YAML file that builds and publishes the artifacts for both web applications.

After all your edits are done commit the changes to your YAML file and then run the pipeline. As you can see from the following screenshot of my sample pipeline run the pipeline has two jobs instead of one that the original YAML resulted in. Also, note that the pipeline results in two published artifacts (one per job in our case) instead of the one with the original.

Wrapping Up

As mentioned above there are a lot of reasons you might want to split up your pipeline into multiple jobs and hopefully, you now have a good idea of how that is done. Make sure and check back in the future for a post on how to take repeated tasks and make them reusable.