As one final check for the release, we can execute a smoke test on the deployed production Function App. This smoke test won’t modify any data and will simply allow us to check that the Function App is responding to HTTP requests.The smoke test function won’t be disabled in production but it will be protected by a function key.

The preceding smoke test function returns a 200 OK result and also allows the URL to the AzureDevOps release to be provided, which is written to the logs. This allows ops to see from which release the smoke test function came from.

The first step is to add a variable to the release pipeline called “prodSmokeTestKey” with the value being the function key from production for the smoketest function. Once this key is copied from the Azure Portal it can be copied into the variable value and the padlock icon clicked to mark this as sensitive data:

“Gates allow automatic collection of health signals from external services, and then promote the release when all the signals are successful at the same time or stop the deployment on timeout.” [Microsoft]

Gates can be evaluated before a stage executes and/or after a stage executes.

“Approvals and gates give you additional control over the start and completion of the deployment pipeline. Each stage in a release pipeline can be configured with pre-deployment and post-deployment conditions that can include waiting for users to manually approve or reject deployments, and checking with other automated systems until specific conditions are verified. In addition, you can configure a manual intervention to pause the deployment pipeline and prompt users to carry out manual tasks, then resume or reject the deployment.” [Microsoft]

We can add a post-deployment gate that will mark the release as unsuccessful if the smoke test function does not respond successfully. To do this we can use the “Invoke Azure Function” task. Other tasks that can be executed as part of a gate include “Invoke REST API”, “Query Azure Monitor Alerts”, and “Query Work Items” tasks.

To add a post-deployment gate to the “Deploy to Production” stage, click the post-deployment conditions icon, enable the Gates switch, and click the + Add button.

Choose “Invoke Azure Function”. Paste in the URL to the production smoke test function and for the function key specify “$(prodSmokeTestKey)” to retrieve the key from the pipeline variable that we set up earlier. Select the method as POST and in the body enter “{ "releaseurl" : "$(Release.ReleaseWebURL)" }”. This is the JSON payload that will be sent to the smoke test function.

Hit save and queue a new release.

Once the production deployment is complete, there will be a 5 minute delay before the gate(s) are evaluated for the first time and a 15 minute wait between re-evaluation of gates which ultimately means a lengthy delay between the deployment and smoke test execution. To improve this you can set the the “The delay before evaluation” to 1 minute, and in the Evaluation options section at the bottom set “The time between re-evaluation of gates” to5 mins and “The timeout after which gates fail” to 6 mins.

As an alternative to using gates in this way (especially if you need to add multiple gates or complex smoke testing requirements) could be to have another post-production-deployment task/job/stage that calls the smoke test function(s) in another source-controlled test project in a similar way to how the end-to-end tests were executed against the test Function App.

In the previous instalment we created the release pipeline and now have continuous deployment working. Currently if the unit tests pass (and the rest of the build is ok) in the build pipeline, the release pipeline will automatically execute and deploy the Function App to Azure.

In this instalment of this series we’ll add some additional stages in the release pipeline to deploy first to a test Function App in Azure, then run some functional test against this test deployment, and only if those tests pass, deploy to the production Function App.

Deploying a Function App to Test

The first step is to edit the release pipeline that we created earlier in this series and add a new stage. A quick way to do this is to click the Clone button on the existing “Deploy to Production” stage:

Change the name of the cloned stage to “Deploy to Production” and the original stage to “Deploy to Test”, it should now look like the following screenshot:

Next edit the tasks in the “Deploy to Test” phase, click the Disable Testing Functions task and click Remove to delete the task from the test stage.

We need to change the deployment target, so click the Azure App Service Deploy task and change the App Service name to “InvestFunctionAppDemoTest” – we want to deploy to the test Azure Function App not the production one.

Check that the test Investor created in step 1 has had its stock value updated – this is done by called the test function GetInvestor

Side Note: In this example we’ve create 2 additional Azure Functions to help facilitate testing, one to create a test investor and one to retrieve Investor details so we can assert against the final result. We could have just manipulated Azure Table storage directly within the test methods but I wanted to show this approach to demonstrate a number of features such as automating function disabling as part of deployments and passing pipeline variables to test code. Normally we don’t want to deploy testing-related items to production for a whole host of reasons (performance, security, data integrity, etc.), but this approach if properly managed and secured can be quite a useful quick win. If you do implement these kind of test functions you must ensure that they cannot be called if deployed to production by restricting them at multiple layers: first by securing the functions with a secret function key and second by ensuring as part of the deployment the testing functions are disabling in the app settings. You could even add a 3rd level of checking by asserting that the function is executing in a testing environment with like an AssertInTestEnvironment being called at the start of each test function. All that being said, deploying test functions to production adds all this additional complexity and risk and so is best avoided.

There’s a few things that this end-to-end test needs.

Firstly it needs to know the Azure Function keys for the two test functions and also the Portfolio function. We don’t want to commit these keys to source control, so we can instead create pipeline variables for them and then access them via environment variables in the C# test code by using Environment.GetEnvironmentVariable(variableName).

Secondly there is an extra level of checking around the test functions being able to be called in production. The functions will be disabled in production, in addition to being protected by a function key. Whilst these two things should make it impossible for them to be called, there is an extra check implemented in the following class:

For the test functions to be enabled in the test Azure Function App, there needs to be an application setting called “InvestFunctionApp.IsTestEnvironment” set to “true”.

Adding Function Keys as Pipeline Variables

As we did earlier in the build pipeline, we can add variables to the release pipeline, the variables are called CreateInvestorFunctionKey, GetInvestorInvestorFunctionKey, and PortfolioFunctionKey. The values of these should be the function keys of those functions deployed to the test environment. You may need to create a release and run the release pipeline first to deploy the app to test before you can get the test function keys (a bit of the chicken and the egg here).

Notice in the preceding screenshot that we’re not marking these test keys as secret, though we should ideally go and do that but it introduces a little extra complexity if we want to access them as environment variables in the C# test code.

Adding a Functional End-to-End Test Stage

Now we have the app deployed to test, we want to run the functional tests against it.

To do this we create a new empty stage called “Run Functional Tests” and modify the “Deploy to Production” stage trigger to be run after the new testing stage completes as the following screenshot shows:

Notice here that we’re creating a completely new stage to run the functional tests, this is for demonstration purposes to show the flexibility of being able to design your release pipeline however you want though this approach doesn’t align fully with the conceptual idea of a stage being a: “logical and independent entity that represents where you want to deploy a release generated from a release pipeline.” [Microsoft] . It does however conform to the idea that a stage is a “logical entity that can be used to perform any automation”[Microsoft]. In any case, it is more likely that in a real scenario we wouldn’t create a new stage just to run the functional tests. What we could have instead are a couple of stages, one called “QA” (a testing environment deployment) and one called “Production”. We could then run the functional tests as a separate task in the “QA” stage. You should make sure you read the documentation to fully understand what stages are and the nuances such as “Having one or more release pipelines, each with multiple stages intended to run test automation for a build, is a common practice. This is fine if each of the stages deploys the build independently, and then runs tests. However, if you set up the first stage to deploy the build, and subsequent stages to test the same shared deployment, you risk overriding the shared stage with a newer build while testing of the previous builds is still in progress” [Microsoft].Another option would be to create a new Function App in Azure (with a unique name) for each execution of the stage, run the functional tests against it, and then delete the Function App.The great thing about Azure Pipelines is the flexibility they offer, however this flexibility comes at the cost of potentially shooting yourself in the foot.In the future I intend to write more about good practices and concepts when designing pipelines.

Continuing with the demo scenario, we now need a task in the new testing stage to execute dotnet test on the functional end-to-end test project.

To do this we can add a .Net Core task, set the command to test and the path to project as “$(System.DefaultWorkingDirectory)/_InvestFunctionApp/e2etests/InvestFunctionApp.EndToEndTests/InvestFunctionApp.EndToEndTests.csproj” (notice in this path we’re accessing the e2etests artifact created in the YAML build).

Setting Test Azure Function Application Settings

When deploying to the test Function App in Azure we need to set the application setting “InvestFunctionApp.IsTestEnvironment” to “true”. Rather than using Azure CLI we can do this as part of the Azure App Service Deploy task in the Application and Configurations Settings as the following screenshot shows:

Testing the Updated Release Pipeline

Once you’ve made all these changes and saved them you can queue up another manual release to see if everything works. Just click the + Release button and choose “Create a release”. Specify the latest build in the artifacts section and click Create.

This will queue and start a new release:

After a while the release should complete and all stages should complete successfully:

Clicking on the “Run Functional Tests” stage and then the Tests tab you can see the “AddToPortfolioFunctionShould.BuyStocks” test executed and passed:

In the final part of this series, we’ll see how to execute a smoke test against the deployed production Function App to verify at a high level that all is well with the deployment.

In the previous instalment we saw how unit tests are executed as part of the build pipeline and that if tests fail then the build fails. If the tests (and the rest of the build) succeed, a release pipeline can be triggered to deploy the Function App automatically to Azure.

Creating an Initial Function App Release Pipeline

In your Azure DevOps project, click Pipelines and Releases.If you have no pipelines currently defined you will see a “No release pipelines found” message and a New Pipeline button.Clicking this button will start the release creation wizard, the first step of which is to choose a starter template (or start with an empty job).You should explorer these templates to get an idea for what’s possible. For the purposes of this series, click the Empty job link:

This will automatically create a stage in the new release pipeline, rename this stage to “Deploy To Production” and click the close button:

Adding Build Artifacts to a Release Pipeline

Next we need to add artifacts from the build pipeline to be used in the release. To do this click the Add link in the Artifacts section.

Leave the source type as Build (artifacts are being published from a build pipeline). As the source, select the build definition that we created earlier in this series. For the source alias enter “_InvestFunctionApp” and click Add:

Adding an Azure App Service Deploy Release Task

In the “Deploy to production” stage click the “1 job, 0 task” link:

Click the + button next to “Agent job” to show a list of prebuilt release tasks that can be added to the stage:

In the search box type “app service deploy” and click the Azure App Service Deploy task that shows up, then click the Add button to add the task to the stage:

You will now have a new task in the stage that will state: “Some settings need attention”. Click on the newly added Azure App Service Deploy task to configure it.

First you’ll need to specify the Azure subscription you want to deploy into - you will need to click the Authorize button and go through the authorization process.

Next change “App type” to “Function App”.

From the App Service name dropdown select the Function App you want to deploy into, in this series it’s the InvestFunctionAppDemo function created earlier in this series in the Azure Portal.

The next step is to choose what to deploy. To do this use the “…” next to Package or folder. This will allow you to choose the artifact that was created in the build pipeline. In this case we want to select the “app” artifact that contains the published Function App:

Select the “app” folder and click OK. You should see something similar to the following screenshot though your Azure Subscription details will be different:

Disabling Testing Azure Functions in Production

Later in this series we’ll be adding functional end-to-end tests that make use of a number of test functions. We do not want these test functions to be enabled in the deployed production Function App in Azure. (In the next part in the series we’ll discuss these testing functions more).

To disable functions in Azure Functions V2, a setting can be added in the format “AzureWebJobs.FUNCTION_NAME.Disabled” and set the value to “true”.

One way to do this as part of the release to production is to use the Azure CLI.

To execute Azure CLI commands, you can add a new task to the stage, the Azure CLI task. This task should come before the Azure App Service Deploy task. The task can be configured as follows:

Display name: Disable Testing Functions

Azure subscription: your Azure Subscription / resource group that contains the Function App you are deploying to

Script location: Inline script (you could also supply a path to a source controlled script)

The contents of the inline script box are as follows:

call echo These could also be set in the Azure App Service Deploy task in the Application and Configuration Settings
call echo We are doing it this way so we ca have a separate task in the stage to make it more obvious
call az webapp config appsettings set -g DontCodeTiredDemos -n InvestFunctionAppDemo --settings AzureWebJobs.CreateInvestor.Disabled=true
call az webapp config appsettings set -g DontCodeTiredDemos -n InvestFunctionAppDemo --settings AzureWebJobs.GetInvestor.Disabled=true

This preceding script uses the az webapp config appsettings set command specifying the resource group “-g DontCodeTiredDemos”, the name of the Function App “-n InvestFunctionAppDemo” and the setting name and value “--settings AzureWebJobs.CreateInvestor.Disabled=true”.

Once configured, click the Save button and move the task above the deploy task as the following screenshot shows:

At this point you can also name the release pipeline by clicking “New Release Pipeline” at the top and choosing your own name, for example “InvestFunctionAppReleasePipeline”.

Enabling Continuous Deployment in an Azure Release Pipeline

If you want the release pipeline to automatically start when the build pipeline finishes, head back to the pipeline view and click the lightening bolt icon in the Artifacts area and enable the continuous deployment toggle switch as the following screenshot shows:

If you get a “Problem connecting to the service” error message it may not actually prevent the CD trigger but you may want to refer here.

Click the Save button.

Creating a Manual Release

Click the + Release button at the top right and click Create a release.

Choose the latest build from the build pipeline in the Artifacts section and click the Create button.

You will see a message saying “Release Release-1 has been created” with a handy link to click – click this and it will take you to the release being executed:

And once the release pipeline has executed you should see a “Succeeded” message in the Deploy to Production stage.

If you head over to the Function App in the Azure Portal you should see the functions deployed, along with the two testing functions being disabled and the deployment showing for Release 1 as the following screenshot shows:

Testing the CI Trigger

To test the CI trigger, make a change in the repository, commit the change, and then push to GitHub.

After a short while, the build pipeline should notice the change and kick off a new run of the build pipeline.

Once the build pipeline completes without error, the release pipeline should be automatically triggered and the changes (for Release 2) automatically deployed to the production Function App in Azure.

So now, we can add a feature or fix a bug, push the changes, and within a few minutes have the new feature or fix in production without us having to do anything else.

In the next instalment in this series we’ll add some functional end-to-end tests because at the moment ,all we have are unit tests in the build pipeline to verify correctness.

Notice in the preceding YAML that the test project being executed is “src/InvestFunctionApp/InvestFunctionApp.Tests” and the trx option is being specified to log test results into a Visual Studio Test Results File (TRX) format file.

Once the tests execute, the results need to be made available to the pipeline so we can see/explore the test results in the GUI. To do this the PublishTestResults@2 task can be used:

In the next part of this series, now that we have a build pipeline that will run tests and create artifacts, we can create a release pipeline to automatically deploy the app to production if the build succeeds.

In the previous instalment we created a build pipeline, defined the build steps using YAML, but when we tried to execute the build, it failed with the error:”buildConfiguration: command not found”. This is because in the YAML build definition we are referencing the $(buildConfiguration) variable which has not yet been defined.

An Overview of Azure Pipeline Build Variables

“Variables give you a convenient way to get key bits of data into various parts of your build pipeline.” [Microsoft]

Essentially variables allow you to define key/value pairs that can be used throughout the pipeline. They can be passed to YAML build definitions for use in steps, they can be accessed as environment variables in scrips and from unit tests, and can be used in the pipeline GUI when configuring tasks.

There are a whole host of predefined variables such as Agent.JobStatus, Build.BuildNumber, System.PullRequest.PullRequestId, etc.

Creating a Custom Variable in an Azure Pipeline

First off, open the build pipeline by clicking Builds, select the build pipeline you want to edit, and click the Edit button:

In the Variables section click Add:

For the variable Name enter “buildConfiguration” and for the variable Value enter “Release”, also check the “Settable at queue time check box”. This variable will default to “Release” but can also be set to “Debug” when manually queuing a build if you want to create a debug version of the Function App:

Notice the little padlock to the right of the value, clicking this enables you to store “sensitive values in a way that they cannot be seen or changed by users of the release pipelines” [Microsoft]

“The values of hidden (secret) variables are stored securely on the server and cannot be viewed by users after they are saved. During a deployment, the Azure Pipelines release service decrypts these values when referenced by the tasks and passes them to the agent over a secure HTTPS channel.” [Microsoft]

Another nice thing about designating a value as sensitive is that it will also be redacted in log messages produced by Azure Pipelines. Sensitive variables can also be passed to scripts – you should read the documentation and be careful about any custom logging or other output you create as you may inadvertently reveal secrets.

To save the changes, click the Save & queue dropdown and choose Save:

Once defined, build variables can be consumed in the YAML build template by using the token pattern $(VARIABLENAME), for example $(buildConfiguration):

Manually Queuing an Azure Pipeline Build

To queue up a new build, click the Queue button, optionally choose a branch/commit and click Queue. (Also notice here that it’s possible to set the buildConfiguration variable to a custom value for this build or leave it as the default “Release”):

Click the handy shortcut in the notification:

And this will take you to the build that should start executing.

Now that we have the custom variable defined, the build should now complete successfully:

Also notice that the steps in the build from “Build solution” to “Publish end to end test artifact” correspond to the steps defined in the YAML and that these are set using displayName: 'Run unit tests'.

In the next instalment of this series we’ll look at how unit tests are executed as part of the build pipeline.

In the previous instalment we created an Azure DevOps organization and project, we can now create a build pipeline by clicking the New pipeline button:

Next we need to specify where the source code is located, this could be hosted inside Azure DevOps using Azure Repos or GitHub. The demo app is located in GitHub.

Next we need to give permission from GitHub to enable the pipeline to access the repository. The easiest way to do this is to click the Install our app from the GitHub Marketplace link as shown in the following screenshot:

You can allow access to all current (and future) repositories or specified ones:

In the preceding YAML, the displayName items help to describe what is happening and are hopefully fairly descriptive (they will also appear in the Azure Pipeline GUI/logs when builds are run).

Creating Multiple Build Artifacts in Azure Pipelines

One important thing to note in the preceding YAML, is that we are creating two separate build artifacts, one that contains only the Function App contents (for deployment to Azure) and one to be able to run the tests.

The Function App artifact is created by first calling dotnet publish and choosing the (temporary) output directory app with the switch --output '$(Build.ArtifactStagingDirectory)/app'. To actually create the build artifact that can be consumed in a release pipeline, the prebuilt PublishBuildArtifacts@1 task is called. This task takes its input from pathtoPublish: '$(Build.ArtifactStagingDirectory)/app' and will create a build artifact with the name app by virtue of the artifactName: app setting.

To create a separate artifact for the tests, the CopyFiles@2 task is used to copy (effectively the entire solution including the test projects) to the (temporary) targetFolder: '$(Build.ArtifactStagingDirectory)/e2etests'. The final PublishBuildArtifacts@1 task creates a second build artifact called e2etests.

We’ll see these artifacts used later in this series of blog posts.

The reason we create two artifacts here is to separate out what will get deployed to Azure Functions from the code/binaries that contain the tests. What we don’t want is to publish the tests, xUnit.net DLLs etc. to the deployed function apps in Azure. This can also help the in readability of the release pipeline and potentially help to segregate things so that only the indented things get deployed.

We can now save the changes to azure-pipelines.yml and push them to GitHub (which will actually trigger a new CI build).

Viewing Azure Pipeline Builds

Once the changes are pushed, the Azure Pipeline that we created will notice the changes and execute. If you click on Builds you will see the build running:

Clicking in the build will show you all the steps:

Notice in the preceding screenshot that the “Build solution” step is failing. Clicking it will show you the logs for the step:

Notice in the preceding screenshot the error”buildConfiguration: command not found”. This is because in the YAML build definition we are referencing the $(buildConfiguration) variable which is not yet defined.

In the next instalment of this series we’ll learn how to add variables to a pipeline and fix this problem.

Creating Azure Function Apps

There will be two Function Apps created in Azure. One will be a self-contained test environment (with separate Azure Storage account) and to keep things simple they both belong to the same resource group. The other Function App will be the production version. For simplicity we won’t be using slots, proxies, etc.

The two Function Apps are:

InvestFunctionAppDemo (production)

InvestFunctionAppDemoTest (test)

The test environment will be used to check the deployment artifacts and as a target to execute functional end-to-end tests against.

Once these two Function Aps have been created in the Portal, all of the deployments will happen automatically from the release pipeline, including setting test/production specific Function App settings.

Signing Up or Azure DevOps

Now that there are some Function Apps created in Azure to deploy to, you can deploy to them from an Azure release pipelines. To do this you’ll need to sign up to Azure DevOps.

Azure DevOps contains a number of services, in this series we’ll be using the Azure Pipeline service.

An Overview of Azure Pipelines

Azure DevOps is currently free for open source projects and small teams with some limitations on things such as parallel job execution all the way up to 1,000 users for $7,833.26 USD/month. You can find the latest pricing information as part of the product information and compare features of different plans.

You can just use Azure Pipelines on their own, you don’t have to use all of the DevOps services (Boards, etc.).

10 free parallel jobs and unlimited build minutes for all open source projects

Advanced workflow features, testing, reporting, gates, etc.

Azure DevOps Organizations and Projects

Once you’ve signed up for Azure DevOps you’ll need an organization and a project.

“..an organization is a mechanism for organizing and connecting groups of related projects. Examples are business divisions, regional divisions, or other organizational structure. You can choose one organization for your entire company, or separate organizations for specific business units, or an organization just for you.” [Microsoft]

“Each organization contains one or more projects. Each project contains a set of features: boards and backlogs for agile planning, pipelines for continuous integration and deployment, repos for version control and management of source code and artifacts, and continuous test integration throughout the life cycle.” [Microsoft]

Once the project is created, you can navigate to Pipelines section to create and manage build and release pipelines.

In the next part of this series we’ll create an initial build pipeline and get an introduction to defining source-controlled build definitions using YAML.

InvestFunctionApp Overview

The InvestFunctionApp contains a number of functions that allow the investing of money in a portfolio. The initial entry point is a HTTP-triggered function that allows the investor id to be specified along with how much to add to the investors portfolio.Depending on the investors target allocations, the amount will be investing in either bonds or shares.

With over 15 years experience, Jason Roberts is a former 5-time Microsoft .NET MVP, freelance developer, writer, and Pluralsight course author. He has written multiple books and is an open source contributor. In addition to enterprise software development, he has also designed and developed both Windows Phone and Windows Store apps.