Cognitive Services with PowerApps using Custom Connectors

Microsoft Cognitive Services are a set of APIs, SDKs and services available to developers to make their applications more intelligent, engaging and discoverable. This blog is an attempt to share an approach for PowerApps makers to use these Cognitive Services using custom connectors.

There is an out-of-the-box Computer Vision API connector, however the API uses dynamic types, which are not supported in PowerApps yet. There is also an out-of-the-box Face API connector, however it isn’t much useful in enterprise scenarios yet, as it’s an API Key based connector and when a PowerApps app created using this connector is shared with other users, each user has to bring in their own API keys for using the app.

Then, we shall register a new custom connector in PowerApps using the swagger definition from the API App.

Finally, we shall create the PowerApps app using this new custom connector.

In PowerApps, when a maker shares an app created using a custom connector, the connector is automatically shared and hence other users can use the app (without needing to type in an API key). In the blog below, we shall also protect our API app endpoint with AAD and also register an app in Azure AD for the custom connector, so that other users can use their own AAD credentials to authenticate and get access to the APIs.

I have chosen to use an API App for the custom connector versus Azure Functions primarily as API apps can generate the Swagger document for use with PowerApps using the Swashbuckle Nuget package. This auto generates the definitions for all the reference classes like AnalysisResult and others used by the APIs in the swagger document. Since I have used the Swashbuckle feature to include XML Comments from my assembly to generate Summary and Description tags in swagger, it becomes quite handy for both the API registration and also for providing intelli-sense when authoring the functions from the PowerApps app.

You will need the following Azure Services setup before you can complete this step:

If you haven’t created an API App for use with PowerApps, I recommend reading this tutorial to familiarize with the basic concepts. I have used Visual Studio 2017 for the code used in this app. Extract the contents of the zip file and open the ImageUtilities.sln file to launch the project in Visual Studio. We have added the following Controllers to correspond to the respective APIs from Azure Cognitive Services. The list of APIs for reference is given below:

Follow the steps mentioned in this article to secure your API with Azure Active Directory. If your Azure Subscription is in a different directory than your organizational Azure AD (like I had), follow the steps mentioned in the Alternative Method but make sure that you do the app registration in your organization’s Active Directory (Tip: you can open the AAD specific https://aad.portal.azure.com portal in another browser tab or window). I have registered my app as ImageUtilitiesAPI in my active directory.

We now need to register a second application in AAD for the PowerApps connector. This is required to secure the PowerApps custom connector with AAD so that other users within the organization can use their own AAD credentials to authenticate against the connector for the app. I have registered this app as ImageUtilities_PAConnector in my Azure Active Directory.

The second AAD application is used to secure the custom connector registration and acquire delegated access to the ImageUtilitiesAPI app protected by the first application. Name this one ImageUtilities_PAConnector.

Sign-on URL: https://login.windows.net

Reply URL: https://msmanaged-na.consent.azure-apim.net/redirect

Add permissions to have delegated access to ImageUtilitiesAPI.

You need the application ID of this application later as well, so note it.

Generate a client key and store is somewhere safe. We need this key later.

Once you have uploaded the OpenAPI (Swagger) file, the wizard auto-detects that you are using the AAD authentication for your web API. (If you are curious how, we have already added a security definitions in the SwaggerConfig file itself, using the c.OAuth2("oauth2") method)

Open the PowerApps Portal. Make sure that you have the same Environment selected as the one in which you created the Custom connector in the previous step.

Click on Apps from the left nav bar.

Download the PowerApps app resources from this location: https://aka.ms/CognitiveServicesPowerAppsDemo. Right click the downloaded zip file (on windows) and select Properties to unblock the contents by checking the Unblock checkbox, before extracting the contents of the zip file.

Then click on Create an App from the top right. This will navigate you to the PowerApps web authoring site. Click on Open from the left nav, then click on Browse.

Then select the CognitiveDemos.msapp file that you extracted from the downloaded zip file. This will load the PowerApps app in edit mode.

Click on View > DataSources

Click on + Add data source

Click on + New connection, and select the ImageUtilitiesAPI (custom connector that you created earlier). Authenticate using your AAD credentials to add it to your application. In case you are wondering the other two connections already present in the app, they are static data files imported from Excel. The source of the file is shared along with the PowerApps app file AppData.xlsx.