Usage

Concepts

OAuth2 Support

Usage of Azure Data Lake Storage requires an OAuth2 bearer token to be present as part of the HTTPS header as per the OAuth2 specification. A valid OAuth2 bearer token must be obtained from the Azure Active Directory service for those valid users who have access to Azure Data Lake Storage Account.

Azure Active Directory (Azure AD) is Microsoft’s multi-tenant cloud based directory and identity management service. See What is ActiveDirectory.

Using MSI (Managed Service Identity)

Azure VMs can be provisioned with “service identities” that are managed by the Identity extension within the VM. The advantage of doing this is that the credentials are managed by the extension, and do not have to be put into core-site.xml.

To use MSI, modify the VM deployment template to use the identity extension. Note the port number you specified in the template: this is the port number for the REST endpoint of the token service exposed to localhost by the identity extension in the VM. The default recommended port number is 50342 - if the recommended port number is used, then the msi.port setting below can be omitted in the configuration.

Configure core-site.xml

Add the following properties to your core-site.xml

<property>
<name>fs.adl.oauth2.access.token.provider.type</name>
<value>Msi</value>
</property>
<property>
<name>fs.adl.oauth2.msi.port</name>
<value>PORT NUMBER FROM ABOVE (if different from the default of 50342)</value>
</property>

Using Device Code Auth for interactive login

Note: This auth method is suitable for running interactive tools, but will not work for jobs submitted to a cluster.

Running this will print a URL and device code that can be used to login from any browser (even on a different machine, outside of the ssh session). Once the login is done, the command continues.

Protecting the Credentials with Credential Providers

In many Hadoop clusters, the core-site.xml file is world-readable. To protect these credentials, it is recommended that you use the credential provider framework to securely store them and access them.

All ADLS credential properties can be protected by credential providers. For additional reading on the credential provider API, see Credential Provider API.

NOTE: You may optionally add the provider path property to the distcp command line instead of added job specific configuration to a generic core-site.xml. The square brackets above illustrate this capability.`

Accessing adl URLs

After credentials are configured in core-site.xml, any Hadoop component may reference files in that Azure Data Lake Storage account by using URLs of the following format:

adl://<Account Name>.azuredatalakestore.net/<path>

The schemes adl identifies a URL on a Hadoop-compatible file system backed by Azure Data Lake Storage. adl utilizes encrypted HTTPS access for all interaction with the Azure Data Lake Storage API.

For example, the following FileSystem Shell commands demonstrate access to a storage account named youraccount.

User/Group Representation

The hadoop-azure-datalake module provides support for configuring how User/Group information is represented during getFileStatus(), listStatus(), and getAclStatus() calls..

Add the following properties to core-site.xml

<property>
<name>adl.feature.ownerandgroup.enableupn</name>
<value>true</value>
<description>
When true : User and Group in FileStatus/AclStatus response is
represented as user friendly name as per Azure AD profile.
When false (default) : User and Group in FileStatus/AclStatus
response is represented by the unique identifier from Azure AD
profile (Object ID as GUID).
For performance optimization, Recommended default value.
</description>
</property>

Testing the azure-datalake-store Module

The hadoop-azure module includes a full suite of unit tests. Most of the tests will run without additional configuration by running mvn test. This includes tests against mocked storage, which is an in-memory emulation of Azure Data Lake Storage.

A selection of tests can run against the Azure Data Lake Storage. To run these tests, please create src/test/resources/auth-keys.xml with Adl account information mentioned in the above sections and the following properties.