OAuth2 Support

Usage of Azure Data Lake Storage requires OAuth2 bearer token to be present as part of the HTTPS header as per OAuth2 specification. Valid OAuth2 bearer token should be obtained from Azure Active Directory for valid users who have access to Azure Data Lake Storage Account.

Protecting the Credentials with Credential Providers

In many Hadoop clusters, the core-site.xml file is world-readable. To protect these credentials from prying eyes, it is recommended that you use the credential provider framework to securely store them and access them through configuration.

All ADLS credential properties can be protected by credential providers. For additional reading on the credential provider API, see Credential Provider API.

NOTE: You may optionally add the provider path property to the distcp command line instead of added job specific configuration to a generic core-site.xml. The square brackets above illustrate this capability.

User/Group Representation

The hadoop-azure-datalake module provides support for configuring how User/Group information is represented during getFileStatus/listStatus/getAclStatus.

Add the following properties to your core-site.xml

<property>
<name>adl.feature.ownerandgroup.enableupn</name>
<value>true</value>
<description>
When true : User and Group in FileStatus/AclStatus response is
represented as user friendly name as per Azure AD profile.
When false (default) : User and Group in FileStatus/AclStatus
response is represented by the unique identifier from Azure AD
profile (Object ID as GUID).
For performance optimization, Recommended default value.
</description>
</property>

Testing the azure-datalake-store Module

The hadoop-azure module includes a full suite of unit tests. Most of the tests will run without additional configuration by running mvn test. This includes tests against mocked storage, which is an in-memory emulation of Azure Data Lake Storage.

A selection of tests can run against the Azure Data Lake Storage. To run these tests, please create src/test/resources/auth-keys.xml with Adl account information mentioned in the above sections and the following properties.