Before you use the executor, you must perform some prerequisite tasks.

You cannot perform multiple tasks in the same executor. To perform more than one task,
use additional executors.

Use the ADLS Gen1 File Metadata executor as part of an event stream. For example, you
might use the executor to move a file or change file permissions after it receives a
file closure event from the Azure Data Lake Storage Gen1 destination.

When changing metadata, you configure an expression that represents the location and name
of the file to process, and then specify the changes you want to perform. When creating
an empty file, you specify the output location for the file, and can optionally specify
the owner, permissions, and ACLs for the file. When removing a file or directory, you
specify the location of the file or directory.

When necessary, you can configure advanced properties to pass to the underlying Hadoop
file system.

You can also configure the executor to generate events for
another event stream. For more information about dataflow
triggers and the event framework, see Dataflow Triggers Overview.

For information on
accessing this from the Azure portal, see "Step 4:
Get the OAuth 2.0 token endpoint" in the Azure
documentation.

Application Key - Authentication key for the Azure Active Directory
Data Collector application. Also known as the client
key.

For information on
accessing this from the Azure portal, see "Get
values for signing in" in the Azure
documentation.

Related Event Generating Stages

Use the ADLS Gen1 File Metadata executor in the
event stream of a pipeline. The ADLS Gen1 File Metadata executor is optimized to update
file metadata for output files or whole files processed by the Azure Data Lake Storage
Gen1 destination or another ADLS Gen1 File Metadata executor. However, you can use the
executor in any logical way.

Changing Metadata

You can configure the ADLS Gen1 File Metadata
executor to change metadata for a file in Azure Data Lake Storage Gen1 after receiving
an event. For example, you might use the executor to change file permissions after a
destination closes a file.

When changing file metadata, the ADLS Gen1 File Metadata executor can change the
following file metadata at the same time:

File name

File location

File owner and group

File permissions

Access control lists (ACLs)

To change metadata, the Azure Active Directory application for Data Collector
must have the required permission.

Specifying the File Path

When using the ADLS Gen1 File Metadata executor to change
file metadata, specify an expression for the File Path property that provides an
absolute path to the files you want to use.

Use the default file path expression, ${record:value('/filepath')}, to
update output files closed by the Azure Data Lake Storage Gen1 destination. The file
closure event records generated by this destination include a filepath
field that contains the location and name of the closed output files.

To update whole files that the Azure Data Lake Storage Gen1 destination has completed
streaming, use the following
expression:

${record:value('/targetFileInfo/path')}

The whole file processed event records from the destination include a
/targetFileInfo/path field that contains the location and name of
the processed whole files.

For more information about the event records generated by the Azure Data Lake Storage
Gen1 destination, see Event Records.

Changing the File Name or Location

When using the ADLS Gen1 File Metadata executor
to change file metadata, you can change the name or location of files after they close.
When you specify new file names and locations, you can enter constants or expressions.
Use any expression that evaluates to the values that you want to use.

When needed, you can include file functions in expressions to specify part of the
existing file path. File functions can return any part of a path, file name, or
extension.

After the files are written, you want the ADLS Gen1 File Metadata executor
to move the files to a different root directory. When moving files you need
to specify the new location for the files. So you configure the executor to
move files to a different directory while still using the rest of the path,
as
follows:

Defining the Owner, Group, Permissions, and ACLs

When using the ADLS Gen1 File Metadata executor
to change file metadata or create an empty file, you can define the file owner, group,
file permissions, and the access control list (ACL).

Important: When the
executor changes permissions, it removes existing permissions and implements the
requested permissions. The executor does not add permissions to the existing
permissions, so be sure to configure permissions exactly how you want
them.

You can set permissions using any combination of the following methods:

Define a new owner and group

You can define the owner and group for files. When you use this option, you
must enter both an owner and a group name.

Set file permissions using the octal or symbolic formats

You can set file permissions by entering the permissions you want to use in
octal or symbolic format.

For example, you can use the following octal format to make files
read-only:

0444

You can alternatively use the following
symbolic format to make files
read-only:

-r--r--r--

To make them read-only for the user and the group, forbidding all access to
other users, you could use either of the following
formats:

0440
-r--r-----

Define ACLs

You can define the ACLs for files. When you define ACLs, note that Azure
Data Lake Storage expects permissions defined for the user, group, and
other. You can alternatively add permissions for additional users or groups.

Define permissions using the symbolic format, with r, w, x, or -
representing the permission type.

For example, the following ACLs allow read access for the user and group
only:

user::r--,group::r--,other::–-

If you wanted to allow read access to the operations group in addition to
the group associated with the file, you would enter the following
permissions:

user::r--,group::r--,other::–-,group:operations:r--

Creating an Empty File

You can configure the ADLS Gen1 File Metadata executor
to create empty files in Azure Data Lake Storage Gen1 upon receiving an event. You might
create empty files to trigger downstream actions in other applications, such as Oozie.

To create an empty file, specify an expression for the File Path property that provides
an absolute path to the location where you want the file created.

Note: In most cases, you will not want to use the
default expression. The default expression is more appropriate for changing
file metadata.

When creating an empty file, you can also specify the following file details:

Removing a File or Directory

You can configure the ADLS Gen1 File Metadata executor
to remove a file or directory from Azure Data Lake Storage Gen1 after receiving an
event.

For example, say you run a daily pipeline that writes data to Azure Data Lake Storage
Gen1. You can use the ADLS Gen1 File Metadata executor to remove the target directory
and all of its contents before a pipeline starts processing data. Simply configure the
pipeline to pass the pipeline start event to an ADLS Gen1 File Metadata executor, then
specify the target directory when you configure the executor. For more information about
using pipeline events, see Pipeline Event Generation.

Remove directories with caution. The executor removes directories recursively, deleting
any subdirectories and their contents in addition to the specified directory.

Event Generation

The ADLS Gen1 File Metadata
executor can generate events that you can use in an event stream. When you enable event
generation, the executor generates events each time it changes file metadata, creates an
empty file, or removes a file or directory.

ADLS Gen1 File Metadata events can be used in any logical way. For example:

With the Email executor to send a custom email
after receiving an event.

By default, the property uses
${record:value('/filepath')}, which
processes data in the filepath field.
The Azure Data Lake Storage Gen1 destination generates
file-closure event records that include the path to
closed files in a filepath
field.

To update whole files that the Azure Data
Lake Storage Gen1 destination has completed streaming,
use the following
expression: