Purpose

This document describes, at a high-level, the way to implement new Applications for YARN.

Concepts and Flow

The general concept is that an 'Application Submission Client' submits an 'Application' to the YARN Resource Manager. The client communicates with the ResourceManager using the 'ClientRMProtocol' to first acquire a new 'ApplicationId' if needed via ClientRMProtocol#getNewApplication and then submit the 'Application' to be run via ClientRMProtocol#submitApplication. As part of the ClientRMProtocol#submitApplication call, the client needs to provide sufficient information to the ResourceManager to 'launch' the application's first container i.e. the ApplicationMaster. You need to provide information such as the details about the local files/jars that need to be available for your application to run, the actual command that needs to be executed (with the necessary command line arguments), any Unix environment settings (optional), etc. Effectively, you need to describe the Unix process(es) that needs to be launched for your ApplicationMaster.

Meanwhile, the client can monitor the application's status by querying the ResourceManager or by directly querying the ApplicationMaster if it supports such a service. If needed, it can also kill the application via ClientRMProtocol#forceKillApplication.

Interfaces

The protocol for a client that wishes to communicate with the ResourceManager to launch a new application (i.e. the ApplicationMaster), check on the status of the application or kill the application. For example, a job-client (a job launching program from the gateway) would use this protocol.

The response from the ASM for a new application also contains information about the cluster such as the minimum/maximum resource capabilities of the cluster. This is required so that to ensure that you can correctly set the specifications of the container in which the ApplicationMaster would be launched. Please refer to GetNewApplicationResponse for more details.

Queue, Priority info: Queue to which the application will be submitted, the priority to be assigned for the application.

User: The user submitting the application

ContainerLaunchContext: The information defining the container in which the ApplicationMaster will be launched and run. The ContainerLaunchContext, as mentioned previously, defines all the required information needed to run the ApplicationMaster such as the local resources (binaries, jars, files etc.), security tokens, environment settings (CLASSPATH etc.) and the command to be executed.

1// Create a new ApplicationSubmissionContext 2ApplicationSubmissionContextappContext = 3Records.newRecord(ApplicationSubmissionContext.class); 4// set the ApplicationId 5appContext.setApplicationId(appId); 6// set the application name 7appContext.setApplicationName(appName); 8 9// Create a new container launch context for the AM's container 10ContainerLaunchContextamContainer = 11Records.newRecord(ContainerLaunchContext.class); 12 13// Define the local resources required 14Map<String, LocalResource> localResources = 15newHashMap<String, LocalResource>(); 16// Lets assume the jar we need for our ApplicationMaster is available in 17// HDFS at a certain known path to us and we want to make it available to 18// the ApplicationMaster in the launched container 19PathjarPath; // <- known path to jar file 20FileStatusjarStatus = fs.getFileStatus(jarPath); 21LocalResourceamJarRsrc = Records.newRecord(LocalResource.class); 22// Set the type of resource - file or archive 23// archives are untarred at the destination by the framework 24amJarRsrc.setType(LocalResourceType.FILE); 25// Set visibility of the resource 26// Setting to most private option i.e. this file will only 27// be visible to this instance of the running application 28amJarRsrc.setVisibility(LocalResourceVisibility.APPLICATION); 29// Set the location of resource to be copied over into the 30// working directory 31amJarRsrc.setResource(ConverterUtils.getYarnUrlFromPath(jarPath)); 32// Set timestamp and length of file so that the framework 33// can do basic sanity checks for the local resource 34// after it has been copied over to ensure it is the same 35// resource the client intended to use with the application 36amJarRsrc.setTimestamp(jarStatus.getModificationTime()); 37amJarRsrc.setSize(jarStatus.getLen()); 38// The framework will create a symlink called AppMaster.jar in the 39// working directory that will be linked back to the actual file. 40// The ApplicationMaster, if needs to reference the jar file, would 41// need to use the symlink filename. 42localResources.put("AppMaster.jar", amJarRsrc); 43// Set the local resources into the launch context 44amContainer.setLocalResources(localResources); 45 46// Set up the environment needed for the launch context 47Map<String, String> env = newHashMap<String, String>(); 48// For example, we could setup the classpath needed. 49// Assuming our classes or jars are available as local resources in the 50// working directory from which the command will be run, we need to append 51// "." to the path. 52// By default, all the hadoop specific classpaths will already be available 53// in $CLASSPATH, so we should be careful not to overwrite it. 54StringclassPathEnv = "$CLASSPATH:./*:"; 55env.put("CLASSPATH", classPathEnv); 56amContainer.setEnvironment(env); 57 58// Construct the command to be executed on the launched container 59Stringcommand = 60"${JAVA_HOME}" + /bin/java" + 61 "MyAppMaster" + 62 "arg1arg2arg3" + 63 "1>" + ApplicationConstants.LOG_DIR_EXPANSION_VAR + "/stdout" + 64 "2>" + ApplicationConstants.LOG_DIR_EXPANSION_VAR + "/stderr"; 65 66List<String> commands = newArrayList<String>(); 67commands.add(command); 68// add additional commands if needed 69 70// Set the command array into the container spec 71amContainer.setCommands(commands); 72 73// Define the resource requirements for the container 74// For now, YARN only supports memory so we set the memory 75// requirements. 76// If the process takes more than its allocated memory, it will 77// be killed by the framework. 78// Memory being requested for should be less than max capability 79// of the cluster and all asks should be a multiple of the min capability. 80Resourcecapability = Records.newRecord(Resource.class); 81capability.setMemory(amMemory); 82amContainer.setResource(capability); 83 84// Set the container launch content into the ApplicationSubmissionContext 85appContext.setAMContainerSpec(amContainer);

After the setup process is complete, the client is finally ready to submit the application to the ASM.

1// Create the request to send to the ApplicationsManager 2SubmitApplicationRequestappRequest = 3Records.newRecord(SubmitApplicationRequest.class); 4appRequest.setApplicationSubmissionContext(appContext); 5 6// Submit the application to the ApplicationsManager 7// Ignore the response as either a valid response object is returned on 8// success or an exception thrown to denote the failure 9applicationsManager.submitApplication(appRequest);

At this point, the ResourceManager will have accepted the application and in the background, will go through the process of allocating a container with the required specifications and then eventually setting up and launching the ApplicationMaster on the allocated container.

There are multiple ways a client can track progress of the actual task.

It can communicate with the ResourceManager and request for a report of the application via ClientRMProtocol#getApplicationReport.

Application tracking information: If the application supports some form of progress tracking, it can set a tracking url which is available via ApplicationReport#getTrackingUrl that a client can look at to monitor progress.

If the ApplicationMaster supports it, a client can directly query the ApplicationMaster itself for progress updates via the host:rpcport information obtained from the ApplicationReport. It can also use the tracking url obtained from the report if available.

In certain situations, if the application is taking too long or due to other factors, the client may wish to kill the application. The ClientRMProtocol supports the forceKillApplication call that allows a client to send a kill signal to the ApplicationMaster via the ResourceManager. An ApplicationMaster if so designed may also support an abort call via its rpc layer that a client may be able to leverage.

Writing an ApplicationMaster

The ApplicationMaster is the actual owner of the job. It will be launched by the ResourceManager and via the client will be provided all the necessary information and resources about the job that it has been tasked with to oversee and complete.

As the ApplicationMaster is launched within a container that may (likely will) be sharing a physical host with other containers, given the multi-tenancy nature, amongst other issues, it cannot make any assumptions of things like pre-configured ports that it can listen on.

1// Connect to the Scheduler of the ResourceManager. 2YarnConfigurationyarnConf = newYarnConfiguration(conf); 3InetSocketAddressrmAddress = 4NetUtils.createSocketAddr(yarnConf.get( 5YarnConfiguration.RM_SCHEDULER_ADDRESS, 6YarnConfiguration.DEFAULT_RM_SCHEDULER_ADDRESS)); 7LOG.info("Connecting to ResourceManager at " + rmAddress); 8AMRMProtocolresourceManager = 9 (AMRMProtocol) rpc.getProxy(AMRMProtocol.class, rmAddress, conf); 10 11// Register the AM with the RM 12// Set the required info into the registration request: 13// ApplicationAttemptId, 14// host on which the app master is running 15// rpc port on which the app master accepts requests from the client 16// tracking url for the client to track app master progress 17RegisterApplicationMasterRequestappMasterRequest = 18Records.newRecord(RegisterApplicationMasterRequest.class); 19appMasterRequest.setApplicationAttemptId(appAttemptID); 20appMasterRequest.setHost(appMasterHostname); 21appMasterRequest.setRpcPort(appMasterRpcPort); 22appMasterRequest.setTrackingUrl(appMasterTrackingUrl); 23 24 25// The registration response is useful as it provides information about the 26// cluster. 27// Similar to the GetNewApplicationResponse in the client, it provides 28// information about the min/mx resource capabilities of the cluster that 29// would be needed by the ApplicationMaster when requesting for containers. 30RegisterApplicationMasterResponseresponse = 31resourceManager.registerApplicationMaster(appMasterRequest);

The ApplicationMaster has to emit heartbeats to the ResourceManager to keep it informed that the ApplicationMaster is alive and still running. The timeout expiry interval at the ResourceManager is defined by a config setting accessible via YarnConfiguration.RM_AM_EXPIRY_INTERVAL_MS with the default being defined by YarnConfiguration.DEFAULT_RM_AM_EXPIRY_INTERVAL_MS. The AMRMProtocol#allocate calls to the ResourceManager count as heartbeats as it also supports sending progress update information. Therefore, an allocate call with no containers requested and progress information updated if any is a valid way for making heartbeat calls to the ResourceManager.

Hostname: If containers are required to be hosted on a particular rack or a specific host. '*' is a special value that implies any host will do.

Resource capability: Currently, YARN only supports memory based resource requirements so the request should define how much memory is needed. The value is defined in MB and has to less than the max capability of the cluster and an exact multiple of the min capability.

Priority: When asking for sets of containers, an ApplicationMaster may define different priorities to each set. For example, the Map-Reduce ApplicationMaster may assign a higher priority to containers needed for the Map tasks and a lower priority for the Reduce tasks' containers.

Released containers: There may be situations when the ApplicationMaster may have requested for more containers that it needs or due to failure issues, decide to use other containers allocated to it. In all such situations, it is beneficial to the cluster if the ApplicationMaster releases these containers back to the ResourceManager so that they can be re-allocated to other applications.

ResponseId: The response id that will be sent back in the response from the allocate call.

1List<ResourceRequest> requestedContainers; 2List<ContainerId> releasedContainers 3AllocateRequestreq = Records.newRecord(AllocateRequest.class); 4 5// The response id set in the request will be sent back in 6// the response so that the ApplicationMaster can 7// match it to its original ask and act appropriately. 8req.setResponseId(rmRequestID); 9 10// Set ApplicationAttemptId 11req.setApplicationAttemptId(appAttemptID); 12 13// Add the list of containers being asked for 14req.addAllAsks(requestedContainers); 15 16// If the ApplicationMaster has no need for certain 17// containers due to over-allocation or for any other 18// reason, it can release them back to the ResourceManager 19req.addAllReleases(releasedContainers); 20 21// Assuming the ApplicationMaster can track its progress 22req.setProgress(currentProgress); 23 24AllocateResponseallocateResponse = resourceManager.allocate(req);

Allocated containers: The containers that have been allocated to the ApplicationMaster.

Headroom: Headroom for resources in the cluster. Based on this information and knowing its needs, an ApplicationMaster can make intelligent decisions such as re-prioritizing sub-tasks to take advantage of currently allocated containers, bailing out faster if resources are not becoming available etc.

Completed containers: Once an ApplicationMaster triggers a launch an allocated container, it will receive an update from the ResourceManager when the container completes. The ApplicationMaster can look into the status of the completed container and take appropriate actions such as re-trying a particular sub-task in case of a failure.

One thing to note is that containers will not be immediately allocated to the ApplicationMaster. This does not imply that the ApplicationMaster should keep on asking the pending count of required containers. Once an allocate request has been sent, the ApplicationMaster will eventually be allocated the containers based on cluster capacity, priorities and the scheduling policy in place. The ApplicationMaster should only request for containers again if and only if its original estimate changed and it needs additional containers.

The ApplicationMaster, as mentioned previously, will get updates of completed containers as part of the response from the AMRMProtocol#allocate calls. It can also monitor its launched containers pro-actively by querying the ContainerManager for the status.

FAQ

How can I distribute my application's jars to all of the nodes in the YARN cluster that need it?

You can use the LocalResource to add resources to your application request. This will cause YARN to distribute the resource to the ApplicationMaster node. If the resource is a tgz, zip, or jar - you can have YARN unzip it. Then, all you need to do is add the unzipped folder to your classpath. For example, when creating your application request:

As you can see, the setLocalResources command takes a map of names to resources. The name becomes a sym link in your application's cwd, so you can just refer to the artifacts inside by using ./package/*. Note: Java's classpath (cp) argument is VERY sensitive. Make sure you get the syntax EXACTLY correct.

Once your package is distributed to your ApplicationMaster, you'll need to follow the same process whenever your ApplicationMaster starts a new container (assuming you want the resources to be sent to your container). The code for this is the same. You just need to make sure that you give your ApplicationMaster the package path (either HDFS, or local), so that it can send the resource URL along with the container ctx.

My container is being killed by the Node Manager

This is likely due to high memory usage exceeding your requested container memory size. There are a number of reasons that can cause this. First, look at the process tree that the node manager dumps when it kills your container. The two things you're interested in are physical memory and virtual memory. If you have exceeded physical memory limits your app is using too much physical memory. If you're running a Java app, you can use -hprof to look at what is taking up space in the heap. If you have exceeded virtual memory, things are slightly more complicated.

How can my ApplicationMaster kill a container? Releasing it via AMRMProtocol#allocate does not seem to work.

A container can only be released back to the ResourceManager if it has not been launched. To kill a launched container, the ApplicationMaster can send a stop command to the container via ContainerManager#stopContainer(StopContainerRequest request). This will trigger a kill event to the launched container and this container will eventually be part of the list of completed Containers in the RM's response to the AM on an AMRMProtocol#allocate call.