YARN-2747.
Major test reported by Xuan Gong and fixed by Xuan Gong TestAggregatedLogFormat fails in trunk

YARN-2744.
Critical sub-task reported by Sumit Mohanty and fixed by Wangda Tan (capacityscheduler)Under some scenario, it is possible to end up with capacity scheduler configuration that uses labels that no longer exist

YARN-2743.
Blocker bug reported by Arpit Gupta and fixed by Jian He (resourcemanager)Yarn jobs via oozie fail with failed to renew token (secure) or digest mismatch (unsecure) errors when RM is being killed

YARN-2741.
Major bug reported by Craig Welch and fixed by Craig Welch (nodemanager)Windows: Node manager cannot serve up log files via the web user interface when yarn.nodemanager.log-dirs to any drive letter other than C: (or, the drive that nodemanager is running on)

YARN-2734.
Major bug reported by Sumit Mohanty and fixed by Xuan Gong (log-aggregation)If a sub-folder is encountered by log aggregator it results in invalid aggregated file

YARN-2732.
Major bug reported by Jian He and fixed by Jian He Fix syntax error in SecureContainer.apt.vm

YARN-2730.
Critical bug reported by Siqi Li and fixed by Siqi Li DefaultContainerExecutor runs only one localizer at a time

YARN-2726.
Minor sub-task reported by Phil D'Amore and fixed by Wangda Tan (capacityscheduler)CapacityScheduler should explicitly log when an accessible label has no capacity

YARN-2724.
Major bug reported by Sumit Mohanty and fixed by Xuan Gong (log-aggregation)If an unreadable file is encountered during log aggregation then aggregated file in HDFS badly formed

YARN-2723.
Major sub-task reported by Phil D'Amore and fixed by Naganarasimha G R (client)rmadmin -replaceLabelsOnNode does not correctly parse port

MAPREDUCE-6073.
Trivial bug reported by Tsuyoshi OZAWA and fixed by Tsuyoshi OZAWA (documentation)Description of mapreduce.job.speculative.slowtaskthreshold in mapred-default should be moved into description tags

HDFS-7359.
Major bug reported by Chris Nauroth and fixed by Chris Nauroth (journal-node , namenode)NameNode in secured HA cluster fails to start if dfs.namenode.secondary.http-address cannot be interpreted as a network address.

HDFS-7355.
Trivial test reported by Chris Nauroth and fixed by Chris Nauroth (test)TestDataNodeVolumeFailure#testUnderReplicationAfterVolFailure fails on Windows, because we cannot deny access to the file owner.

HDFS-6755.
Major improvement reported by Mit Desai and fixed by Mit Desai There is an unnecessary sleep in the code path where DFSOutputStream#close gives up its attempt to contact the namenode

HDFS-6754.
Major bug reported by Mit Desai and fixed by Mit Desai TestNamenodeCapacityReport.testXceiverCount may sometimes fail due to lack of retry

HDFS-6750.
Major sub-task reported by Colin Patrick McCabe and fixed by Colin Patrick McCabe (datanode , hdfs-client)The DataNode should use its shared memory segment to mark short-circuit replicas that have been unlinked as stale

HDFS-6511.
Minor improvement reported by Juan Yu and fixed by Juan Yu BlockManager#computeInvalidateWork() could do nothing

HDFS-6506.
Major bug reported by Binglin Chang and fixed by Binglin Chang (balancer & mover , test)Newly moved block replica been invalidated and deleted in TestBalancer

HDFS-6482.
Major improvement reported by James Thomas and fixed by James Thomas (datanode)Use block ID-based block layout on datanodes

The directory structure for finalized replicas on DNs has been changed. Now, the directory that a finalized replica goes in is determined uniquely by its ID. Specifically, we use a two-level directory structure, with the 24th through 17th bits identifying the correct directory at the first level and the 16th through 8th bits identifying the correct directory at the second level.

HDFS-6478.
Major bug reported by Ming Ma and fixed by Ming Ma RemoteException can't be retried properly for non-HA scenario

HDFS-6456.
Major bug reported by Yesha Vora and fixed by Abhiraj Butala (nfs)NFS should throw error for invalid entry in dfs.nfs.exports.allowed.hosts

HDFS-6455.
Major bug reported by Yesha Vora and fixed by Abhiraj Butala (nfs)NFS: Exception should be added in NFS log for invalid separator in nfs.exports.allowed.hosts

HDFS-6451.
Major bug reported by Brandon Li and fixed by Abhiraj Butala (nfs)NFS should not return NFS3ERR_IO for AccessControlException

Allow distcp to copy data between HA clusters. Users can use a new configuration property "dfs.internal.nameservices" to explicitly specify the name services belonging to the local cluster, while continue using the configuration property "dfs.nameservices" to specify all the name services in the local and remote clusters.

HDFS-6188.
Major improvement reported by Benoy Antony and fixed by Benoy Antony (security)An ip whitelist based implementation of TrustedChannelResolver

HDFS-6134.
Major new feature reported by Alejandro Abdelnur and fixed by Charles Lamb (security)Transparent data at rest encryption

HDFS-6114.
Critical bug reported by Vinayakumar B and fixed by Vinayakumar B (datanode)Block Scan log rolling will never happen if blocks written continuously leading to huge size of dncp_block_verification.log.curr

HDFS-6036.
Major sub-task reported by Colin Patrick McCabe and fixed by Colin Patrick McCabe (caching , datanode)Forcibly timeout misbehaving DFSClients that try to do no-checksum reads that extend too long

SASL now can be used to secure the DataTransferProtocol, which transfers file block content between HDFS clients and DataNodes. In this configuration, it is no longer required for secured clusters to start the DataNode as root and bind to privileged ports.

HDFS-573.
Major improvement reported by Ziliang Guo and fixed by Chris Nauroth (libhdfs)Porting libhdfs to Windows

HADOOP-11228.
Major bug reported by Remus Rusanu and fixed by Remus Rusanu winutils task: unsecure path should not call AddNodeManagerAndUserACEsToObject

HADOOP-11221.
Major bug reported by Jinghui Wang and fixed by Jinghui Wang (util)JAVA specification for hashcode does not enforce it to be non-negative, but IdentityHashStore assumes System.identityHashCode() is non-negative

HADOOP-11217.
Blocker bug reported by Robert Kanter and fixed by Robert Kanter (kms)Disable SSLv3 in KMS

HADOOP-11181.
Major bug reported by Zhijie Shen and fixed by Zhijie Shen (security)o.a.h.security.token.delegation.DelegationTokenManager should be more generalized to handle other DelegationTokenIdentifier

HADOOP-11179.
Major bug reported by Hitesh Shah and fixed by Craig Welch Tarball as local resource type archive fails to localize on Windows

The "hadoop classpath" command has been enhanced to support options for automatic expansion of wildcards in classpath elements and writing the classpath to a jar file manifest. These options make it easier to construct a correct classpath for libhdfs applications.

HADOOP-10902.
Minor improvement reported by Stephen Chu and fixed by Stephen Chu Deletion of directories with snapshots will not output reason for trash move failure

The MetricsSystem abstract class has added a new abstract method, unregisterSource, for unregistering a previously registered metrics source. Custom subclasses of MetricsSystem must be updated to provide an implementation of this method.

HADOOP-10838.
Major improvement reported by James Thomas and fixed by James Thomas (performance)Byte array native checksumming

HADOOP-10793.
Major improvement reported by Mike Yoder and fixed by Andrew Wang (security)KeyShell args should use single-dash style

HADOOP-10791.
Major improvement reported by Alejandro Abdelnur and fixed by Robert Kanter (security)AuthenticationFilter should support externalizing the secret for signing and provide rotation support

HADOOP-10431.
Major improvement reported by Alejandro Abdelnur and fixed by Alejandro Abdelnur (security)Change visibility of KeyStore.Options getter methods to public

HADOOP-10430.
Major improvement reported by Alejandro Abdelnur and fixed by Alejandro Abdelnur (security)KeyProvider Metadata should have an optional description, there should be a method to retrieve the metadata from all keys

HADOOP-10429.
Major improvement reported by Alejandro Abdelnur and fixed by Alejandro Abdelnur (security)KeyStores should have methods to generate the materials themselves, KeyShell should use them

HADOOP-10150.
Major new feature reported by Yi Liu and fixed by Yi Liu (security)Hadoop cryptographic file system

HADOOP-10141.
Major bug reported by Owen O'Malley and fixed by Owen O'Malley (security)Create an API to separate encryption key storage from applications

HADOOP-10131.
Major bug reported by Vinayakumar B and fixed by Vinayakumar B NetWorkTopology#countNumOfAvailableNodes() is returning wrong value if excluded nodes passed are not part of the cluster tree

HADOOP-9989.
Major bug reported by Jinghui Wang and fixed by zhihai xu (security , util)Bug introduced in HADOOP-9374, which parses the -tokenCacheFile as binary file but set it to the configuration as JSON file.

HADOOP-9921.
Major bug reported by Vinayakumar B and fixed by Vinayakumar B daemon scripts should remove pid file on stop call after stop or process is found not running

HADOOP-7664.
Minor improvement reported by Ravi Prakash and fixed by Ravi Prakash (conf)o.a.h.conf.Configuration complains of overriding final parameter even if the value with which its attempting to override is the same.

Hadoop 2.4.1 Release Notes

Hadoop 2.4.1 Release Notes

These release notes include new developer and user-facing incompatibilities, features, and major improvements.

Changes since Hadoop 2.4.0

java.lang.AssertionError: expected:<1> but was:<0>
at org.junit.Assert.fail(Assert.java:88)
at org.junit.Assert.failNotEquals(Assert.java:743)
at org.junit.Assert.assertEquals(Assert.java:118)
at org.junit.Assert.assertEquals(Assert.java:555)
at org.junit.Assert.assertEquals(Assert.java:542)
at org.apache.hadoop.yarn.applications.distributedshell.TestDistributedShell.testDSShell(TestDistributedShell.java:198)

YARN-2066.
Minor bug reported by Ted Yu and fixed by Hong Zhiguo Wrong field is referenced in GetApplicationsRequestPBImpl#mergeLocalToBuilder()

{code}
if (this.finish != null) {
builder.setFinishBegin(start.getMinimumLong());
builder.setFinishEnd(start.getMaximumLong());
}
{code}
this.finish should be referenced in the if block.

YARN-2053.
Major sub-task reported by Sumit Mohanty and fixed by Wangda Tan (resourcemanager)Slider AM fails to restart: NPE in RegisterApplicationMasterResponseProto$Builder.addAllNmTokensFromPreviousAttempts

YARN-2016.
Major bug reported by Venkat Ranganathan and fixed by Junping Du (resourcemanager)Yarn getApplicationRequest start time range is not honored

When we query for the previous applications by creating an instance of GetApplicationsRequest and setting the start time range and application tag, we see that the start range provided is not honored and all applications with the tag are returned
Attaching a reproducer.

After upgrade from 2.2.0 to 2.4.0, NPE on first job start.
-After RM was restarted, the job runs without a problem.-
{noformat}
19:11:13,441 FATAL ResourceManager:600 - Error in handling event type NODE_UPDATE to the scheduler
java.lang.NullPointerException
at org.apache.hadoop.yarn.server.resourcemanager.scheduler.fifo.FifoScheduler.assignContainers(FifoScheduler.java:462)
at org.apache.hadoop.yarn.server.resourcemanager.scheduler.fifo.FifoScheduler.nodeUpdate(FifoScheduler.java:714)
at org.apache.hadoop.yarn.server.resourcemanager.scheduler.fifo.FifoScheduler.handle(FifoScheduler.java:743)
at org.apache.hadoop.yarn.server.resourcemanager.scheduler.fifo.FifoScheduler.handle(FifoScheduler.java:104)
at org.apache.hadoop.yarn.server.resourcemanager.ResourceManager$SchedulerEventDispatcher$EventProcessor.run(ResourceManager.java:591)
at java.lang.Thread.run(Thread.java:744)
19:11:13,443 INFO ResourceManager:604 - Exiting, bbye..
{noformat}

Used resources displays as &amp;lt;memory:1111, vCores;&amp;gt; with capacity scheduler

YARN-1962.
Major sub-task reported by Mohammad Kamrul Islam and fixed by Mohammad Kamrul Islam Timeline server is enabled by default

Since Timeline server is not matured and secured yet, enabling it by default might create some confusion.
We were playing with 2.4.0 and found a lot of exceptions for distributed shell example related to connection refused error. Btw, we didn't run TS because it is not secured yet.
Although it is possible to explicitly turn it off through yarn-site config. In my opinion, this extra change for this new service is not worthy at this point,.
This JIRA is to turn it off by default.
If there is an agreement, i can put a simple patch about this.
{noformat}
14/04/17 23:24:33 ERROR impl.TimelineClientImpl: Failed to get the response from the timeline server.
com.sun.jersey.api.client.ClientHandlerException: java.net.ConnectException: Connection refused
at com.sun.jersey.client.urlconnection.URLConnectionClientHandler.handle(URLConnectionClientHandler.java:149)
at com.sun.jersey.api.client.Client.handle(Client.java:648)
at com.sun.jersey.api.client.WebResource.handle(WebResource.java:670)
at com.sun.jersey.api.client.WebResource.access$200(WebResource.java:74)
at com.sun.jersey.api.client.WebResource$Builder.post(WebResource.java:563)
at org.apache.hadoop.yarn.client.api.impl.TimelineClientImpl.doPostingEntities(TimelineClientImpl.java:131)
at org.apache.hadoop.yarn.client.api.impl.TimelineClientImpl.putEntities(TimelineClientImpl.java:104)
at org.apache.hadoop.yarn.applications.distributedshell.ApplicationMaster.publishApplicationAttemptEvent(ApplicationMaster.java:1072)
at org.apache.hadoop.yarn.applications.distributedshell.ApplicationMaster.run(ApplicationMaster.java:515)
at org.apache.hadoop.yarn.applications.distributedshell.ApplicationMaster.main(ApplicationMaster.java:281)
Caused by: java.net.ConnectException: Connection refused
at java.net.PlainSocketImpl.socketConnect(Native Method)
at java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:339)
at java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:198)
at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:182)
at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392)
at java.net.Socket.connect(Socket.java:579)
at java.net.Socket.connect(Socket.java:528)
at sun.net.NetworkClient.doConnect(NetworkClient.java:180)
at sun.net.www.http.HttpClient.openServer(HttpClient.java:432)
at sun.net.www.http.HttpClient.openServer(HttpClient.java:527)
at sun.net.www.http.HttpClient.<in14/04/17 23:24:33 ERROR impl.TimelineClientImpl: Failed to get the response from the timeline server.
com.sun.jersey.api.client.ClientHandlerException: java.net.ConnectException: Connection refused
at com.sun.jersey.client.urlconnection.URLConnectionClientHandler.handle(URLConnectionClientHandler.java:149)
at com.sun.jersey.api.client.Client.handle(Client.java:648)
at com.sun.jersey.api.client.WebResource.handle(WebResource.java:670)
at com.sun.jersey.api.client.WebResource.access$200(WebResource.java:74)
at com.sun.jersey.api.client.WebResource$Builder.post(WebResource.java:563)
at org.apache.hadoop.yarn.client.api.impl.TimelineClientImpl.doPostingEntities(TimelineClientImpl.java:131)
at org.apache.hadoop.yarn.client.api.impl.TimelineClientImpl.putEntities(TimelineClientImpl.java:104)
at org.apache.hadoop.yarn.applications.distributedshell.ApplicationMaster.publishApplicationAttemptEvent(ApplicationMaster.java:1072)
at org.apache.hadoop.yarn.applications.distributedshell.ApplicationMaster.run(ApplicationMaster.java:515)
at org.apache.hadoop.yarn.applications.distributedshell.ApplicationMaster.main(ApplicationMaster.java:281)
Caused by: java.net.ConnectException: Connection refused
at java.net.PlainSocketImpl.socketConnect(Native Method)
at java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:339)
at java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:198)
at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:182)
at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392)
at java.net.Socket.connect(Socket.java:579)
at java.net.Socket.connect(Socket.java:528)
at sun.net.NetworkClient.doConnect(NetworkClient.java:180)
at sun.net.www.http.HttpClient.openServer(HttpClient.java:432)
at sun.net.www.http.HttpClient.openServer(HttpClient.java:527)
at sun.net.www.http.HttpClient.<init>(HttpClient.java:211)
at sun.net.www.http.HttpClient.New(HttpClient.java:308)
at sun.net.www.http.HttpClient.New(HttpClient.java:326)
at sun.net.www.protocol.http.HttpURLConnection.getNewHttpClient(HttpURLConnection.java:996)
at sun.net.www.protocol.http.HttpURLConnection.plainConnect(HttpURLConnection.java:932)
at sun.net.www.protocol.http.HttpURLConnection.connect(HttpURLConnection.java:850)
at sun.net.www.protocol.http.HttpURLConnection.getOutputStream(HttpURLConnection.java:1091)
at com.sun.jersey.client.urlconnection.URLConnectionClientHandler$1$1.getOutputStream(URLConnectionClientHandler.java:225)
at com.sun.jersey.api.client.CommittingOutputStream.commitWrite(CommittingOutputStream.java:117)
at com.sun.jersey.api.client.CommittingOutputStream.write(CommittingOutputStream.java:89)
at org.codehaus.jackson.impl.Utf8Generator._flushBuffer(Utf8Generator.java:1754)
at org.codehaus.jackson.impl.Utf8Generator.flush(Utf8Generator.java:1088)
at org.codehaus.jackson.map.ObjectMapper.writeValue(ObjectMapper.java:1354)
at org.codehaus.jackson.jaxrs.JacksonJsonProvider.writeTo(JacksonJsonProvider.java:527)
at com.sun.jersey.api.client.RequestWriter.writeRequestEntity(RequestWriter.java:300)
at com.sun.jersey.client.urlconnection.URLConnectionClientHandler._invoke(URLConnectionClientHandler.java:204)
at com.sun.jersey.client.urlconnection.URLConnectionClientHandler.handle(URLConnectionClientHandler.java:147)
... 9 moreit>(HttpClient.java:211)
at sun.net.www.http.HttpClient.New(HttpClient.java:308)
at sun.net.www.http.HttpClient.New(HttpClient.java:326)
at sun.net.www.protocol.http.HttpURLConnection.getNewHttpClient(HttpURLConnection.java:996)
at sun.net.www.protocol.http.HttpURLConnection.plainConnect(HttpURLConnection.java:932)
at sun.net.www.protocol.http.HttpURLConnection.connect(HttpURLConnection.java:850)
at sun.net.www.protocol.http.HttpURLConnection.getOutputStream(HttpURLConnection.java:1091)
at com.sun.jersey.client.urlconnection.URLConnectionClientHandler$1$1.getOutputStream(URLConnectionClientHandler.java:225)
at com.sun.jersey.api.client.CommittingOutputStream.commitWrite(CommittingOutputStream.java:117)
at com.sun.jersey.api.client.CommittingOutputStream.write(CommittingOutputStream.java:89)
at org.codehaus.jackson.impl.Utf8Generator._flushBuffer(Utf8Generator.java:1754)
at org.codehaus.jackson.impl.Utf8Generator.flush(Utf8Generator.java:1088)
at org.codehaus.jackson.map.ObjectMapper.writeValue(ObjectMapper.java:1354)
at org.codehaus.jackson.jaxrs.JacksonJsonProvider.writeTo(JacksonJsonProvider.java:527)
at com.sun.jersey.api.client.RequestWriter.writeRequestEntity(RequestWriter.java:300)
at com.sun.jersey.client.urlconnection.URLConnectionClientHandler._invoke(URLConnectionClientHandler.java:204)
at com.sun.jersey.client.urlconnection.URLConnectionClientHandler.handle(URLConnectionClientHandler.java:147)
... 9 more
{noformat}

The current version of ProportionalCapacityPreemptionPolicy should be improved to deal with the following two scenarios:
1) when rebalancing over-capacity allocations, it potentially preempts without considering the maxCapacity constraints of a queue (i.e., preempting possibly more than strictly necessary)
2) a zero capacity queue is preempted even if there is no demand (coherent with old use of zero-capacity to disabled queues)
The proposed patch fixes both issues, and introduce few new test cases.

YARN-1947.
Major test reported by Jian He and fixed by Jian He TestRMDelegationTokens#testRMDTMasterKeyStateOnRollingMasterKey is failing intermittently

java.lang.AssertionError: null
at org.junit.Assert.fail(Assert.java:92)
at org.junit.Assert.assertTrue(Assert.java:43)
at org.junit.Assert.assertTrue(Assert.java:54)
at org.apache.hadoop.yarn.server.resourcemanager.security.TestRMDelegationTokens.testRMDTMasterKeyStateOnRollingMasterKey(TestRMDelegationTokens.java:117)

YARN-1933.
Major bug reported by Jian He and fixed by Jian He TestAMRestart and TestNodeHealthService failing sometimes on Windows

TestNodeHealthService failures:
testNodeHealthScript(org.apache.hadoop.yarn.server.nodemanager.TestNodeHealthService) Time elapsed: 1.405 sec <<< ERROR!
java.io.FileNotFoundException: C:\Users\Administrator\Documents\hadoop-common\hadoop-yarn-project\hadoop-yarn\hadoop-yarn-server\hadoop-yarn-server-nodemanager\target\org.apache.hadoop.yarn.server.nodemanager.TestNodeHealthService-localDir\failingscript.cmd (The process cannot access the file because it is being used by another process)
at java.io.FileOutputStream.open(Native Method)
at java.io.FileOutputStream.<init>(FileOutputStream.java:221)
at java.io.FileOutputStream.<init>(FileOutputStream.java:171)
at org.apache.hadoop.yarn.server.nodemanager.TestNodeHealthService.writeNodeHealthScriptFile(TestNodeHealthService.java:82)
at org.apache.hadoop.yarn.server.nodemanager.TestNodeHealthService.testNodeHealthScript(TestNodeHealthService.java:154)
testNodeHealthScriptShouldRun(org.apache.hadoop.yarn.server.nodemanager.TestNodeHealthService) Time elapsed: 0 sec <<< ERROR!
java.io.FileNotFoundException: C:\Users\Administrator\Documents\hadoop-common\hadoop-yarn-project\hadoop-yarn\hadoop-yarn-server\hadoop-yarn-server-nodemanager\target\org.apache.hadoop.yarn.server.nodemanager.TestNodeHealthService-localDir\failingscript.cmd (Access is denied)
at java.io.FileOutputStream.open(Native Method)
at java.io.FileOutputStream.<init>(FileOutputStream.java:221)
at java.io.FileOutputStream.<init>(FileOutputStream.java:171)
at org.apache.hadoop.yarn.server.nodemanager.TestNodeHealthService.writeNodeHealthScriptFile(TestNodeHealthService.java:82)
at org.apache.hadoop.yarn.server.nodemanager.TestNodeHealthService.testNodeHealthScriptShouldRun(TestNodeHealthService.java:103)

Scripts can be injected into the job status page as the diagnostics field is
not sanitized. Whatever string you set there will show up to the jobs page as it is ... ie. if you put any script commands, they will be executed in the browser of the user who is opening the page.
We need escaping the diagnostic string in order to not run the scripts.

YARN-1824 broke compatibility with previous 2.x releases by changes the API's in org.apache.hadoop.yarn.util.Apps.{setEnvFromInputString,addToEnvironment} The old api should be added back in.
This affects any ApplicationMasters who were using this api. It also breaks previously built MapReduce libraries from working with the new Yarn release as MR uses this api.

YARN-1929.
Blocker bug reported by Rohith and fixed by Karthik Kambatla (resourcemanager)DeadLock in RM when automatic failover is enabled.

Dead lock detected in RM when automatic failover is enabled.
{noformat}
Found one Java-level deadlock:
=============================
"Thread-2":
waiting to lock monitor 0x00007fb514303cf0 (object 0x00000000ef153fd0, a org.apache.hadoop.ha.ActiveStandbyElector),
which is held by "main-EventThread"
"main-EventThread":
waiting to lock monitor 0x00007fb514750a48 (object 0x00000000ef154020, a org.apache.hadoop.yarn.server.resourcemanager.EmbeddedElectorService),
which is held by "Thread-2"
{noformat}

{code}
junit.framework.AssertionFailedError: expected:<0> but was:<4>
at junit.framework.Assert.fail(Assert.java:50)
at junit.framework.Assert.failNotEquals(Assert.java:287)
at junit.framework.Assert.assertEquals(Assert.java:67)
at junit.framework.Assert.assertEquals(Assert.java:199)
at junit.framework.Assert.assertEquals(Assert.java:205)
at org.apache.hadoop.yarn.server.resourcemanager.applicationsmanager.TestAMRMRPCNodeUpdates.testAMRMUnusableNodes(TestAMRMRPCNodeUpdates.java:136)
{code}

The TestFSDownload.testDownloadPublicWithStatCache test in hadoop-yarn-common consistently fails on Windows environments.
The root cause is that the test checks for execute permission for all users on every ancestor of the target directory. In windows, by default, group "Everyone" has no permissions on any directory in the install drive. It's unreasonable to expect this test to pass and we should skip it on Windows.

YARN-1910.
Major bug reported by Xuan Gong and fixed by Xuan Gong TestAMRMTokens fails on windows

Create test1.sh having "pwd".
Run this command as user1:
hadoop jar /usr/lib/hadoop-yarn/hadoop-yarn-applications-distributedshell.jar -jar /usr/lib/hadoop-yarn/hadoop-yarn-applications-distributedshell.jar -shell_script test1.sh
NM is run by yarn user. An exception is thrown because yarn user has no permissions on custom script in hdfs path. The custom script is created with distributed shell app.
{code}
Caused by: org.apache.hadoop.ipc.RemoteException(org.apache.hadoop.security.AccessControlException): Permission denied: user=yarn, access=WRITE, inode="/user/user1/DistributedShell/70":user1:user1:drwxr-xr-x
at org.apache.hadoop.hdfs.server.namenode.FSPermissionChecker.checkFsPermission(FSPermissionChecker.java:265)
{code}

The test has 10000 containers that it tries to cleanup.
The cleanup has a timeout of 20000ms in which the test sometimes cannot do the cleanup completely and gives out an Assertion Failure.

YARN-1905.
Trivial test reported by Chris Nauroth and fixed by Chris Nauroth (nodemanager)TestProcfsBasedProcessTree must only run on Linux.

The tests in {{TestProcfsBasedProcessTree}} only make sense on Linux, where the process tree calculations are based on reading the /proc file system. Right now, not all of the individual tests are skipped when the OS is not Linux. This patch will make it consistent.

YARN-1903.
Major bug reported by Zhijie Shen and fixed by Zhijie Shen Killing Container on NEW and LOCALIZING will result in exitCode and diagnostics not set

The container status after stopping container is not expected.
{code}
java.lang.AssertionError: 4:
at org.junit.Assert.fail(Assert.java:93)
at org.junit.Assert.assertTrue(Assert.java:43)
at org.apache.hadoop.yarn.client.api.impl.TestNMClient.testGetContainerStatus(TestNMClient.java:382)
at org.apache.hadoop.yarn.client.api.impl.TestNMClient.testContainerManagement(TestNMClient.java:346)
at org.apache.hadoop.yarn.client.api.impl.TestNMClient.testNMClient(TestNMClient.java:226)
{code}

YARN-1883.
Major bug reported by Mit Desai and fixed by Mit Desai TestRMAdminService fails due to inconsistent entries in UserGroups

testRefreshUserToGroupsMappingsWithFileSystemBasedConfigurationProvider fails with the following error:
{noformat}
java.lang.AssertionError: null
at org.junit.Assert.fail(Assert.java:92)
at org.junit.Assert.assertTrue(Assert.java:43)
at org.junit.Assert.assertTrue(Assert.java:54)
at org.apache.hadoop.yarn.server.resourcemanager.TestRMAdminService.testRefreshUserToGroupsMappingsWithFileSystemBasedConfigurationProvider(TestRMAdminService.java:421)
at org.apache.hadoop.yarn.server.resourcemanager.TestRMAdminService.testOrder(TestRMAdminService.java:104)
{noformat}
Line Numbers will be inconsistent as I was testing to run it in a particular order. But the Line on which the failure occurs is
{code}
Assert.assertTrue(groupBefore.contains("test_group_A")
&& groupBefore.contains("test_group_B")
&& groupBefore.contains("test_group_C") && groupBefore.size() == 3);
{code}
testRMInitialsWithFileSystemBasedConfigurationProvider() and
testRefreshUserToGroupsMappingsWithFileSystemBasedConfigurationProvider()
calls the function {{MockUnixGroupsMapping.updateGroups();}} which changes the list of userGroups.
testRefreshUserToGroupsMappingsWithFileSystemBasedConfigurationProvider() tries to verify the groups before changing it and fails if testRMInitialsWithFileSystemBasedConfigurationProvider() already ran and made the changes.

TestMoveApplication#testMoveRejectedByScheduler fails because of NullPointerException. It looks caused by unhandled exception handling at server-side.

YARN-1750.
Major test reported by Ming Ma and fixed by Wangda Tan (nodemanager)TestNodeStatusUpdater#testNMRegistration is incorrect in test case

This test case passes. However, the test output log has
java.lang.AssertionError: Number of applications should only be one! expected:<1> but was:<2>
at org.junit.Assert.fail(Assert.java:93)
at org.junit.Assert.failNotEquals(Assert.java:647)
at org.junit.Assert.assertEquals(Assert.java:128)
at org.junit.Assert.assertEquals(Assert.java:472)
at org.apache.hadoop.yarn.server.nodemanager.TestNodeStatusUpdater$MyResourceTracker.nodeHeartbeat(TestNodeStatusUpdater.java:267)
at org.apache.hadoop.yarn.server.nodemanager.NodeStatusUpdaterImpl$1.run(NodeStatusUpdaterImpl.java:469)
at java.lang.Thread.run(Thread.java:695)
TestNodeStatusUpdater.java has invalid asserts.
} else if (heartBeatID == 3) {
// Checks on the RM end
Assert.assertEquals("Number of applications should only be one!", 1,
appToContainers.size());
Assert.assertEquals("Number of container for the app should be two!",
2, appToContainers.get(appId2).size());
We should fix the assert and add more check to the test.

YARN-1701.
Major sub-task reported by Gera Shegalov and fixed by Tsuyoshi OZAWA Improve default paths of timeline store and generic history store

When I enable AHS via yarn.ahs.enabled, the app history is still not visible in AHS webUI. This is due to NullApplicationHistoryStore as yarn.resourcemanager.history-writer.class. It would be good to have just one key to enable basic functionality.
yarn.ahs.fs-history-store.uri uses {code}${hadoop.log.dir}{code}, which is local file system location. However, FileSystemApplicationHistoryStore uses DFS by default.

YARN-1873.
Major bug reported by Mit Desai and fixed by Mit Desai TestDistributedShell#testDSShell fails when the test cases are out of order

testDSShell fails when the tests are run in random order. I see a cleanup issue here.
{noformat}
Tests run: 1, Failures: 1, Errors: 0, Skipped: 0, Time elapsed: 72.222 sec <<< FAILURE! - in org.apache.hadoop.yarn.applications.distributedshell.TestDistributedShell
testOrder(org.apache.hadoop.yarn.applications.distributedshell.TestDistributedShell) Time elapsed: 44.127 sec <<< FAILURE!
java.lang.AssertionError: expected:<1> but was:<6>
at org.junit.Assert.fail(Assert.java:93)
at org.junit.Assert.failNotEquals(Assert.java:647)
at org.junit.Assert.assertEquals(Assert.java:128)
at org.junit.Assert.assertEquals(Assert.java:472)
at org.junit.Assert.assertEquals(Assert.java:456)
at org.apache.hadoop.yarn.applications.distributedshell.TestDistributedShell.testDSShell(TestDistributedShell.java:204)
at org.apache.hadoop.yarn.applications.distributedshell.TestDistributedShell.testOrder(TestDistributedShell.java:134)
Results :
Failed tests:
TestDistributedShell.testOrder:134->testDSShell:204 expected:<1> but was:<6>
{noformat}
The Line numbers will be little deviated because I was trying to reproduce the error by running the tests in specific order. But the Line that causes the assert fail is {{Assert.assertEquals(1, entitiesAttempts.getEntities().size());}}

We ran into the following NPE when fetching applications using the REST API:
{noformat}
INTERNAL_SERVER_ERROR
java.lang.NullPointerException
at org.apache.hadoop.yarn.server.security.ApplicationACLsManager.checkAccess(ApplicationACLsManager.java:104)
at org.apache.hadoop.yarn.server.resourcemanager.webapp.RMWebServices.hasAccess(RMWebServices.java:123)
at org.apache.hadoop.yarn.server.resourcemanager.webapp.RMWebServices.getApps(RMWebServices.java:418)
{noformat}

This happened in Hadoop-Yarn-trunk - Build # 516 and can be reproduced:
{code}
testWebAppProxyInStandAloneMode(org.apache.hadoop.yarn.client.TestRMFailover) Time elapsed: 5.834 sec <<< FAILURE!
java.lang.AssertionError: null
at org.junit.Assert.fail(Assert.java:92)
at org.junit.Assert.assertTrue(Assert.java:43)
at org.junit.Assert.assertTrue(Assert.java:54)
at org.apache.hadoop.yarn.client.TestRMFailover.verifyExpectedException(TestRMFailover.java:250)
at org.apache.hadoop.yarn.client.TestRMFailover.testWebAppProxyInStandAloneMode(TestRMFailover.java:216)
testEmbeddedWebAppProxy(org.apache.hadoop.yarn.client.TestRMFailover) Time elapsed: 5.341 sec <<< FAILURE!
java.lang.AssertionError: null
at org.junit.Assert.fail(Assert.java:92)
at org.junit.Assert.assertTrue(Assert.java:43)
at org.junit.Assert.assertTrue(Assert.java:54)
at org.apache.hadoop.yarn.client.TestRMFailover.verifyExpectedException(TestRMFailover.java:250)
at org.apache.hadoop.yarn.client.TestRMFailover.testEmbeddedWebAppProxy(TestRMFailover.java:241)
{code}

YARN-1859.
Major bug reported by Zhijie Shen and fixed by Zhijie Shen WebAppProxyServlet will throw ApplicationNotFoundException if the app is no longer cached in RM

WebAppProxyServlet checks null to determine whether the application is not found or not.
{code}
ApplicationReport applicationReport = getApplicationReport(id);
if(applicationReport == null) {
LOG.warn(req.getRemoteUser()+" Attempting to access "+id+
" that was not found");
{code}
However, WebAppProxyServlet calls AppReportFetcher, which consequently calls ClientRMService. When application is not found, ClientRMService throws ApplicationNotFoundException. Therefore, in WebAppProxyServlet, the following logic to create the tracking url for a non-cached app will no longer be in use.

There is race in test.
TestRMHA#testStartAndTransitions calls verifyClusterMetrics() immediately after application is submitted, but QueueMetrics are updated after app attempt is sheduled. Calling verifyClusterMetrics() without verifying app attempt is in Scheduled state cause random test failures.
MockRM.submitApp() return when application is in ACCEPTED, but QueueMetrics updated at APP_ATTEMPT_ADDED event. There is high chance of getting queue metrics before app attempt is Scheduled.
{noformat}
testStartAndTransitions(org.apache.hadoop.yarn.server.resourcemanager.TestRMHA) Time elapsed: 5.883 sec <<< FAILURE!
java.lang.AssertionError: Incorrect value for metric availableMB expected:<2048> but was:<4096>
at org.junit.Assert.fail(Assert.java:93)
at org.junit.Assert.failNotEquals(Assert.java:647)
at org.junit.Assert.assertEquals(Assert.java:128)
at org.junit.Assert.assertEquals(Assert.java:472)
at org.apache.hadoop.yarn.server.resourcemanager.TestRMHA.assertMetric(TestRMHA.java:396)
at org.apache.hadoop.yarn.server.resourcemanager.TestRMHA.verifyClusterMetrics(TestRMHA.java:387)
at org.apache.hadoop.yarn.server.resourcemanager.TestRMHA.testStartAndTransitions(TestRMHA.java:160)
Results :
Failed tests:
TestRMHA.testStartAndTransitions:160->verifyClusterMetrics:387->assertMetric:396 Incorrect value for metric availableMB expected:<2048> but was:<4096>
{noformat}

YARN-1852.
Major bug reported by Rohith and fixed by Rohith (resourcemanager)Application recovery throws InvalidStateTransitonException for FAILED and KILLED jobs

Recovering for failed/killed application throw InvalidStateTransitonException.
These are logged during recovery of applications.

YARN-1850.
Major sub-task reported by Zhijie Shen and fixed by Zhijie Shen Make enabling timeline service configurable

Like generic history service, we'd better to make enabling timeline service configurable, in case the timeline server is not up

While running an UnmanagedAM on secure cluster, ran into an NPE on failover/restart. This is similar to YARN-1821.

YARN-1846.
Major bug reported by Robert Kanter and fixed by Robert Kanter TestRM#testNMTokenSentForNormalContainer assumes CapacityScheduler

TestRM.testNMTokenSentForNormalContainer assumes the CapacityScheduler is being used and tries to do:
{code:java}
CapacityScheduler cs = (CapacityScheduler) rm.getResourceScheduler();
{code}
This throws a {{ClassCastException}} if you're not using the CapacityScheduler.

Use single-node cluster. Turn on capacity scheduler preemption. Run MR sleep job as app 1. Take entire cluster. Run MR sleep job as app 2. Preempt app1 out. Wait till app 2 finishes. App 1 AM attempt 2 will start. It won't be able to launch a task container with this error stack trace in AM logs:
{code}
2014-03-13 20:13:50,254 INFO [AsyncDispatcher event handler] org.apache.hadoop.mapreduce.v2.app.job.impl.TaskAttemptImpl: Diagnostics report from attempt_1394741557066_0001_m_000000_1009: Container launch failed for container_1394741557066_0001_02_000021 : org.apache.hadoop.security.token.SecretManager$InvalidToken: No NMToken sent for <host>:45454
at org.apache.hadoop.yarn.client.api.impl.ContainerManagementProtocolProxy$ContainerManagementProtocolProxyData.newProxy(ContainerManagementProtocolProxy.java:206)
at org.apache.hadoop.yarn.client.api.impl.ContainerManagementProtocolProxy$ContainerManagementProtocolProxyData.<init>(ContainerManagementProtocolProxy.java:196)
at org.apache.hadoop.yarn.client.api.impl.ContainerManagementProtocolProxy.getProxy(ContainerManagementProtocolProxy.java:117)
at org.apache.hadoop.mapreduce.v2.app.launcher.ContainerLauncherImpl.getCMProxy(ContainerLauncherImpl.java:403)
at org.apache.hadoop.mapreduce.v2.app.launcher.ContainerLauncherImpl$Container.launch(ContainerLauncherImpl.java:138)
at org.apache.hadoop.mapreduce.v2.app.launcher.ContainerLauncherImpl$EventProcessor.run(ContainerLauncherImpl.java:369)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:722)
{code}

YARN-1838.
Major sub-task reported by Srimanth Gunturi and fixed by Billie Rinaldi Timeline service getEntities API should provide ability to get entities from given id

To support pagination, we need ability to get entities from a certain ID by providing a new param called {{fromid}}.
For example on a page of 10 jobs, our first call will be like
[http://server:8188/ws/v1/timeline/HIVE_QUERY_ID?fields=events,primaryfilters,otherinfo&limit=11]
When user hits next, we would like to call
[http://server:8188/ws/v1/timeline/HIVE_QUERY_ID?fields=events,primaryfilters,otherinfo&fromid=JID11&limit=11]
and continue on for further _Next_ clicks
On hitting back, we will make similar calls for previous items
[http://server:8188/ws/v1/timeline/HIVE_QUERY_ID?fields=events,primaryfilters,otherinfo&fromid=JID1&limit=11]
{{fromid}} should be inclusive of the id given.

YARN-1833.
Major bug reported by Mit Desai and fixed by Mit Desai TestRMAdminService Fails in trunk and branch-2 : Assert Fails due to different count of UserGroups for currentUser()

In the test testRefreshUserToGroupsMappingsWithFileSystemBasedConfigurationProvider, the following assert is not needed.
{code}
Assert.assertTrue(groupWithInit.size() != groupBefore.size());
{code}
As the assert takes the default groups for groupWithInit (which in my case are users, sshusers and wheel), it fails as the size of both groupWithInit and groupBefore are same.
I do not think we need to have this assert here. Moreover we are also checking that the groupInit does not have the userGroups that are in the groupBefore so removing the assert may not be harmful.

YARN-1824.
Major bug reported by Jian He and fixed by Jian He Make Windows client work with Linux/Unix cluster

YARN-1821.
Blocker sub-task reported by Karthik Kambatla and fixed by Karthik Kambatla (resourcemanager)NPE on registerNodeManager if the request has containers for UnmanagedAMs

On RM restart (or failover), NM re-registers with the RM. If it was running containers for Unmanaged AMs, it runs into the following NPE:
{noformat}
Caused by: org.apache.hadoop.ipc.RemoteException(java.lang.NullPointerException): java.lang.NullPointerException
at org.apache.hadoop.yarn.server.resourcemanager.ResourceTrackerService.registerNodeManager(ResourceTrackerService.java:213)
at org.apache.hadoop.yarn.server.api.impl.pb.service.ResourceTrackerPBServiceImpl.registerNodeManager(ResourceTrackerPBServiceImpl.java:54)
{noformat}

YARN-1816.
Major sub-task reported by Arpit Gupta and fixed by Jian He Succeeded application remains in accepted after RM restart

YARN-1812.
Major sub-task reported by Yesha Vora and fixed by Jian He Job stays in PREP state for long time after RM Restarts

Steps followed:
1) start a sort job with 80 maps and 5 reducers
2) restart Resource manager when 60 maps and 0 reducers are finished
3) Wait for job to come out of PREP state.
The job does not come out of PREP state after 7-8 mins.
After waiting for 7-8 mins, test kills the job.
However, Sort job should not take this long time to come out of PREP state

YARN-1811.
Major sub-task reported by Robert Kanter and fixed by Robert Kanter (resourcemanager)RM HA: AM link broken if the AM is on nodes other than RM

When using RM HA, if you click on the "Application Master" link in the RM web UI while the job is running, you get an Error 500:

Trying to kill an Unmanaged AM though CLI (yarn application -kill <id>) logs a success, but doesn't actually kill the AM or reclaim the containers allocated to it.

YARN-1789.
Minor improvement reported by Akira AJISAKA and fixed by Tsuyoshi OZAWA (resourcemanager)ApplicationSummary does not escape newlines in the app name

YARN-side of MAPREDUCE-5778.
ApplicationSummary is not escaping newlines in the app name. This can result in an application summary log entry that spans multiple lines when users are expecting one-app-per-line output.

yarn applicationattempt prints:
{code}
Invalid Command Usage :
usage: application
-appStates <States> Works with -list to filter applications
based on input comma-separated list of
application states. The valid application
state can be one of the following:
ALL,NEW,NEW_SAVING,SUBMITTED,ACCEPTED,RUN
NING,FINISHED,FAILED,KILLED
-appTypes <Types> Works with -list to filter applications
based on input comma-separated list of
application types.
-help Displays help for all commands.
-kill <Application ID> Kills the application.
-list <arg> List application attempts for aplication
from AHS.
-movetoqueue <Application ID> Moves the application to a different
queue.
-queue <Queue Name> Works with the movetoqueue command to
specify which queue to move an
application to.
-status <Application ID> Prints the status of the application.
{code}
yarn container prints:
{code}
Invalid Command Usage :
usage: application
-appStates <States> Works with -list to filter applications
based on input comma-separated list of
application states. The valid application
state can be one of the following:
ALL,NEW,NEW_SAVING,SUBMITTED,ACCEPTED,RUN
NING,FINISHED,FAILED,KILLED
-appTypes <Types> Works with -list to filter applications
based on input comma-separated list of
application types.
-help Displays help for all commands.
-kill <Application ID> Kills the application.
-list <arg> List application attempts for aplication
from AHS.
-movetoqueue <Application ID> Moves the application to a different
queue.
-queue <Queue Name> Works with the movetoqueue command to
specify which queue to move an
application to.
-status <Application ID> Prints the status of the application.
{code}
Both commands print irrelevant yarn application usage information.

When invoking the /ws/v1/cluster/apps endpoint, RM will eventually get to RMAppImpl#createAndGetApplicationReport, which calls RMAppAttemptImpl#getApplicationResourceUsageReport, which looks up the app in the scheduler, which may or may not exist. So FairScheduler shouldn't log an error for every lookup failure:
{noformat}
2014-02-17 08:23:21,240 ERROR org.apache.hadoop.yarn.server.resourcemanager.scheduler.fair.FairScheduler: Request for appInfo of unknown attemptappattempt_1392419715319_0135_000001
{noformat}

YARN-1783.
Critical bug reported by Arpit Gupta and fixed by Jian He yarn application does not make any progress even when no other application is running when RM is being restarted in the background

Noticed that during HA tests some tests took over 3 hours to run when the test failed.
Looking at the logs i see the application made no progress for a very long time. However if i look at application log from yarn it actually ran in 5 mins
I am seeing same behavior when RM was being restarted in the background and when both RM and AM were being restarted. This does not happen for all applications but a few will hit this in the nightly run.

YARN-1781.
Major sub-task reported by Varun Vasudev and fixed by Varun Vasudev (nodemanager)NM should allow users to specify max disk utilization for local disks

This is related to YARN-257(it's probably a sub task?). Currently, the NM does not detect full disks and allows full disks to be used by containers leading to repeated failures. YARN-257 deals with graceful handling of full disks. This ticket is only about detection of full disks by the disk health checkers.
The NM should allow users to set a maximum disk utilization for local disks and mark disks as bad once they exceed that utilization. At the very least, the NM should at least detect full disks.

YARN-1780.
Major sub-task reported by Zhijie Shen and fixed by Zhijie Shen Improve logging in timeline service

It's difficult to trace whether the client has successfully posted the entity to the timeline service or not.

Instead of catching ApplicationNotFound and logging a simple app not found message, the whole stack trace is logged.

YARN-1766.
Major sub-task reported by Xuan Gong and fixed by Xuan Gong When RM does the initiation, it should use loaded Configuration instead of bootstrap configuration.

Right now, we have FileSystemBasedConfigurationProvider to let Users upload the configurations into remote File System, and let different RMs share the same configurations. During the initiation, RM will load the configurations from Remote File System. So when RM initiates the services, it should use the loaded Configurations instead of using the bootstrap configurations.

YARN-1765.
Major sub-task reported by Xuan Gong and fixed by Xuan Gong Write test cases to verify that killApplication API works in RM HA

YARN-1764.
Major sub-task reported by Xuan Gong and fixed by Xuan Gong Handle RM fail overs after the submitApplication call.

YARN-1761.
Major sub-task reported by Xuan Gong and fixed by Xuan Gong RMAdminCLI should check whether HA is enabled before executes transitionToActive/transitionToStandby

YARN-1611 adds TestRMAdminService which assumes the use of CapacityScheduler.
{noformat}
java.lang.ClassCastException: org.apache.hadoop.yarn.server.resourcemanager.scheduler.fair.FairScheduler cannot be cast to org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.CapacityScheduler
at org.apache.hadoop.yarn.server.resourcemanager.TestRMAdminService.testAdminRefreshQueuesWithFileSystemBasedConfigurationProvider(TestRMAdminService.java:115)
{noformat}

YARN-1752.
Major bug reported by Jian He and fixed by Rohith Unexpected Unregistered event at Attempt Launched state

{code}
2014-02-21 14:56:03,453 ERROR org.apache.hadoop.yarn.server.resourcemanager.rmapp.attempt.RMAppAttemptImpl: Can't handle this event at current state
org.apache.hadoop.yarn.state.InvalidStateTransitonException: Invalid event: UNREGISTERED at LAUNCHED
at org.apache.hadoop.yarn.state.StateMachineFactory.doTransition(StateMachineFactory.java:305)
at org.apache.hadoop.yarn.state.StateMachineFactory.access$300(StateMachineFactory.java:46)
at org.apache.hadoop.yarn.state.StateMachineFactory$InternalStateMachine.doTransition(StateMachineFactory.java:448)
at org.apache.hadoop.yarn.server.resourcemanager.rmapp.attempt.RMAppAttemptImpl.handle(RMAppAttemptImpl.java:647)
at org.apache.hadoop.yarn.server.resourcemanager.rmapp.attempt.RMAppAttemptImpl.handle(RMAppAttemptImpl.java:103)
at org.apache.hadoop.yarn.server.resourcemanager.ResourceManager$ApplicationAttemptEventDispatcher.handle(ResourceManager.java:733)
at org.apache.hadoop.yarn.server.resourcemanager.ResourceManager$ApplicationAttemptEventDispatcher.handle(ResourceManager.java:714)
at org.apache.hadoop.yarn.event.AsyncDispatcher.dispatch(AsyncDispatcher.java:173)
at org.apache.hadoop.yarn.event.AsyncDispatcher$1.run(AsyncDispatcher.java:106)
at java.lang.Thread.run(Thread.java:695)
{code}

YARN-1749.
Major sub-task reported by Zhijie Shen and fixed by Zhijie Shen Review AHS configs and sync them up with the timeline-service configs

In YarnConfiguration.java,
{code}
/**
* By default, at least 5% of disks are to be healthy to say that the node
* is healthy in terms of disks.
*/
public static final float DEFAULT_NM_MIN_HEALTHY_DISKS_FRACTION
= 0.25F;
{code}
25% is the correct.

YARN-1734.
Critical sub-task reported by Xuan Gong and fixed by Xuan Gong RM should get the updated Configurations when it transits from Standby to Active

Currently, we have ConfigurationProvider which can support LocalConfiguration, and FileSystemBasedConfiguration. When HA is enabled, and FileSystemBasedConfiguration is enabled, RM can not get the updated Configurations when it transits from Standby to Active

YARN-1732.
Major sub-task reported by Billie Rinaldi and fixed by Billie Rinaldi Change types of related entities and primary filters in ATSEntity

The current types Map<String, List<String>> relatedEntities and Map<String, Object> primaryFilters have issues. The List<String> value of the related entities map could have multiple identical strings in it, which doesn't make sense. A more major issue is that we cannot allow primary filter values to be overwritten, because otherwise we will be unable to find those primary filter entries when we want to delete an entity (without doing a nearly full scan).
I propose changing related entities to Map<String, Set<String>> and primary filters to Map<String, Set<Object>>. The basic methods to add primary filters and related entities are of the form add(key, value) and will not need to change.

Primary filters and secondary filter values can be arbitrary json-compatible Object. The web services should determine if the filters specified as query parameters are objects or strings before passing them to the store.

YARN-1724.
Critical bug reported by Sandy Ryza and fixed by Sandy Ryza (scheduler)Race condition in Fair Scheduler when continuous scheduling is turned on

These don't appear to affect how the web services work, but the following warnings are logged:
{noformat}
WARNING: The following warnings have been detected with resource and/or provider
classes:
WARNING: A sub-resource method, public org.apache.hadoop.yarn.server.applicati
onhistoryservice.webapp.ATSWebServices$AboutInfo org.apache.hadoop.yarn.server.a
pplicationhistoryservice.webapp.ATSWebServices.about(javax.servlet.http.HttpServ
letRequest,javax.servlet.http.HttpServletResponse), with URI template, "/", is t
reated as a resource method
WARNING: A sub-resource method, public org.apache.hadoop.yarn.api.records.appt
imeline.ATSPutErrors org.apache.hadoop.yarn.server.applicationhistoryservice.web
app.ATSWebServices.postEntities(javax.servlet.http.HttpServletRequest,javax.serv
let.http.HttpServletResponse,org.apache.hadoop.yarn.api.records.apptimeline.ATSE
ntities), with URI template, "/", is treated as a resource method
{noformat}

We saw a ConcurrentModificationException thrown in the fair scheduler:
{noformat}
2014-02-07 01:40:01,978 ERROR org.apache.hadoop.yarn.server.resourcemanager.scheduler.fair.FairScheduler: Exception in fair scheduler UpdateThread
java.util.ConcurrentModificationException
at java.util.HashMap$HashIterator.nextEntry(HashMap.java:926)
at java.util.HashMap$ValueIterator.next(HashMap.java:954)
at org.apache.hadoop.yarn.server.resourcemanager.scheduler.fair.AppSchedulable.updateDemand(AppSchedulable.java:85)
at org.apache.hadoop.yarn.server.resourcemanager.scheduler.fair.FSLeafQueue.updateDemand(FSLeafQueue.java:125)
at org.apache.hadoop.yarn.server.resourcemanager.scheduler.fair.FSParentQueue.updateDemand(FSParentQueue.java:82)
at org.apache.hadoop.yarn.server.resourcemanager.scheduler.fair.FairScheduler.update(FairScheduler.java:217)
at org.apache.hadoop.yarn.server.resourcemanager.scheduler.fair.FairScheduler$UpdateThread.run(FairScheduler.java:195)
at java.lang.Thread.run(Thread.java:724)
{noformat}
The map that gets returned by FSSchedulerApp.getResourceRequests() are iterated on without proper synchronization.

YARN-1689.
Critical bug reported by Deepesh Khandelwal and fixed by Vinod Kumar Vavilapalli (resourcemanager)RMAppAttempt is not killed when RMApp is at ACCEPTED

When running some Hive on Tez jobs, the RM after a while gets into an unusable state where no jobs run. In the RM log I see the following exception:
{code}
2014-02-04 20:28:08,553 WARN ipc.Server (Server.java:run(1978)) - IPC Server handler 0 on 8030, call org.apache.hadoop.yarn.api.ApplicationMasterProtocolPB.registerApplicationMaster from 172.18.145.156:40474 Call#0 Retry#0: error: java.lang.NullPointerException
java.lang.NullPointerException
at org.apache.hadoop.yarn.server.resourcemanager.scheduler.AbstractYarnScheduler.getTransferredContainers(AbstractYarnScheduler.java:48)
at org.apache.hadoop.yarn.server.resourcemanager.ApplicationMasterService.registerApplicationMaster(ApplicationMasterService.java:278)
at org.apache.hadoop.yarn.api.impl.pb.service.ApplicationMasterProtocolPBServiceImpl.registerApplicationMaster(ApplicationMasterProtocolPBServiceImpl.java:90)
at org.apache.hadoop.yarn.proto.ApplicationMasterProtocol$ApplicationMasterProtocolService$2.callBlockingMethod(ApplicationMasterProtocol.java:95)
at org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:585)
at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:928)
at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:1962)
at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:1958)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:396)
at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1548)
at org.apache.hadoop.ipc.Server$Handler.run(Server.java:1956)
......
2014-02-04 20:28:08,544 ERROR rmapp.RMAppImpl (RMAppImpl.java:handle(626)) - Can't handle this event at current state
org.apache.hadoop.yarn.state.InvalidStateTransitonException: Invalid event: ATTEMPT_REGISTERED at KILLED
at org.apache.hadoop.yarn.state.StateMachineFactory.doTransition(StateMachineFactory.java:305)
at org.apache.hadoop.yarn.state.StateMachineFactory.access$300(StateMachineFactory.java:46)
at org.apache.hadoop.yarn.state.StateMachineFactory$InternalStateMachine.doTransition(StateMachineFactory.java:448)
at org.apache.hadoop.yarn.server.resourcemanager.rmapp.RMAppImpl.handle(RMAppImpl.java:624)
at org.apache.hadoop.yarn.server.resourcemanager.rmapp.RMAppImpl.handle(RMAppImpl.java:81)
at org.apache.hadoop.yarn.server.resourcemanager.ResourceManager$ApplicationEventDispatcher.handle(ResourceManager.java:656)
at org.apache.hadoop.yarn.server.resourcemanager.ResourceManager$ApplicationEventDispatcher.handle(ResourceManager.java:640)
at org.apache.hadoop.yarn.event.AsyncDispatcher.dispatch(AsyncDispatcher.java:173)
at org.apache.hadoop.yarn.event.AsyncDispatcher$1.run(AsyncDispatcher.java:106)
at java.lang.Thread.run(Thread.java:662)
2014-02-04 20:28:08,549 INFO resourcemanager.RMAuditLogger (RMAuditLogger.java:logSuccess(140)) - USER=hrt_qa IP=172.18.145.156 OPERATION=Kill Application Request TARGET=ClientRMService RESULT=SUCCESS APPID=application_1391543307203_0001
2014-02-04 20:28:08,553 WARN ipc.Server (Server.java:run(1978)) - IPC Server handler 0 on 8030, call org.apache.hadoop.yarn.api.ApplicationMasterProtocolPB.registerApplicationMaster from 172.18.145.156:40474 Call#0 Retry#0: error: java.lang.NullPointerException
java.lang.NullPointerException
at org.apache.hadoop.yarn.server.resourcemanager.scheduler.AbstractYarnScheduler.getTransferredContainers(AbstractYarnScheduler.java:48)
at org.apache.hadoop.yarn.server.resourcemanager.ApplicationMasterService.registerApplicationMaster(ApplicationMasterService.java:278)
at org.apache.hadoop.yarn.api.impl.pb.service.ApplicationMasterProtocolPBServiceImpl.registerApplicationMaster(ApplicationMasterProtocolPBServiceImpl.java:90)
at org.apache.hadoop.yarn.proto.ApplicationMasterProtocol$ApplicationMasterProtocolService$2.callBlockingMethod(ApplicationMasterProtocol.java:95)
at org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:585)
at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:928)
at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:1962)
at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:1958)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:396)
at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1548)
at org.apache.hadoop.ipc.Server$Handler.run(Server.java:1956)
{code}

YARN-1686.
Major bug reported by Rohith and fixed by Rohith (nodemanager)NodeManager.resyncWithRM() does not handle exception which cause NodeManger to Hang.

During start of NodeManager,if registration with resourcemanager throw exception then nodemager shutdown happens.
Consider case where NM-1 is registered with RM. RM issued Resync to NM. If any exception thrown in "resyncWithRM" (starts new thread which does not handle exception) during RESYNC evet, then this thread is lost. NodeManger enters hanged state.

YARN-1685.
Major sub-task reported by Mayank Bansal and fixed by Zhijie Shen Bugs around log URL

1. Log URL should be different when the container is running and finished
2. Null case needs to be handled
3. The way of constructing log URL should be corrected

YARN-1684.
Major sub-task reported by Billie Rinaldi and fixed by Billie Rinaldi Fix history server heap size in yarn script

The yarn script currently has the following:
{noformat}
if [ "$YARN_RESOURCEMANAGER_HEAPSIZE" != "" ]; then
JAVA_HEAP_MAX="-Xmx""$YARN_HISTORYSERVER_HEAPSIZE""m"
fi
{noformat}

YARN-1676.
Major sub-task reported by Xuan Gong and fixed by Xuan Gong Make admin refreshUserToGroupsMappings of configuration work across RM failover

yarn application -kill <application ID>
used to work previously. In 2.4.0 it prints out help message and does not kill the application.

YARN-1672.
Trivial bug reported by Karthik Kambatla and fixed by Naren Koneru (nodemanager)YarnConfiguration is missing a default for yarn.nodemanager.log.retain-seconds

YarnConfiguration is missing a default for yarn.nodemanager.log.retain-seconds

YARN-1670.
Critical bug reported by Thomas Graves and fixed by Mit Desai aggregated log writer can write more log data then it says is the log length

We have seen exceptions when using 'yarn logs' to read log files.
at java.lang.NumberFormatException.forInputString(NumberFormatException.java:65)
at java.lang.Long.parseLong(Long.java:441)
at java.lang.Long.parseLong(Long.java:483)
at org.apache.hadoop.yarn.logaggregation.AggregatedLogFormat$LogReader.readAContainerLogsForALogType(AggregatedLogFormat.java:518)
at org.apache.hadoop.yarn.logaggregation.LogDumper.dumpAContainerLogs(LogDumper.java:178)
at org.apache.hadoop.yarn.logaggregation.LogDumper.run(LogDumper.java:130)
at org.apache.hadoop.yarn.logaggregation.LogDumper.main(LogDumper.java:246)
We traced it down to the reader trying to read the file type of the next file but where it reads is still log data from the previous file. What happened was the Log Length was written as a certain size but the log data was actually longer then that.
Inside of the write() routine in LogValue it first writes what the logfile length is, but then when it goes to write the log itself it just goes to the end of the file. There is a race condition here where if someone is still writing to the file when it goes to be aggregated the length written could be to small.
We should have the write() routine stop when it writes whatever it said was the length. It would be nice if we could somehow tell the user it might be truncated but I'm not sure of a good way to do this.
We also noticed that a bug in readAContainerLogsForALogType where it is using an int for curRead whereas it should be using a long.
while (len != -1 && curRead < fileLength) {
This isn't actually a problem right now as it looks like the underlying decoder is doing the right thing and the len condition exits.

YARN-1669.
Major sub-task reported by Xuan Gong and fixed by Xuan Gong Make admin refreshServiceAcls work across RM failover

YARN-1668.
Major sub-task reported by Xuan Gong and fixed by Xuan Gong Make admin refreshAdminAcls work across RM failover

Change the handling of admin-acls to be available across RM failover by making using of a remote configuration-provider

YARN-1667.
Major sub-task reported by Xuan Gong and fixed by Xuan Gong Make admin refreshSuperUserGroupsConfiguration work across RM failover

YARN-1666.
Major sub-task reported by Xuan Gong and fixed by Xuan Gong Make admin refreshNodes work across RM failover

In order to enable HA (automatic failover) i had to set the following configs
{code}
<property>
<name>yarn.resourcemanager.ha.enabled</name>
<value>true</value>
</property>
<property>
<name>yarn.resourcemanager.ha.automatic-failover.enabled</name>
<value>true</value>
</property>
<property>
<name>yarn.resourcemanager.ha.automatic-failover.embedded</name>
<value>true</value>
</property>
{code}
I believe the user should just have to set yarn.resourcemanager.ha.enabled=true and the rest should be set as defaults. Basically automatic failover should be the default.

YARN-1661.
Major bug reported by Tassapol Athiapinya and fixed by Vinod Kumar Vavilapalli (applications/distributed-shell)AppMaster logs says failing even if an application does succeed.

YARN-1660.
Major sub-task reported by Arpit Gupta and fixed by Xuan Gong (resourcemanager)add the ability to set yarn.resourcemanager.hostname.rm-id instead of setting all the various host:port properties for RM

Currently the user has to specify all the various host:port properties for RM. We should follow the pattern that we do for non HA setup where we can specify yarn.resourcemanager.hostname.rm-id and the defaults are used for all other affected properties.

YARN-1659.
Major sub-task reported by Billie Rinaldi and fixed by Billie Rinaldi Define the ApplicationTimelineStore store as an abstraction for implementing different storage impls for storing timeline information

These will be used by ApplicationTimelineStore interface. The web services will convert the store-facing obects to the user-facing objects.

YARN-1658.
Major sub-task reported by Cindy Li and fixed by Cindy Li Webservice should redirect to active RM when HA is enabled.

When HA is enabled, web service to standby RM should be redirected to the active RM. This is a related Jira to YARN-1525.

YARN-1641.
Major sub-task reported by Karthik Kambatla and fixed by Karthik Kambatla (resourcemanager)ZK store should attempt a write periodically to ensure it is still Active

Fencing in ZK store kicks in when the RM tries to write something to the store. If the RM doesn't write anything to the store, it doesn't get fenced and can continue to assume being the Active.
By periodically writing a file (say, every RM_ZK_TIMEOUT_MS seconds), we can ensure it gets fenced.

YARN-1640.
Blocker sub-task reported by Xuan Gong and fixed by Xuan Gong Manual Failover does not work in secure clusters

NodeManager gets rejected after manually making one RM as active.

YARN-1639.
Major sub-task reported by Arpit Gupta and fixed by Xuan Gong (resourcemanager)YARM RM HA requires different configs on different RM hosts

We need to set yarn.resourcemanager.ha.id to rm1 or rm2 based on which rm you want to first or second.
This means we have different configs on different RM nodes. This is unlike HDFS HA where the same configs are pushed to both NN's and it would be better to have the same setup for RM as this would make installation and managing easier.

YARN-1637.
Major sub-task reported by Vinod Kumar Vavilapalli and fixed by Zhijie Shen Implement a client library for java users to post entities+events

This is a wrapper around the web-service to facilitate easy posting of entity+event data to the time-line server.

YARN-1632.
Minor bug reported by Chen He and fixed by Chen He TestApplicationMasterServices should be under org.apache.hadoop.yarn.server.resourcemanager package

ApplicationMasterService is under org.apache.hadoop.yarn.server.resourcemanager package. However, its unit test file TestApplicationMasterService is placed under org.apache.hadoop.yarn.server.resourcemanager.applicationmasterservice package which only contains one file (TestApplicationMasterService).

When I ran dev-support/test-patch.sh, following message output.
{code}
mvn apache-rat:check -DHadoopPatchProcess > /tmp/patchReleaseAuditOutput.txt 2>&1
There appear to be 1 release audit warnings after applying the patch.
{code}
{code}
!????? /home/sinchii/git/YARN-321-test/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/main/resources/webapps/applicationhistory/.keep
Lines that start with ????? in the release audit report indicate files that do not have an Apache license header.
{code}
To avoid release audit warning, it should fix pom.xml.

YARN-1617.
Major bug reported by Sandy Ryza and fixed by Sandy Ryza (scheduler)Remove ancient comment and surround LOG.debug in AppSchedulingInfo.allocate

As evidenced by Jenkins at https://issues.apache.org/jira/browse/YARN-1041?focusedCommentId=13868621&page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-13868621.
It's failing randomly on trunk on my local box too

_HOST is not properly substituted when we use VIP address. Currently it always used the host name of the machine and disregard the VIP address. It is true mainly for RM, NM, WebProxy, and JHS rpc service. Looks like it is working fine for webservice authentication.
On the other hand, the same thing is working fine for NN and SNN in RPC as well as webservice.

YARN-1588.
Major sub-task reported by Jian He and fixed by Jian He Rebind NM tokens for previous attempt's running containers to the new attempt

YARN-1578.
Major sub-task reported by Shinichi Yamashita and fixed by Shinichi Yamashita Fix how to read history file in FileSystemApplicationHistoryStore

I carried out PiEstimator job at Hadoop cluster which applied YARN-321.
After the job end and when I accessed Web UI of HistoryServer, it displayed "500". And HistoryServer daemon log was output as follows.
{code}
2014-01-09 13:31:12,227 ERROR org.apache.hadoop.yarn.webapp.Dispatcher: error handling URI: /applicationhistory/appattempt/appattempt_1389146249925_0008_000001
java.lang.reflect.InvocationTargetException
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:597)
at org.apache.hadoop.yarn.webapp.Dispatcher.service(Dispatcher.java:153)
at javax.servlet.http.HttpServlet.service(HttpServlet.java:820)
(snip...)
Caused by: java.lang.NullPointerException
at org.apache.hadoop.yarn.server.applicationhistoryservice.FileSystemApplicationHistoryStore.mergeContainerHistoryData(FileSystemApplicationHistoryStore.java:696)
at org.apache.hadoop.yarn.server.applicationhistoryservice.FileSystemApplicationHistoryStore.getContainers(FileSystemApplicationHistoryStore.java:429)
at org.apache.hadoop.yarn.server.applicationhistoryservice.ApplicationHistoryManagerImpl.getContainers(ApplicationHistoryManagerImpl.java:201)
at org.apache.hadoop.yarn.server.webapp.AppAttemptBlock.render(AppAttemptBlock.java:110)
(snip...)
{code}
I confirmed that there was container which was not finished from ApplicationHistory file.
In ResourceManager daemon log, ResourceManager reserved this container, but did not allocate it.
When FileSystemApplicationHistoryStore reads container information without finish data in history file, this problem occurs.
In consideration of the case which there is not finish data, we should fix how to read history file in FileSystemApplicationHistoryStore.

YARN-1577.
Blocker sub-task reported by Jian He and fixed by Jian He Unmanaged AM is broken because of YARN-1493

Today unmanaged AM client is waiting for app state to be Accepted to launch the AM. This is broken since we changed in YARN-1493 to start the attempt after the application is Accepted. We may need to introduce an attempt state report that client can rely on to query the attempt state and choose to launch the unmanaged AM.

YARN-1570.
Minor improvement reported by Akira AJISAKA and fixed by Akira AJISAKA (documentation)Formatting the lines within 80 chars in YarnCommands.apt.vm

In YarnCommands.apt.vm, there are some lines longer than 80 characters.
For example:
{code}
Yarn commands are invoked by the bin/yarn script. Running the yarn script without any arguments prints the description for all commands.
{code}

YARN-1566.
Major sub-task reported by Jian He and fixed by Jian He Change distributed-shell to retain containers from previous AppAttempt

Change distributed-shell to reuse previous AM's running containers when AM is restarting. It can also be made configurable whether to enable this feature or not.

YARN-1553.
Major bug reported by Haohui Mai and fixed by Haohui Mai Do not use HttpConfig.isSecure() in YARN

HDFS-5305 and related jira decide that each individual project will have their own configuration on http policy. {{HttpConfig.isSecure}} is a global static method which does not fit the design anymore. The same functionality should be moved into the YARN code base.

YARN-1536.
Minor improvement reported by Karthik Kambatla and fixed by Anubhav Dhoot (resourcemanager)Cleanup: Get rid of ResourceManager#get*SecretManager() methods and use the RMContext methods instead

Both ResourceManager and RMContext have methods to access the secret managers, and it should be safe (cleaner) to get rid of the ResourceManager methods.

There are some options which are not written to Yarn Command document.
For example, "yarn rmadmin" command options are as follows:
{code}
Usage: yarn rmadmin
-refreshQueues
-refreshNodes
-refreshSuperUserGroupsConfiguration
-refreshUserToGroupsMappings
-refreshAdminAcls
-refreshServiceAcl
-getGroups [username]
-help [cmd]
-transitionToActive <serviceId>
-transitionToStandby <serviceId>
-failover [--forcefence] [--forceactive] <serviceId> <serviceId>
-getServiceState <serviceId>
-checkHealth <serviceId>
{code}
But some of the new options such as "-getGroups", "-transitionToActive", and "-transitionToStandby" are not documented.

YARN-1493.
Major sub-task reported by Jian He and fixed by Jian He Schedulers don't recognize apps separately from app-attempts

Today, scheduler is tied to attempt only.
We need to separate app-level handling logic in scheduler. We can add new app-level events to the scheduler and separate the app-level logic out. This is good for work-preserving AM restart, RM restart, and also needed for differentiating app-level metrics and attempt-level metrics.

YARN-1490.
Major sub-task reported by Vinod Kumar Vavilapalli and fixed by Jian He RM should optionally not kill all containers when an ApplicationMaster exits

This is needed to enable work-preserving AM restart. Some apps can chose to reconnect with old running containers, some may not want to. This should be an option.

YARN-1461.
Major sub-task reported by Karthik Kambatla and fixed by Karthik Kambatla (resourcemanager)RM API and RM changes to handle tags for running jobs

YARN-1459.
Major sub-task reported by Karthik Kambatla and fixed by Xuan Gong (resourcemanager)RM services should depend on ConfigurationProvider during startup too

YARN-1667, YARN-1668, YARN-1669 already changed RM to depend on a configuration provider so as to be able to refresh many configuration files across RM fail-over. The dependency on the configuration-provider by the RM should happen at its boot up time too.

YARN-1452.
Major task reported by Zhijie Shen and fixed by Zhijie Shen Document the usage of the generic application history and the timeline data service

We need to write a bunch of documents to guide users. such as command line tools, configurations and REST APIs

I have tried to force reducers to execute on certain nodes. What I did is I changed for reduce tasks, the RMContainerRequestor#addResourceRequest(req.priority, ResourceRequest.ANY, req.capability) to RMContainerRequestor#addResourceRequest(req.priority, HOST_NAME, req.capability).
However, this change lead to RM crashes when reducers needs to be assigned with the following exception:
FATAL org.apache.hadoop.yarn.server.resourcemanager.ResourceManager: Error in handling event type NODE_UPDATE to the scheduler
java.lang.NullPointerException
at org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.LeafQueue.assignContainers(LeafQueue.java:841)
at org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.ParentQueue.assignContainersToChildQueues(ParentQueue.java:640)
at org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.ParentQueue.assignContainers(ParentQueue.java:554)
at org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.CapacityScheduler.nodeUpdate(CapacityScheduler.java:695)
at org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.CapacityScheduler.handle(CapacityScheduler.java:739)
at org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.CapacityScheduler.handle(CapacityScheduler.java:86)
at org.apache.hadoop.yarn.server.resourcemanager.ResourceManager$SchedulerEventDispatcher$EventProcessor.run(ResourceManager.java:549)
at java.lang.Thread.run(Thread.java:722)

YARN-1428.
Major sub-task reported by Zhijie Shen and fixed by Zhijie Shen RM cannot write the final state of RMApp/RMAppAttempt to the application history store in the transition to the final state

ApplicationFinishData and ApplicationAttemptFinishData are written in the final transitions of RMApp/RMAppAttempt respectively. However, in the transitions, getState() is not getting the state that RMApp/RMAppAttempt is going to enter, but prior one.

YARN-1417.
Blocker bug reported by Omkar Vinit Joshi and fixed by Jian He RM may issue expired container tokens to AM while issuing new containers.

Today we create new container token when we create container in RM as a part of schedule cycle. However that container may get reserved or assigned. If the container gets reserved and remains like that (in reserved state) for more than container token expiry interval then RM will end up issuing container with expired token.

YARN-1410.
Major sub-task reported by Bikas Saha and fixed by Xuan Gong Handle RM fails over after getApplicationID() and before submitApplication().

App submission involves
1) creating appId
2) using that appId to submit an ApplicationSubmissionContext to the user.
The client may have obtained an appId from an RM, the RM may have failed over, and the client may submit the app to the new RM.
Since the new RM has a different notion of cluster timestamp (used to create app id) the new RM may reject the app submission resulting in unexpected failure on the client side.
The same may happen for other 2 step client API operations.

getQueueInfo in parentQueue will call child.getQueueInfo().
This will try acquire the leaf queue lock over parent queue lock.
Now at same time if a completedContainer call comes and acquired LeafQueue lock and it will wait for ParentQueue's completedConatiner call.
This lock usage is not in synchronous and can lead to deadlock.
With JCarder, this is showing as a potential deadlock scenario.

YARN-1389.
Major sub-task reported by Mayank Bansal and fixed by Mayank Bansal ApplicationClientProtocol and ApplicationHistoryProtocol should expose analogous APIs

As we plan to have the APIs in ApplicationHistoryProtocol to expose the reports of *finished* application attempts and containers, we should do the same for ApplicationClientProtocol, which will return the reports of *running* attempts and containers.
Later on, we can improve YarnClient to direct the query of running instance to ApplicationClientProtocol, while that of finished instance to ApplicationHistoryProtocol, making it transparent to the users.

YARN-1379.
Major sub-task reported by Vinod Kumar Vavilapalli and fixed by Vinod Kumar Vavilapalli [YARN-321] AHS protocols need to be in yarn proto package name after YARN-1170

Found this while merging YARN-321 to the latest branch-2. Without this, compilation fails.

YARN-1345.
Major sub-task reported by Zhijie Shen and fixed by Zhijie Shen Removing FINAL_SAVING from YarnApplicationAttemptState

Whenever YARN-891 is done, we need to add the mapping of RMAppAttemptState.FINAL_SAVING -> YarnApplicationAttemptState.FINAL_SAVING in RMServerUtils#createApplicationAttemptState

YARN-1301.
Minor bug reported by Zhijie Shen and fixed by Tsuyoshi OZAWA Need to log the blacklist additions/removals when YarnSchedule#allocate

Now without the log, it's hard to debug whether blacklist is updated on the scheduler side or not

The Fair Scheduler doc is missing the following properties.
- defaultMinSharePreemptionTimeout
- queueMaxAppsDefault

YARN-1166.
Blocker bug reported by Srimanth Gunturi and fixed by Zhijie Shen (resourcemanager)YARN 'appsFailed' metric should be of type 'counter'

Currently in YARN's queue metrics, the cumulative metric 'appsFailed' is of type 'guage' - which means the exact value will be reported.
All other cumulative queue metrics (AppsSubmitted, AppsCompleted, AppsKilled) are all of type 'counter' - meaning Ganglia will use slope to provide deltas between time-points.
To be consistent, AppsFailed metric should also be of type 'counter'.

YARN-1041.
Major sub-task reported by Steve Loughran and fixed by Jian He (resourcemanager)Protocol changes for RM to bind and notify a restarted AM of existing containers

For long lived containers we don't want the AM to be a SPOF.
When the RM restarts a (failed) AM, it should be given the list of containers it had already been allocated. the AM should then be able to contact the NMs to get details on them. NMs would also need to do any binding of the containers needed to handle a moved/restarted AM.

YARN-1023.
Major sub-task reported by Devaraj K and fixed by Zhijie Shen [YARN-321] Webservices REST API's support for Application History

YARN-1017.
Blocker sub-task reported by Jian He and fixed by Jian He (resourcemanager)Document RM Restart feature

This should give users a general idea about how RM Restart works and how to use RM Restart

YARN-1007.
Major sub-task reported by Devaraj K and fixed by Mayank Bansal [YARN-321] Enhance History Reader interface for Containers

If we want to show the containers used by application/app attempt, We need to have two more API's which returns collection of ContainerHistoryData for application id and applcation attempt id something like below.
{code:xml}
Collection<ContainerHistoryData> getContainers(
ApplicationAttemptId appAttemptId);
Collection<ContainerHistoryData> getContainers(ApplicationId appId);
{code}
{code:xml}
/**
* This method returns {@link Container} for specified {@link ContainerId}.
*
* @param {@link ContainerId}
* @return {@link Container} for ContainerId
*/
ContainerHistoryData getAMContainer(ContainerId containerId);
{code}
In the above API, we need to change the argument to application attempt id or we can remove this API because every attempt history data has master container id field, using master container id, history data can get using this below API if it takes argument as container id.
{code:xml}
/**
* This method returns {@link ContainerHistoryData} for specified
* {@link ApplicationAttemptId}.
*
* @param {@link ApplicationAttemptId}
* @return {@link ContainerHistoryData} for ApplicationAttemptId
*/
ContainerHistoryData getContainer(ApplicationAttemptId appAttemptId);
{code}
Here application attempt can use numbers of containers but we cannot choose which container history data to return. This API argument also need to be changed to take container id instead of app attempt id.

YARN-987.
Major sub-task reported by Mayank Bansal and fixed by Mayank Bansal Adding ApplicationHistoryManager responsible for exposing reports to all clients

YARN-986.
Blocker sub-task reported by Vinod Kumar Vavilapalli and fixed by Karthik Kambatla RM DT token service should have service addresses of both RMs

Previously: YARN should use cluster-id as token service address
This needs to be done to support non-ip based fail over of RM. Once the server sets the token service address to be this generic ClusterId/ServiceId, clients can translate it to appropriate final IP and then be able to select tokens via TokenSelectors.
Some workarounds for other related issues were put in place at YARN-945.

YARN-984.
Major sub-task reported by Devaraj K and fixed by Devaraj K [YARN-321] Move classes from applicationhistoryservice.records.pb.impl package to applicationhistoryservice.records.impl.pb

While creating instance for applicationhistoryservice.records.* pb records, It is throwing the ClassNotFoundException.
{code:xml}
Caused by: java.lang.ClassNotFoundException: Class org.apache.hadoop.yarn.server.applicationhistoryservice.records.impl.pb.ApplicationHistoryDataPBImpl not found
at org.apache.hadoop.conf.Configuration.getClassByName(Configuration.java:1619)
at org.apache.hadoop.yarn.factories.impl.pb.RecordFactoryPBImpl.newRecordInstance(RecordFactoryPBImpl.java:56)
... 49 more
{code}

YARN-979.
Major sub-task reported by Mayank Bansal and fixed by Mayank Bansal [YARN-321] Add more APIs related to ApplicationAttempt and Container in ApplicationHistoryProtocol

ApplicationHistoryProtocol should have the following APIs as well:
* getApplicationAttemptReport
* getApplicationAttempts
* getContainerReport
* getContainers
The corresponding request and response classes need to be added as well.

YARN-975.
Major sub-task reported by Zhijie Shen and fixed by Zhijie Shen Add a file-system implementation for history-storage

HDFS implementation should be a standard persistence strategy of history storage

YARN-974.
Major sub-task reported by Zhijie Shen and fixed by Zhijie Shen RMContainer should collect more useful information to be recorded in Application-History

To record the history of a container, users may be also interested in the following information:
1. Start Time
2. Stop Time
3. Diagnostic Information
4. URL to the Log File
5. Actually Allocated Resource
6. Actually Assigned Node
These should be remembered during the RMContainer's life cycle.

YARN-967.
Major sub-task reported by Devaraj K and fixed by Mayank Bansal [YARN-321] Command Line Interface(CLI) for Reading Application History Storage Data

YARN-954.
Major sub-task reported by Vinod Kumar Vavilapalli and fixed by Zhijie Shen [YARN-321] History Service should create the webUI and wire it to HistoryStorage

YARN-953.
Major sub-task reported by Vinod Kumar Vavilapalli and fixed by Zhijie Shen [YARN-321] Enable ResourceManager to write history data

YARN-947.
Major sub-task reported by Zhijie Shen and fixed by Zhijie Shen Defining the history data classes for the implementation of the reading/writing interface

We need to define the history data classes have the exact fields to be stored. Therefore, all the implementations don't need to have the duplicate logic to exact the required information from RMApp, RMAppAttempt and RMContainer.
We use protobuf to define these classes, such that they can be ser/des to/from bytes, which are easier for persistence.

YARN-935.
Major sub-task reported by Zhijie Shen and fixed by Zhijie Shen YARN-321 branch is broken due to applicationhistoryserver module's pom.xml

The branch was created from branch-2, hadoop-yarn-server-applicationhistoryserver/pom.xml should use 2.2.0-SNAPSHOT, not 3.0.0-SNAPSHOT. Otherwise, the sub-project cannot be built correctly because of wrong dependency.

YARN-934.
Major sub-task reported by Zhijie Shen and fixed by Zhijie Shen HistoryStorage writer interface for Application History Server

YARN-713.
Critical bug reported by Jason Lowe and fixed by Jian He (resourcemanager)ResourceManager can exit unexpectedly if DNS is unavailable

As discussed in MAPREDUCE-5261, there's a possibility that a DNS outage could lead to an unhandled exception in the ResourceManager's AsyncDispatcher, and that ultimately would cause the RM to exit. The RM should not exit during DNS hiccups.

HDFS-6061.
Major sub-task reported by Colin Patrick McCabe and fixed by Colin Patrick McCabe (datanode)Allow dfs.datanode.shared.file.descriptor.path to contain multiple entries and fall back when needed

HDFS-6060.
Major sub-task reported by Brandon Li and fixed by Brandon Li (namenode)NameNode should not check DataNode layout version

HDFS-6059.
Major bug reported by Akira AJISAKA and fixed by Akira AJISAKA TestBlockReaderLocal fails if native library is not available

HDFS-6057.
Blocker bug reported by Eric Sirianni and fixed by Colin Patrick McCabe (hdfs-client)DomainSocketWatcher.watcherThread should be marked as daemon thread

HDFS-6055.
Major improvement reported by Suresh Srinivas and fixed by Chris Nauroth (namenode)Change default configuration to limit file name length in HDFS

The default configuration of HDFS now sets dfs.namenode.fs-limits.max-component-length to 255 for improved interoperability with other file system implementations. This limits each component of a file system path to a maximum of 255 bytes in UTF-8 encoding. Attempts to create new files that violate this rule will fail with an error. Existing files that violate the rule are not effected. Previously, dfs.namenode.fs-limits.max-component-length was set to 0 (ignored). If necessary, it is possible to set the value back to 0 in the cluster's configuration to restore the old behavior.

HDFS-6053.
Major bug reported by Jing Zhao and fixed by Jing Zhao (namenode)Fix TestDecommissioningStatus and TestDecommission in branch-2

HDFS-5949.
Minor bug reported by Travis Thompson and fixed by Travis Thompson (namenode)New Namenode UI when trying to download a file, the browser doesn't know the file name

HDFS-5948.
Major bug reported by Andrew Wang and fixed by Haohui Mai TestBackupNode flakes with port in use error

HDFS-5944.
Major bug reported by zhaoyunjiong and fixed by zhaoyunjiong (namenode)LeaseManager:findLeaseWithPrefixPath can't handle path like /a/b/ right and cause SecondaryNameNode failed do checkpoint

HDFS-5943.
Major bug reported by Yesha Vora and fixed by Suresh Srinivas 'dfs.namenode.https-address.ns1' property is not used in federation setup

HDFS-5828.
Major bug reported by Taylor, Buddy and fixed by Taylor, Buddy (namenode)BlockPlacementPolicyWithNodeGroup can place multiple replicas on the same node group when dfs.namenode.avoid.write.stale.datanode is true

HDFS-5821.
Major bug reported by Gera Shegalov and fixed by Gera Shegalov (test)TestHDFSCLI fails for user names with the dash character

If a read from a block is slow, start up another parallel, 'hedged' read against a different block replica. We then take the result of which ever read returns first (the outstanding read is cancelled). This 'hedged' read feature will help rein in the outliers, the odd read that takes a long time because it hit a bad patch on the disc, etc.
This feature is off by default. To enable this feature, set <code>dfs.client.hedged.read.threadpool.size</code> to a positive number. The threadpool size is how many threads to dedicate to the running of these 'hedged', concurrent reads in your client.
Then set <code>dfs.client.hedged.read.threshold.millis</code> to the number of milliseconds to wait before starting up a 'hedged' read. For example, if you set this property to 10, then if a read has not returned within 10 milliseconds, we will start up a new read against a different block replica.
This feature emits new metrics:
+ hedgedReadOps
+ hedgeReadOpsWin -- how many times the hedged read 'beat' the original read
+ hedgedReadOpsInCurThread -- how many times we went to do a hedged read but we had to run it in the current thread because dfs.client.hedged.read.threadpool.size was at a maximum.

HDFS-5775.
Major improvement reported by Haohui Mai and fixed by Haohui Mai (namenode)Consolidate the code for serialization in CacheManager

HDFS-5768.
Major improvement reported by Haohui Mai and fixed by Haohui Mai (namenode)Consolidate the serialization code in DelegationTokenSecretManager

HDFS-5767.
Blocker bug reported by Yongjun Zhang and fixed by Yongjun Zhang (nfs)NFS implementation assumes userName userId mapping to be unique, which is not true sometimes

HDFS-5759.
Major bug reported by Haohui Mai and fixed by Haohui Mai Web UI does not show up during the period of loading FSImage

HDFS-5483.
Major sub-task reported by Arpit Agarwal and fixed by Arpit Agarwal (namenode)NN should gracefully handle multiple block replicas on same DN

HDFS-5339.
Major bug reported by Stephen Chu and fixed by Haohui Mai (webhdfs)WebHDFS URI does not accept logical nameservices when security is enabled

HDFS-5321.
Major sub-task reported by Haohui Mai and fixed by Haohui Mai Clean up the HTTP-related configuration in HDFS

dfs.http.port and dfs.https.port are removed. Filesystem clients, such as WebHdfsFileSystem, now have fixed instead of configurable default ports (i.e., 50070 for http and 50470 for https).
Users can explicitly specify the port in the URI to access the file system which runs on non-default ports.

HDFS-5318.
Major improvement reported by Eric Sirianni and fixed by (namenode)Support read-only and read-write paths to shared replicas

HADOOP-10295.
Major improvement reported by Jing Zhao and fixed by Jing Zhao (tools/distcp)Allow distcp to automatically identify the checksum type of source files and use it for the target

Add option for distcp to preserve the checksum type of the source files. Users can use "-pc" as distcp command option to preserve the checksum type.

HADOOP-10285.
Major sub-task reported by Chris Li and fixed by Admin interface to swap callqueue at runtime

HADOOP-10280.
Major sub-task reported by Chris Li and fixed by Chris Li Make Schedulables return a configurable identity of user or group

HADOOP-10278.
Major sub-task reported by Chris Li and fixed by Chris Li (ipc)Refactor to make CallQueue pluggable

HADOOP-10249.
Major bug reported by Dilli Arumugam and fixed by Dilli Arumugam LdapGroupsMapping should trim ldap password read from file

HADOOP-10221.
Major improvement reported by Benoy Antony and fixed by Benoy Antony (security)Add a plugin to specify SaslProperties for RPC protocol based on connection properties

HADOOP-10211.
Major improvement reported by Benoy Antony and fixed by Benoy Antony (security)Enable RPC protocol to negotiate SASL-QOP values between clients and servers

The hadoop.rpc.protection configuration property previously supported specifying a single value: one of authentication, integrity or privacy. An unrecognized value was silently assumed to mean authentication. This configuration property now accepts a comma-separated list of any of the 3 values, and unrecognized values are rejected with an error. Existing configurations containing an invalid value must be corrected. If the property is empty or not specified, authentication is assumed.

YARN-1630.
Major bug reported by Aditya Acharya and fixed by Aditya Acharya (client)Introduce timeout for async polling operations in YarnClientImpl

I ran an MR2 application that would have been long running, and killed it programmatically using a YarnClient. The app was killed, but the client hung forever. The message that I saw, which spammed the logs, was "Watiting for application application_1389036507624_0018 to be killed."
The RM log indicated that the app had indeed transitioned from RUNNING to KILLED, but for some reason future responses to the RPC to kill the application did not indicate that the app had been terminated.
I tracked this down to YarnClientImpl.java, and though I was unable to reproduce the bug, I wrote a patch to introduce a bound on the number of times that YarnClientImpl retries the RPC before giving up.

The Test fails with the following error
{noformat}
java.lang.IllegalArgumentException: java.net.UnknownHostException: InvalidHost
at org.apache.hadoop.security.SecurityUtil.buildTokenService(SecurityUtil.java:377)
at org.apache.hadoop.yarn.server.security.BaseNMTokenSecretManager.newInstance(BaseNMTokenSecretManager.java:145)
at org.apache.hadoop.yarn.server.security.BaseNMTokenSecretManager.createNMToken(BaseNMTokenSecretManager.java:136)
at org.apache.hadoop.yarn.server.TestContainerManagerSecurity.testNMTokens(TestContainerManagerSecurity.java:253)
at org.apache.hadoop.yarn.server.TestContainerManagerSecurity.testContainerManager(TestContainerManagerSecurity.java:144)
{noformat}

YARN-1624.
Major bug reported by Aditya Acharya and fixed by Aditya Acharya (scheduler)QueuePlacementPolicy format is not easily readable via a JAXB parser

The current format for specifying queue placement rules in the fair scheduler allocations file does not lend itself to easy parsing via a JAXB parser. In particular, relying on the tag name to encode information about which rule to use makes it very difficult for an xsd-based JAXB parser to preserve the order of the rules, which is essential.

YARN-1623.
Major improvement reported by Sandy Ryza and fixed by Sandy Ryza (scheduler)Include queue name in RegisterApplicationMasterResponse

This provides the YARN change necessary to support MAPREDUCE-5732.

YARN-1618.
Blocker sub-task reported by Karthik Kambatla and fixed by Karthik Kambatla (resourcemanager)Fix invalid RMApp transition from NEW to FINAL_SAVING

YARN-891 augments the RMStateStore to store information on completed applications. In the process, it adds transitions from NEW to FINAL_SAVING. This leads to the RM trying to update entries in the state-store that do not exist. On ZKRMStateStore, this leads to the RM crashing.
Previous description:
ZKRMStateStore fails to handle updates to znodes that don't exist. For instance, this can happen when an app transitions from NEW to FINAL_SAVING. In these cases, the store should create the missing znode and handle the update.

YARN-1616.
Trivial improvement reported by Karthik Kambatla and fixed by Karthik Kambatla (resourcemanager)RMFatalEventDispatcher should log the cause of the event

RMFatalEventDispatcher#handle() logs the receipt of an event and its type, but leaves out the cause. The cause captures why the event was raised and would help debugging issues.

YARN-1608.
Trivial bug reported by Karthik Kambatla and fixed by Karthik Kambatla (nodemanager)LinuxContainerExecutor has a few DEBUG messages at INFO level

LCE has a few INFO level log messages meant to be at debug level. In fact, they are logged both at INFO and DEBUG.

We should either explicitly set the Capacity Scheduler or make it scheduler-agnostic

YARN-1603.
Trivial bug reported by Zhijie Shen and fixed by Zhijie Shen Remove two *.orig files which were unexpectedly committed

FairScheduler.java.orig and TestFifoScheduler.java.orig

YARN-1601.
Major bug reported by Alejandro Abdelnur and fixed by Alejandro Abdelnur 3rd party JARs are missing from hadoop-dist output

With the build changes of YARN-888 we are leaving out all 3rd party JArs used directly by YARN under /share/hadoop/yarn/lib/.
We did not notice this when running minicluster because they all happen to be in the classpath from hadoop-common and hadoop-yarn.
As 3d party JARs are not 'public' interfaces we cannot rely on them being provided to yarn by common and hdfs. (ie if common and hdfs stop using a 3rd party dependency that yarn uses this would break yarn if yarn does not pull that dependency explicitly).
Also, this will break bigtop hadoop build when they move to use branch-2 as they expect to find jars in /share/hadoop/yarn/lib/

YARN-1600.
Blocker bug reported by Jason Lowe and fixed by Haohui Mai (resourcemanager)RM does not startup when security is enabled without spnego configured

We have a custom auth filter in front of our various UI pages that handles user authentication. However currently the RM assumes that if security is enabled then the user must have configured spnego as well for the RM web pages which is not true in our case.

YARN-1574.
Blocker sub-task reported by Xuan Gong and fixed by Xuan Gong RMDispatcher should be reset on transition to standby

Currently, we move rmDispatcher out of ActiveService. But we still register the Event dispatcher, such as schedulerDispatcher, RMAppEventDispatcher when we initiate the ActiveService.
Almost every time when we transit RM from Active to Standby, we need to initiate the ActiveService. That means we will register the same event Dispatcher which will cause the same event will be handled several times.

YARN-1573.
Major sub-task reported by Karthik Kambatla and fixed by Karthik Kambatla (resourcemanager)ZK store should use a private password for root-node-acls

Currently, when HA is enabled, ZK store uses cluster-timestamp as the password for root node ACLs to give the Active RM exclusive access to the store. A more private value like a random number might be better.

The following can be reproduced locally:
{code}
testAMMRTokens(org.apache.hadoop.yarn.client.api.impl.TestYarnClient) Time elapsed: 3.341 sec <<< FAILURE!
junit.framework.AssertionFailedError: null
at junit.framework.Assert.fail(Assert.java:48)
at junit.framework.Assert.assertTrue(Assert.java:20)
at junit.framework.Assert.assertNotNull(Assert.java:218)
at junit.framework.Assert.assertNotNull(Assert.java:211)
at org.apache.hadoop.yarn.client.api.impl.TestYarnClient.testAMMRTokens(TestYarnClient.java:382)
{code}
This test didn't appear in https://builds.apache.org/job/Hadoop-Yarn-trunk/442/consoleFull

RMProxy#INSTANCE is a non-final static field and both ServerRMProxy and ClientRMProxy set it. This leads to races as witnessed on - YARN-1482.
Sample trace:
{noformat}
java.lang.IllegalArgumentException: RM does not support this client protocol
at com.google.common.base.Preconditions.checkArgument(Preconditions.java:88)
at org.apache.hadoop.yarn.client.ClientRMProxy.checkAllowedProtocols(ClientRMProxy.java:119)
at org.apache.hadoop.yarn.client.ConfiguredRMFailoverProxyProvider.init(ConfiguredRMFailoverProxyProvider.java:58)
at org.apache.hadoop.yarn.client.RMProxy.createRMFailoverProxyProvider(RMProxy.java:158)
at org.apache.hadoop.yarn.client.RMProxy.createRMProxy(RMProxy.java:88)
at org.apache.hadoop.yarn.server.api.ServerRMProxy.createRMProxy(ServerRMProxy.java:56)
{noformat}

YARN-1549.
Major test reported by Ted Yu and fixed by haosdent TestUnmanagedAMLauncher#testDSShell fails in trunk

The following error is reproducible:
{code}
testDSShell(org.apache.hadoop.yarn.applications.unmanagedamlauncher.TestUnmanagedAMLauncher) Time elapsed: 14.911 sec <<< ERROR!
java.lang.RuntimeException: Failed to receive final expected state in ApplicationReport, CurrentState=RUNNING, ExpectedStates=FINISHED,FAILED,KILLED
at org.apache.hadoop.yarn.applications.unmanagedamlauncher.UnmanagedAMLauncher.monitorApplication(UnmanagedAMLauncher.java:447)
at org.apache.hadoop.yarn.applications.unmanagedamlauncher.UnmanagedAMLauncher.run(UnmanagedAMLauncher.java:352)
at org.apache.hadoop.yarn.applications.unmanagedamlauncher.TestUnmanagedAMLauncher.testDSShell(TestUnmanagedAMLauncher.java:147)
{code}
See https://builds.apache.org/job/Hadoop-Yarn-trunk/435

YARN-1541.
Major bug reported by Jian He and fixed by Jian He Invalidate AM Host/Port when app attempt is done so that in the mean-while client doesn’t get wrong information.

YARN-1505.
Blocker bug reported by Xuan Gong and fixed by Xuan Gong WebAppProxyServer should not set localhost as YarnConfiguration.PROXY_ADDRESS by itself

At WebAppProxyServer::startServer(), it will set up YarnConfiguration.PROXY_ADDRESS to localhost:9099 by itself. So, no matter what is the value we set YarnConfiguration.PROXY_ADDRESS in configuration, the proxyserver will bind to localhost:9099

There are still four references to test classes that extend from junit.framework.TestCase
hadoop-yarn/hadoop-yarn-common/src/test/java/org/apache/hadoop/yarn/util/TestYarnVersionInfo.java
hadoop-yarn/hadoop-yarn-common/src/test/java/org/apache/hadoop/yarn/util/TestWindowsResourceCalculatorPlugin.java
hadoop-yarn/hadoop-yarn-common/src/test/java/org/apache/hadoop/yarn/util/TestLinuxResourceCalculatorPlugin.java
hadoop-yarn/hadoop-yarn-common/src/test/java/org/apache/hadoop/yarn/util/TestWindowsBasedProcessTree.java

YARN-1485.
Major sub-task reported by Xuan Gong and fixed by Xuan Gong Enabling HA should verify the RM service addresses configurations have been set for every RM Ids defined in RM_HA_IDs

After YARN-1325, the YarnConfiguration.RM_HA_IDS will contain multiple RM_Ids. We need to verify that the RM service addresses configurations have been set for all of RM_Ids.

YARN-1482.
Major sub-task reported by Vinod Kumar Vavilapalli and fixed by Xuan Gong WebApplicationProxy should be always-on w.r.t HA even if it is embedded in the RM

This way, even if an RM goes to standby mode, we can affect a redirect to the active. And more importantly, users will not suddenly see all their links stop working.

This is something I found while reviewing YARN-1318, but didn't halt that patch as many cycles went there already. Some top level issues
- Not easy to follow RM's service life cycle
-- RM adds only AdminService as its service directly.
-- Other services are added to RM when AdminService's init calls RM.activeServices.init()
- Overall, AdminService shouldn't encompass all of RM's HA state management. It was originally supposed to be the implementation of just the RPC server.

Here is stack trace:
{code}
testContainerManager[1](org.apache.hadoop.yarn.server.TestContainerManagerSecurity) Time elapsed: 1.756 sec <<< ERROR!
org.apache.hadoop.yarn.exceptions.YarnRuntimeException: java.io.IOException: ResourceManager failed to start. Final state is STOPPED
at org.apache.hadoop.yarn.server.MiniYARNCluster$ResourceManagerWrapper.serviceStart(MiniYARNCluster.java:253)
at org.apache.hadoop.service.AbstractService.start(AbstractService.java:193)
at org.apache.hadoop.service.CompositeService.serviceStart(CompositeService.java:121)
at org.apache.hadoop.service.AbstractService.start(AbstractService.java:193)
at org.apache.hadoop.yarn.server.TestContainerManagerSecurity.testContainerManager(TestContainerManagerSecurity.java:110)
{code}

YARN-1454.
Critical bug reported by Jian He and fixed by Karthik Kambatla TestRMRestart.testRMDelegationTokenRestoredOnRMRestart is failing intermittently

YARN-1451.
Minor bug reported by Sandy Ryza and fixed by Sandy Ryza TestResourceManager relies on the scheduler assigning multiple containers in a single node update

TestResourceManager rely on the capacity scheduler.
It relies on a scheduler that assigns multiple containers in a single heartbeat, which not all schedulers do by default. It also relies on schedulers that don't consider CPU capacities. It would be simple to change the test to use multiple heartbeats and increase the vcore capacities of the nodes in the test.

YARN-1450.
Major bug reported by Akira AJISAKA and fixed by Binglin Chang (applications/distributed-shell)TestUnmanagedAMLauncher#testDSShell fails on trunk

As described in YARN-1197, we need add some common PB types for container resource change, like ResourceChangeContext, etc. These types will be both used by RM/NM protocols

YARN-1446.
Major sub-task reported by Jian He and fixed by Jian He (resourcemanager)Change killing application to wait until state store is done

When user kills an application, it should wait until the state store is done with saving the killed status of the application. Otherwise, if RM crashes in the middle between user killing the application and writing the status to the store, RM will relaunch this application after it restarts.

YARN-1435.
Major bug reported by Tassapol Athiapinya and fixed by Xuan Gong (applications/distributed-shell)Distributed Shell should not run other commands except "sh", and run the custom script at the same time.

Currently, if we want to run custom script at DS. We can do it like this :
--shell_command sh --shell_script custom_script.sh
But it may be better to separate running shell_command and shell_script

YARN-1425.
Major bug reported by Omkar Vinit Joshi and fixed by Omkar Vinit Joshi TestRMRestart fails because MockRM.waitForState(AttemptId) uses current attempt instead of the attempt passed as argument

TestRMRestart is failing on trunk. Fixing it.

YARN-1423.
Major improvement reported by Sandy Ryza and fixed by Ted Malaska (scheduler)Support queue placement by secondary group in the Fair Scheduler

QueueMetrics holds its data in a static variable causing metrics to bleed over from test to test. clearQueueMetrics is to be called for tests that need to measure metrics correctly for a single test. jdk7 comes into play since tests are run out of order, and in the case make the metrics unreliable.

YARN-1416.
Major bug reported by Omkar Vinit Joshi and fixed by Jian He InvalidStateTransitions getting reported in multiple test cases even though they pass

It might be worth checking why they are reporting this.
Testcase : TestRMAppTransitions, TestRM
there are large number of such errors.
can't handle RMAppEventType.APP_UPDATE_SAVED at RMAppState.FAILED

When HA is turned on, {{YarnConfiguration#getSoketAddress()}} fetches rpc-addresses corresponding to the specified rm-id. This should only be for RM rpc-addresses. Other confs, like NM rpc-addresses shouldn't be affected by this.
Currently, the NM address settings in yarn-site.xml aren't reflected in the actual ports.

YARN-1409.
Major bug reported by Tsuyoshi OZAWA and fixed by Tsuyoshi OZAWA NonAggregatingLogHandler can throw RejectedExecutionException

YARN-1407.
Major bug reported by Sandy Ryza and fixed by Sandy Ryza RM Web UI and REST APIs should uniformly use YarnApplicationState

RMAppState isn't a public facing enum like YarnApplicationState, so we shouldn't return values or list filters that come from it. However, some Blocks and AppInfo are still using RMAppState.
It is not 100% clear to me whether or not fixing this would be a backwards-incompatible change. The change would only reduce the set of possible strings that the API returns, so I think not. We have also been changing the contents of RMAppState since 2.2.0, e.g. in YARN-891. It would still be good to fix this ASAP (i.e. for 2.2.1).

YARN-1405.
Major sub-task reported by Yesha Vora and fixed by Jian He RM hangs on shutdown if calling system.exit in serviceInit or serviceStart

Enable yarn.resourcemanager.recovery.enabled=true and Pass a local path to yarn.resourcemanager.fs.state-store.uri. such as "file:///tmp/MYTMP"
if the directory /tmp/MYTMP is not readable or writable, RM should crash and should print "Permission denied Error"
Currently, RM throws "java.io.FileNotFoundException: File file:/tmp/MYTMP/FSRMStateRoot/RMDTSecretManagerRoot does not exist" Error. RM returns Exiting status 1 but RM process does not shutdown.
Snapshot of Resource manager log:
2013-09-27 18:31:36,621 INFO security.NMTokenSecretManagerInRM (NMTokenSecretManagerInRM.java:rollMasterKey(97)) - Rolling master-key for nm-tokens
2013-09-27 18:31:36,694 ERROR resourcemanager.ResourceManager (ResourceManager.java:serviceStart(640)) - Failed to load/recover state
java.io.FileNotFoundException: File file:/tmp/MYTMP/FSRMStateRoot/RMDTSecretManagerRoot does not exist
at org.apache.hadoop.fs.RawLocalFileSystem.listStatus(RawLocalFileSystem.java:379)
at org.apache.hadoop.fs.FileSystem.listStatus(FileSystem.java:1478)
at org.apache.hadoop.fs.FileSystem.listStatus(FileSystem.java:1518)
at org.apache.hadoop.fs.ChecksumFileSystem.listStatus(ChecksumFileSystem.java:564)
at org.apache.hadoop.yarn.server.resourcemanager.recovery.FileSystemRMStateStore.loadRMDTSecretManagerState(FileSystemRMStateStore.java:188)
at org.apache.hadoop.yarn.server.resourcemanager.recovery.FileSystemRMStateStore.loadState(FileSystemRMStateStore.java:112)
at org.apache.hadoop.yarn.server.resourcemanager.ResourceManager.serviceStart(ResourceManager.java:635)
at org.apache.hadoop.service.AbstractService.start(AbstractService.java:193)
at org.apache.hadoop.yarn.server.resourcemanager.ResourceManager.main(ResourceManager.java:855)
2013-09-27 18:31:36,697 INFO util.ExitUtil (ExitUtil.java:terminate(124)) - Exiting with status 1

YARN-1403.
Major improvement reported by Sandy Ryza and fixed by Sandy Ryza Separate out configuration loading from QueueManager in the Fair Scheduler

YARN-1401.
Major bug reported by Gera Shegalov and fixed by Gera Shegalov (nodemanager)With zero sleep-delay-before-sigkill.ms, no signal is ever sent

If you set in yarn-site.xml yarn.nodemanager.sleep-delay-before-sigkill.ms=0 then an unresponsive child JVM is never killed. In MRv1, TT used to immediately SIGKILL in this case.

Distributed shell launched with the debug flag will run {{ApplicationMaster#dumpOutDebugInfo}}. This method launches an external process to run ls and print the contents of the current working directory. We've seen that this can cause the application master to hang on {{Process#waitFor}}.

YARN-1392.
Major new feature reported by Sandy Ryza and fixed by Sandy Ryza (scheduler)Allow sophisticated app-to-queue placement policies in the Fair Scheduler

Currently the Fair Scheduler supports app-to-queue placement by username. It would be beneficial to allow more sophisticated policies that rely on primary and secondary groups and fallbacks.

When a local resource that should already be present is requested again, the nodemanager checks to see if it still present. However the method it uses to check for presence is via File.exists() as the user of the nodemanager process. If the resource was a private resource localized for another user, it will be localized to a location that is not accessible by the nodemanager user. Therefore File.exists() returns false, the nodemanager mistakenly believes the resource is no longer available, and it proceeds to localize it over and over.

YARN-1378.
Major sub-task reported by Jian He and fixed by Jian He (resourcemanager)Implement a RMStateStore cleaner for deleting application/attempt info

Now that we are storing the final state of application/attempt instead of removing application/attempt info on application/attempt completion(YARN-891), we need a separate RMStateStore cleaner for cleaning the application/attempt state.

YARN-1374.
Blocker bug reported by Devaraj K and fixed by Karthik Kambatla (resourcemanager)Resource Manager fails to start due to ConcurrentModificationException

YARN-1358.
Minor test reported by Chuan Liu and fixed by Chuan Liu (client)TestYarnCLI fails on Windows due to line endings

The unit test fails on Windows due to incorrect line endings was used for comparing the output from command line output. Error messages are as follows.
{noformat}
junit.framework.ComparisonFailure: expected:<...argument for options[]
usage: application
...> but was:<...argument for options[
]
usage: application
...>
at junit.framework.Assert.assertEquals(Assert.java:85)
at junit.framework.Assert.assertEquals(Assert.java:91)
at org.apache.hadoop.yarn.client.cli.TestYarnCLI.testMissingArguments(TestYarnCLI.java:878)
{noformat}

This test fails on Windows due to incorrect use of batch script command. Error messages are as follows.
{noformat}
junit.framework.AssertionFailedError: expected:<java.nio.HeapByteBuffer[pos=0 lim=19 cap=19]> but was:<java.nio.HeapByteBuffer[pos=0 lim=19 cap=19]>
at junit.framework.Assert.fail(Assert.java:50)
at junit.framework.Assert.failNotEquals(Assert.java:287)
at junit.framework.Assert.assertEquals(Assert.java:67)
at junit.framework.Assert.assertEquals(Assert.java:74)
at org.apache.hadoop.yarn.server.nodemanager.containermanager.launcher.TestContainerLaunch.testContainerEnvVariables(TestContainerLaunch.java:508)
{noformat}

While trying to print a warning, two values of the wrong type (Resource instead of int) are passed into a String.format method call, leading to a runtime exception, in the file:
_trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/fair/QueueManager.java_.
The warning was intended to be printed whenever the resources don't fit into each other, either because the number of virtual cores or the memory is too small. I changed the %d's into %s, this way the warning will contain both the cores and the memory.

YARN-1349.
Major bug reported by Chris Nauroth and fixed by Chris Nauroth (client)yarn.cmd does not support passthrough to any arbitrary class.

The yarn shell script supports passthrough to calling any arbitrary class if the first argument is not one of the per-defined sub-commands. The equivalent cmd script does not implement this and instead fails trying to do a labeled goto to the first argument.

YARN-1343.
Critical bug reported by Alejandro Abdelnur and fixed by Alejandro Abdelnur (resourcemanager)NodeManagers additions/restarts are not reported as node updates in AllocateResponse responses to AMs

If a NodeManager joins the cluster or gets restarted, running AMs never receive the node update indicating the Node is running.

YARN-1335.
Major improvement reported by Sandy Ryza and fixed by Sandy Ryza (scheduler)Move duplicate code from FSSchedulerApp and FiCaSchedulerApp into SchedulerApplication

FSSchedulerApp and FiCaSchedulerApp use duplicate code in a lot of places. They both extend SchedulerApplication. We can move a lot of this duplicate code into SchedulerApplication.

YARN-1333.
Major improvement reported by Sandy Ryza and fixed by Tsuyoshi OZAWA (scheduler)Support blacklisting in the Fair Scheduler

YARN-1332.
Minor improvement reported by Sandy Ryza and fixed by Sebastian Wong In TestAMRMClient, replace assertTrue with assertEquals where possible

TestAMRMClient uses a lot of "assertTrue(amClient.ask.size() == 0)" where "assertEquals(0, amClient.ask.size())" would make it easier to see why it's failing at a glance.

YARN-1331.
Trivial bug reported by Chris Nauroth and fixed by Chris Nauroth (client)yarn.cmd exits with NoClassDefFoundError trying to run rmadmin or logs

The yarn shell script was updated so that the rmadmin and logs sub-commands launch {{org.apache.hadoop.yarn.client.cli.RMAdminCLI}} and {{org.apache.hadoop.yarn.client.cli.LogsCLI}}. The yarn.cmd script also needs to be updated so that the commands work on Windows.

Currently, we can enable RM HA configuration without multiple RM ids(YarnConfiguration.RM_HA_IDS). This behaviour can cause wrong operations. ResourceManager should verify that more than 1 RM id must be specified in RM-HA-IDs.
One idea is to support "strict mode" to enforce this check as configuration(e.g. yarn.resourcemanager.ha.strict-mode.enabled).

YARN-1323.
Major sub-task reported by Karthik Kambatla and fixed by Karthik Kambatla Set HTTPS webapp address along with other RPC addresses in HAUtil

YARN-1232 adds the ability to configure multiple RMs, but missed out the https web app address. Need to add that in.

YARN-1321.
Blocker bug reported by Alejandro Abdelnur and fixed by Alejandro Abdelnur (client)NMTokenCache is a singleton, prevents multiple AMs running in a single JVM to work correctly

NMTokenCache is a singleton. Because of this, if running multiple AMs in a single JVM NMTokens for the same node from different AMs step on each other and starting containers fail due to mismatch tokens.
The error observed in the client side is something like:
{code}
ERROR org.apache.hadoop.security.UserGroupInformation: PriviledgedActionException as:llama (auth:PROXY) via llama (auth:SIMPLE) cause:org.apache.hadoop.yarn.exceptions.YarnException: Unauthorized request to start container.
NMToken for application attempt : appattempt_1382038445650_0002_000001 was used for starting container with container token issued for application attempt : appattempt_1382038445650_0001_000001
{code}

YARN-1320.
Major bug reported by Tassapol Athiapinya and fixed by Xuan Gong (applications/distributed-shell)Custom log4j properties in Distributed shell does not work properly.

YARN-1318.
Blocker sub-task reported by Karthik Kambatla and fixed by Karthik Kambatla (resourcemanager)Promote AdminService to an Always-On service and merge in RMHAProtocolService

Per discussion in YARN-1068, we want AdminService to handle HA-admin operations in addition to the regular non-HA admin operations. To facilitate this, we need to move AdminService an Always-On service.

YARN-1315.
Major bug reported by Sandy Ryza and fixed by Sandy Ryza (resourcemanager , scheduler)TestQueueACLs should also test FairScheduler

YARN-1314.
Major bug reported by Tassapol Athiapinya and fixed by Xuan Gong (applications/distributed-shell)Cannot pass more than 1 argument to shell command

Today, APP_ADDED and APP_REMOVED are sent to the scheduler. They are misnomers as schedulers only deal with AppAttempts today. This JIRA is for fixing their names so that we can add App-level events in the near future, notably for work-preserving RM-restart.

Rethink for znode structure for RM HA is proposed in some JIRAs(YARN-659, YARN-1222). The motivation of this JIRA is quoted from Bikas' comment in YARN-1222:
{quote}
We should move to creating a node hierarchy for apps such that all znodes for an app are stored under an app znode instead of the app root znode. This will help in removeApplication and also in scaling better on ZK. The earlier code was written this way to ensure create/delete happens under a root znode for fencing. But given that we have moved to multi-operations globally, this isnt required anymore.
{quote}

YARN-1306.
Major bug reported by Wei Yan and fixed by Wei Yan Clean up hadoop-sls sample-conf according to YARN-1228

When yarn.resourcemanager.ha.enabled is true, RMHAProtocolService#serviceInit calls HAUtil.setAllRpcAddresses. If the configuration values are null, it just throws IllegalArgumentException.
It's messy to analyse which keys are null, so we should handle it and log the name of keys which are null.
A current log dump is as follows:
{code}
2013-10-15 06:24:53,431 INFO org.apache.hadoop.yarn.server.resourcemanager.ResourceManager: registered UNIX signal handlers for [TERM, HUP, INT]
2013-10-15 06:24:54,203 INFO org.apache.hadoop.service.AbstractService: Service RMHAProtocolService failed in state INITED; cause: java.lang.IllegalArgumentException: Property value must not be null
java.lang.IllegalArgumentException: Property value must not be null
at com.google.common.base.Preconditions.checkArgument(Preconditions.java:88)
at org.apache.hadoop.conf.Configuration.set(Configuration.java:816)
at org.apache.hadoop.conf.Configuration.set(Configuration.java:798)
at org.apache.hadoop.yarn.conf.HAUtil.setConfValue(HAUtil.java:100)
at org.apache.hadoop.yarn.conf.HAUtil.setAllRpcAddresses(HAUtil.java:105)
at org.apache.hadoop.yarn.server.resourcemanager.RMHAProtocolService.serviceInit(RMHAProtocolService.java:60)
at org.apache.hadoop.service.AbstractService.init(AbstractService.java:163)
at org.apache.hadoop.service.CompositeService.serviceInit(CompositeService.java:108)
at org.apache.hadoop.yarn.server.resourcemanager.ResourceManager.serviceInit(ResourceManager.java:187)
at org.apache.hadoop.service.AbstractService.init(AbstractService.java:163)
at org.apache.hadoop.yarn.server.resourcemanager.ResourceManager.main(ResourceManager.java:940)
{code}

YARN-1303.
Major improvement reported by Tassapol Athiapinya and fixed by Xuan Gong (applications/distributed-shell)Allow multiple commands separating with ";" in distributed-shell

In shell, we can do "ls; ls" to run 2 commands at once.
In distributed shell, this is not working. We should improve to allow this to occur. There are practical use cases that I know of to run multiple commands or to set environment variables before a command.

I was looking at https://builds.apache.org/job/PreCommit-YARN-Build/2165//testReport/org.apache.hadoop.yarn.sls/TestSLSRunner/testSimulatorRunning/
I am able to reproduce the failure locally.
I found that FairSchedulerConfiguration.getAllocationFile() doesn't read the yarn.scheduler.fair.allocation.file config entry from fair-scheduler.xml
This leads to the following:
{code}
Caused by: org.apache.hadoop.yarn.server.resourcemanager.scheduler.fair.AllocationConfigurationException: Bad fair scheduler config file: top-level element not <allocations>
at org.apache.hadoop.yarn.server.resourcemanager.scheduler.fair.QueueManager.reloadAllocs(QueueManager.java:302)
at org.apache.hadoop.yarn.server.resourcemanager.scheduler.fair.QueueManager.initialize(QueueManager.java:108)
at org.apache.hadoop.yarn.server.resourcemanager.scheduler.fair.FairScheduler.reinitialize(FairScheduler.java:1145)
{code}

Currently, in continuous scheduling (YARN-1010), in each round, the thread iterates over pre-ordered nodes and assigns tasks. This mechanism may overload the first several nodes, while the latter nodes have no tasks.
We should sort all nodes according to available resource. In each round, always assign tasks to nodes with larger capacity, which can balance the load distribution among all nodes.

The Fair Scheduler currently defaults the root queue's acl to empty and all other queues' acl to "*". Now that YARN-1258 enables configuring the root queue, we should reverse this. This will also bring the Fair Scheduler in line with the Capacity Scheduler.
We should also not trim the acl strings, which makes it impossible to only specify groups in an acl.

When LCE & cgroups are enabled, when a container is is killed (in this case by its owning AM, an MRAM) it seems to be a race condition at OS level when doing a SIGTERM/SIGKILL and when the OS does all necessary cleanup.
LCE code, after sending the SIGTERM/SIGKILL and getting the exitcode, immediately attempts to clean up the cgroups entry for the container. But this is failing with an error like:
{code}
2013-10-07 15:21:24,359 WARN org.apache.hadoop.yarn.server.nodemanager.LinuxContainerExecutor: Exit code from container container_1381179532433_0016_01_000011 is : 143
2013-10-07 15:21:24,359 DEBUG org.apache.hadoop.yarn.server.nodemanager.containermanager.container.Container: Processing container_1381179532433_0016_01_000011 of type UPDATE_DIAGNOSTICS_MSG
2013-10-07 15:21:24,359 DEBUG org.apache.hadoop.yarn.server.nodemanager.util.CgroupsLCEResourcesHandler: deleteCgroup: /run/cgroups/cpu/hadoop-yarn/container_1381179532433_0016_01_000011
2013-10-07 15:21:24,359 WARN org.apache.hadoop.yarn.server.nodemanager.util.CgroupsLCEResourcesHandler: Unable to delete cgroup at: /run/cgroups/cpu/hadoop-yarn/container_1381179532433_0016_01_000011
{code}
CgroupsLCEResourcesHandler.clearLimits() has logic to wait for 500 ms for AM containers to avoid this problem. it seems this should be done for all containers.
Still, waiting for extra 500ms seems too expensive.
We should look at a way of doing this in a more 'efficient way' from time perspective, may be spinning while the deleteCgroup() cannot be done with a minimal sleep and a timeout.

YARN-1283.
Major sub-task reported by Yesha Vora and fixed by Omkar Vinit Joshi Invalid 'url of job' mentioned in Job output with yarn.http.policy=HTTPS_ONLY

Only nodes in the RUNNING state are tracked by schedulers. When a node reconnects, RMNodeImpl.ReconnectNodeTransition tries to remove it, even if it's in the RUNNING state. The FairScheduler doesn't guard against this.
I think the best way to fix this is to check to see whether a node is RUNNING before telling the scheduler to remove it.

This would be useful for acls, maxRunningApps, scheduling modes, etc.
The allocation file should be able to accept both:
* An implicit root queue
* A root queue at the top of the hierarchy with all queues under/inside of it

YARN-1253.
Blocker new feature reported by Alejandro Abdelnur and fixed by Roman Shaposhnik (nodemanager)Changes to LinuxContainerExecutor to run containers as a single dedicated user in non-secure mode

When using cgroups we require LCE to be configured in the cluster to start containers.
When LCE starts containers as the user that submitted the job. While this works correctly in a secure setup, in an un-secure setup this presents a couple issues:
* LCE requires all Hadoop users submitting jobs to be Unix users in all nodes
* Because users can impersonate other users, any user would have access to any local file of other users
Particularly, the second issue is not desirable as a user could get access to ssh keys of other users in the nodes or if there are NFS mounts, get to other users data outside of the cluster.

YARN-1241.
Major bug reported by Sandy Ryza and fixed by Sandy Ryza In Fair Scheduler, maxRunningApps does not work for non-leaf queues

Setting the maxRunningApps property on a parent queue should make it that the sum of apps in all subqueues can't exceed it

YARN-1239.
Major sub-task reported by Bikas Saha and fixed by Jian He (resourcemanager)Save version information in the state store

When creating root dir for the first time we should write version 1. If root dir exists then we should check that the version in the state store matches the version from config.

YARN-1232.
Major sub-task reported by Karthik Kambatla and fixed by Karthik Kambatla (resourcemanager)Configuration to support multiple RMs

We should augment the configuration to allow users specify two RMs and the individual RPC addresses for them.

YARN-1222.
Major sub-task reported by Bikas Saha and fixed by Karthik Kambatla Make improvements in ZKRMStateStore for fencing

Using multi-operations for every ZK interaction.
In every operation, automatically creating/deleting a lock znode that is the child of the root znode. This is to achieve fencing by modifying the create/delete permissions on the root znode.

YARN-1210.
Major sub-task reported by Vinod Kumar Vavilapalli and fixed by Omkar Vinit Joshi During RM restart, RM should start a new attempt only when previous attempt exits for real

When RM recovers, it can wait for existing AMs to contact RM back and then kill them forcefully before even starting a new AM. Worst case, RM will start a new AppAttempt after waiting for 10 mins ( the expiry interval). This way we'll minimize multiple AMs racing with each other. This can help issues with downstream components like Pig, Hive and Oozie during RM restart.
In the mean while, new apps will proceed as usual as existing apps wait for recovery.
This can continue to be useful after work-preserving restart, so that AMs which can properly sync back up with RM can continue to run and those that don't are guaranteed to be killed before starting a new attempt.

YARN-1199.
Major improvement reported by Mit Desai and fixed by Mit Desai Make NM/RM Versions Available

Now as we have the NM and RM Versions available, we can display the YARN version of nodes running in the cluster.

YARN-1188.
Trivial bug reported by Akira AJISAKA and fixed by Tsuyoshi OZAWA The context of QueueMetrics becomes 'default' when using FairScheduler

I found the context of QueueMetrics changed to 'default' from 'yarn' when I was using FairScheduler.
The context should always be 'yarn' by adding an annotation to FSQueueMetrics like below:
{code}
+ @Metrics(context="yarn")
public class FSQueueMetrics extends QueueMetrics {
{code}

FileSystemRMStateStore writes directly to the destination file when storing state. However if the RM were to crash in the middle of the write, the recovery method could encounter a partially-written file and either outright crash during recovery or silently load incomplete state.
To avoid this, the data should be written to a temporary file and renamed to the destination file afterwards.

As described in MAPREDUCE-5501 sometimes M/R tests leave MRAppMaster java processes living for several minutes after successful completion of the corresponding test. There is a concurrency issue in MiniYARNCluster shutdown logic which leads to this. Sometimes RM stops before an app master sends it's last report, and then the app master keeps retrying for >6 minutes. In some cases it leads to failures in subsequent tests, and it affects performance of tests as app masters eat resources.

YARN-1182.
Major bug reported by Karthik Kambatla and fixed by Karthik Kambatla MiniYARNCluster creates and inits the RM/NM only on start()

MiniYARNCluster creates and inits the RM/NM only on start(). It should create and init() during init() itself.

YARN-1181.
Major sub-task reported by Karthik Kambatla and fixed by Karthik Kambatla Augment MiniYARNCluster to support HA mode

MiniYARNHACluster, along the lines of MiniYARNCluster, is needed for end-to-end HA tests.

YARN-1180.
Trivial bug reported by Thomas Graves and fixed by Chen He (capacityscheduler)Update capacity scheduler docs to include types on the configs

The capacity scheduler docs (http://hadoop.apache.org/docs/r2.1.0-beta/hadoop-yarn/hadoop-yarn-site/CapacityScheduler.html) don't include types for all the configs. For instance the minimum-user-limit-percent doesn't say its an Int. It also the only setting for the Resource Allocation configs that is an Int rather then a float.

YARN-1138.
Major bug reported by Yingda Chen and fixed by Chuan Liu (api)yarn.application.classpath is set to point to $HADOOP_CONF_DIR etc., which does not work on Windows

yarn-default.xml has "yarn.application.classpath" entry set to $HADOOP_CONF_DIR,$HADOOP_COMMON_HOME/share/hadoop/common/,$HADOOP_COMMON_HOME/share/hadoop/common/lib/,$HADOOP_HDFS_HOME/share/hadoop/hdfs/,$HADOOP_HDFS_HOME/share/hadoop/hdfs/lib/,$HADOOP_YARN_HOME/share/hadoop/yarn/*,$HADOOP_YARN_HOME/share/hadoop/yarn/lib. It does not work on Windows which needs to be fixed.

YARN-1121.
Major sub-task reported by Bikas Saha and fixed by Jian He (resourcemanager)RMStateStore should flush all pending store events before closing

on serviceStop it should wait for all internal pending events to drain before stopping.

YARN-1119.
Major test reported by Robert Parker and fixed by Mit Desai (resourcemanager)Add ClusterMetrics checks to tho TestRMNodeTransitions tests

YARN-1101 identified an issue where UNHEALTHY nodes could double decrement the active nodes. We should add checks for RUNNING node transitions.

YARN-1109.
Major improvement reported by Sandy Ryza and fixed by haosdent (nodemanager)Demote NodeManager "Sending out status for container" logs to debug

Diagnosing NodeManager and container launch problems is made more difficult by the enormous number of logs like
{code}
Sending out status for container: container_id {, app_attempt_id {, application_id {, id: 18, cluster_timestamp: 1377559361179, }, attemptId: 1, }, id: 1337, }, state: C_RUNNING, diagnostics: "Container killed by the ApplicationMaster.\n", exit_status: -1000
{code}
On an NM with a few containers I am seeing tens of these per second.

YARN-1101.
Major bug reported by Robert Parker and fixed by Robert Parker (resourcemanager)Active nodes can be decremented below 0

The issue is in RMNodeImpl where both RUNNING and UNHEALTHY states that transition to a deactive state (LOST, DECOMMISSIONED, REBOOTED) use the same DeactivateNodeTransition class. The DeactivateNodeTransition class naturally decrements the active node, however the in cases where the node has transition to UNHEALTHY the active count has already been decremented.

YARN-1098.
Major sub-task reported by Karthik Kambatla and fixed by Karthik Kambatla (resourcemanager)Separate out RM services into "Always On" and "Active"

From discussion on YARN-1027, it makes sense to separate out services that are stateful and stateless. The stateless services can run perennially irrespective of whether the RM is in Active/Standby state, while the stateful services need to be started on transitionToActive() and completely shutdown on transitionToStandby().
The external-facing stateless services should respond to the client/AM/NM requests depending on whether the RM is Active/Standby.

YARN-1068.
Major sub-task reported by Karthik Kambatla and fixed by Karthik Kambatla (resourcemanager)Add admin support for HA operations

Support HA admin operations to facilitate transitioning the RM to Active and Standby states.

YARN-1060.
Major bug reported by Sandy Ryza and fixed by Niranjan Singh (scheduler)Two tests in TestFairScheduler are missing @Test annotation

If the container launch fails then we send ContainerExitEvent. This event contains exitCode and diagnostic message. Today we are ignoring diagnostic message while handling this event inside ContainerImpl. Fixing it as it is useful in diagnosing the failure.

Go to the scheduler page in RM, and click any queue to display the detailed info. You'll find that none of the resources entries (used, min, or max) would display values.
It is because the values contain brackets ("<" and ">") and are not properly html-escaped.

YARN-1033.
Major sub-task reported by Nemon Lou and fixed by Karthik Kambatla Expose RM active/standby state to Web UI and REST API

Both active and standby RM shall expose it's web server and show it's current state (active or standby) on web page. Users should be able to access this information through the REST API as well.

YARN-1029.
Major sub-task reported by Bikas Saha and fixed by Karthik Kambatla Allow embedding leader election into the RM

It should be possible to embed common ActiveStandyElector into the RM such that ZooKeeper based leader election and notification is in-built. In conjunction with a ZK state store, this configuration will be a simple deployment option.

YARN-1028.
Major sub-task reported by Bikas Saha and fixed by Karthik Kambatla Add FailoverProxyProvider like capability to RMProxy

RMProxy layer currently abstracts RM discovery and implements it by looking up service information from configuration. Motivated by HDFS and using existing classes from Common, we can add failover proxy providers that may provide RM discovery in extensible ways.

The Yarn Scheduler is a fertile area of interest with different implementations, e.g., Fifo, Capacity and Fair schedulers. Meanwhile, several optimizations are also made to improve scheduler performance for different scenarios and workload. Each scheduler algorithm has its own set of features, and drives scheduling decisions by many factors, such as fairness, capacity guarantee, resource availability, etc. It is very important to evaluate a scheduler algorithm very well before we deploy it in a production cluster. Unfortunately, currently it is non-trivial to evaluate a scheduling algorithm. Evaluating in a real cluster is always time and cost consuming, and it is also very hard to find a large-enough cluster. Hence, a simulator which can predict how well a scheduler algorithm for some specific workload would be quite useful.
We want to build a Scheduler Load Simulator to simulate large-scale Yarn clusters and application loads in a single machine. This would be invaluable in furthering Yarn by providing a tool for researchers and developers to prototype new scheduler features and predict their behavior and performance with reasonable amount of confidence, there-by aiding rapid innovation.
The simulator will exercise the real Yarn ResourceManager removing the network factor by simulating NodeManagers and ApplicationMasters via handling and dispatching NM/AMs heartbeat events from within the same JVM.
To keep tracking of scheduler behavior and performance, a scheduler wrapper will wrap the real scheduler.
The simulator will produce real time metrics while executing, including:
* Resource usages for whole cluster and each queue, which can be utilized to configure cluster and queue's capacity.
* The detailed application execution trace (recorded in relation to simulated time), which can be analyzed to understand/validate the scheduler behavior (individual jobs turn around time, throughput, fairness, capacity guarantee, etc).
* Several key metrics of scheduler algorithm, such as time cost of each scheduler operation (allocate, handle, etc), which can be utilized by Hadoop developers to find the code spots and scalability limits.
The simulator will provide real time charts showing the behavior of the scheduler and its performance.
A short demo is available http://www.youtube.com/watch?v=6thLi8q0qLE, showing how to use simulator to simulate Fair Scheduler and Capacity Scheduler.

Currently scheduling for a node is done when a node heartbeats.
For large cluster where the heartbeat interval is set to several seconds this delays scheduling of incoming allocations significantly.
We could have a continuous loop scanning all nodes and doing scheduling. If there is availability AMs will get the allocation in the next heartbeat after the one that placed the request.

YARN-985.
Major improvement reported by Ravi Prakash and fixed by Ravi Prakash (nodemanager)Nodemanager should log where a resource was localized

When a resource is localized, we should log WHERE on the local disk it was localized. This helps in debugging afterwards (e.g. if the disk was to go bad).

YARN-976.
Major sub-task reported by Sandy Ryza and fixed by Sandy Ryza (documentation)Document the meaning of a virtual core

As virtual cores are a somewhat novel concept, it would be helpful to have thorough documentation that clarifies their meaning.

YARN-895.
Major sub-task reported by Jian He and fixed by Jian He (resourcemanager)RM crashes if it restarts while the state-store is down

YARN-891.
Major sub-task reported by Bikas Saha and fixed by Jian He (resourcemanager)Store completed application information in RM state store

Store completed application/attempt info in RMStateStore when application/attempt completes. This solves some problems like finished application get lost after RM restart and some other races like YARN-1195

YARN-888.
Major bug reported by Alejandro Abdelnur and fixed by Alejandro Abdelnur clean up POM dependencies

Intermediate 'pom' modules define dependencies inherited by leaf modules.
This is causing issues in intellij IDE.
We should normalize the leaf modules like in common, hdfs and tools where all dependencies are defined in each leaf module and the intermediate 'pom' module do not define any dependency.

getResources() will return a list of containers that allocated by RM. However, it is now return null directly. The worse thing is: if LOG.debug is enabled, then it will definitely cause NPE exception.

YARN-819.
Major sub-task reported by Robert Parker and fixed by Robert Parker (nodemanager , resourcemanager)ResourceManager and NodeManager should check for a minimum allowed version

Our use case is during upgrade on a large cluster several NodeManagers may not restart with the new version. Once the RM comes back up the NodeManager will re-register without issue to the RM.
The NM should report the version the RM. The RM should have a configuration to disallow the check (default), equal to the RM (to prevent config change for each release), equal to or greater than RM (to allow NM upgrades), and finally an explicit version or version range.
The RM should also have an configuration on how to treat the mismatch: REJECT, or REBOOT the NM.

YARN-807.
Major improvement reported by Sandy Ryza and fixed by Sandy Ryza When querying apps by queue, iterating over all apps is inefficient and limiting

The question "which apps are in queue x" can be asked via the RM REST APIs, through the ClientRMService, and through the command line. In all these cases, the question is answered by scanning through every RMApp and filtering by the app's queue name.
All schedulers maintain a mapping of queues to applications. I think it would make more sense to ask the schedulers which applications are in a given queue. This is what was done in MR1. This would also have the advantage of allowing a parent queue to return all the applications on leaf queues under it, and allow queue name aliases, as in the way that "root.default" and "default" refer to the same queue in the fair scheduler.

It might be good to require users to explicitly ask for this information, as it's a little more expensive to collect than the other fields in AppInfo.

YARN-764.
Major bug reported by Nemon Lou and fixed by Nemon Lou (resourcemanager)blank Used Resources on Capacity Scheduler page

Even when there are jobs running,used resources is empty on Capacity Scheduler page for leaf queue.(I use google-chrome on windows 7.)
After changing resource.java's toString method by replacing "<>" with "{}",this bug gets fixed.

YARN-709.
Major sub-task reported by Jian He and fixed by Jian He (resourcemanager)verify that new jobs submitted with old RM delegation tokens after RM restart are accepted

More elaborate test for restoring RM delegation tokens on RM restart.
New jobs with old RM delegation tokens should be accepted by new RM as long as the token is still valid

In the fair scheduler web UI, you can expand queue information. Refreshing the page causes the expansions to go away, which is annoying for someone who wants to monitor the scheduler page and needs to reopen all the queues they care about each time.

Hadoop 1.0 supported an option to turn on/off FairScheduler event logging using mapred.fairscheduler.eventlog.enabled. In Hadoop 2.0, it looks like this option has been removed (or not ported?) which causes event logging to be enabled by default and there is no way to turn it off.

As the first step, we go for resource change on RM side and expose admin APIs (admin protocol, CLI, REST and JMX API) later. In this jira, we will only contain changes in scheduler.
The flow to update node's resource and awareness in resource scheduling is:
1. Resource update is through admin API to RM and take effect on RMNodeImpl.
2. When next NM heartbeat for updating status comes, the RMNode's resource change will be aware and the delta resource is added to schedulerNode's availableResource before actual scheduling happens.
3. Scheduler do resource allocation according to new availableResource in SchedulerNode.
For more design details, please refer proposal and discussions in parent JIRA: YARN-291.

Running fair scheduler YARN shows that RM has lots of messages like the below.
{noformat}
INFO org.apache.hadoop.yarn.server.resourcemanager.scheduler.fair.AppSchedulable: Node offered to app: application_1357147147433_0002 reserved: false
{noformat}
They dont seem to tell much and same line is dumped many times in RM log. It would be good to have it improved with node information or moved to some other logging level with enough debug information

YARN-7.
Major sub-task reported by Arun C Murthy and fixed by Junping Du Add support for DistributedShell to ask for CPUs along with memory

MAPREDUCE-5550.
Major bug reported by Vrushali C and fixed by Gera Shegalov Task Status message (reporter.setStatus) not shown in UI with Hadoop 2.0

MAPREDUCE-5546.
Major bug reported by Chuan Liu and fixed by Chuan Liu mapred.cmd on Windows set HADOOP_OPTS incorrectly

MAPREDUCE-5522.
Minor bug reported by Jinghui Wang and fixed by Jinghui Wang (test)Incorrectly expect the array of JobQueueInfo returned by o.a.h.mapred.QueueManager#getJobQueueInfos to have a specific order.

MAPREDUCE-5052.
Critical bug reported by Kendall Thrapp and fixed by Chen He (jobhistoryserver , webapps)Job History UI and web services confusing job start time and job submit time

MAPREDUCE-5020.
Major bug reported by Trevor Robinson and fixed by Trevor Robinson (client)Compile failure with JDK8

MAPREDUCE-4680.
Major bug reported by Sandy Ryza and fixed by Robert Kanter (jobhistoryserver)Job history cleaner should only check timestamps of files in old enough directories

MAPREDUCE-4421.
Major improvement reported by Arun C Murthy and fixed by Jason Lowe Run MapReduce framework via the distributed cache

MAPREDUCE-3310.
Major improvement reported by Mathias Herberts and fixed by Alejandro Abdelnur (client)Custom grouping comparator cannot be set for Combiners

MAPREDUCE-1176.
Major new feature reported by BitsOfInfo and fixed by Mariappan Asokan FixedLengthInputFormat and FixedLengthRecordReader

Addition of FixedLengthInputFormat and FixedLengthRecordReader in the org.apache.hadoop.mapreduce.lib.input package. These two classes can be used when you need to read data from files containing fixed length (fixed width) records. Such files have no CR/LF (or any combination thereof), no delimiters etc, but each record is a fixed length, and extra data is padded with spaces. The data is one gigantic line within a file. When creating a job that specifies this input format, the job must have the "mapreduce.input.fixedlengthinputformat.record.length" property set as follows myJobConf.setInt("mapreduce.input.fixedlengthinputformat.record.length",[myFixedRecordLength]);
Please see javadoc for more details.

HDFS-5663.
Major improvement reported by Liang Xie and fixed by Liang Xie (hdfs-client)make the retry time and interval value configurable in openInfo()

Makes the retries and time between retries getting the length of the last block on file configurable. Below are the new configurations.
dfs.client.retry.times.get-last-block-length
dfs.client.retry.interval-ms.get-last-block-length
They are set to the 3 and 4000 respectively, these being what was previously hardcoded.

HDFS-5662.
Major improvement reported by Brandon Li and fixed by Brandon Li (namenode)Can't decommission a DataNode due to file's replication factor larger than the rest of the cluster size

HDFS-5538.
Major sub-task reported by Haohui Mai and fixed by Haohui Mai URLConnectionFactory should pick up the SSL related configuration by default

HDFS-5536.
Major sub-task reported by Haohui Mai and fixed by Haohui Mai Implement HTTP policy for Namenode and DataNode

Add new HTTP policy configuration. Users can use "dfs.http.policy" to control the HTTP endpoints for NameNode and DataNode. Specifically, The following values are supported:
- HTTP_ONLY : Service is provided only on http
- HTTPS_ONLY : Service is provided only on https
- HTTP_AND_HTTPS : Service is provided both on http and https
hadoop.ssl.enabled and dfs.https.enabled are deprecated. When the deprecated configuration properties are still configured, currently http policy is decided based on the following rules:
1. If dfs.http.policy is set to HTTPS_ONLY or HTTP_AND_HTTPS. It picks the specified policy, otherwise it proceeds to 2~4.
2. It picks HTTPS_ONLY if hadoop.ssl.enabled equals to true.
3. It picks HTTP_AND_HTTPS if dfs.https.enable equals to true.
4. It picks HTTP_ONLY for other configurations.

HDFS-5533.
Minor bug reported by Binglin Chang and fixed by Binglin Chang (snapshots)Symlink delete/create should be treated as DELETE/CREATE in snapshot diff report

HDFS-5532.
Major improvement reported by Vinayakumar B and fixed by Vinayakumar B (webhdfs)Enable the webhdfs by default to support new HDFS web UI

HDFS-5506.
Major sub-task reported by Haohui Mai and fixed by Haohui Mai Use URLConnectionFactory in DelegationTokenFetcher

HDFS-5504.
Major bug reported by Vinayakumar B and fixed by Vinayakumar B (snapshots)In HA mode, OP_DELETE_SNAPSHOT is not decrementing the safemode threshold, leads to NN safemode.

HDFS-5502.
Major sub-task reported by Haohui Mai and fixed by Haohui Mai Fix HTTPS support in HsftpFileSystem

Fix the https support in HsftpFileSystem. With the change the client now verifies the server certificate. In particular, client side will verify the Common Name of the certificate using a strategy specified by the configuration property "hadoop.ssl.hostname.verifier".

HDFS-5495.
Major improvement reported by Andrew Wang and fixed by Jarek Jarcec Cecho Remove further JUnit3 usages from HDFS

HDFS-5489.
Major sub-task reported by Haohui Mai and fixed by Haohui Mai Use TokenAspect in WebHDFSFileSystem

HDFS-5488.
Major sub-task reported by Haohui Mai and fixed by Haohui Mai Clean up TestHftpURLTimeout

HDFS-5283.
Critical bug reported by Vinayakumar B and fixed by Vinayakumar B (snapshots)NN not coming out of startup safemode due to under construction blocks only inside snapshots also counted in safemode threshhold

HDFS-5281.
Major sub-task reported by Brandon Li and fixed by Brandon Li (nfs)COMMIT request should not block

HADOOP-10087.
Major bug reported by Yu Gao and fixed by Colin Patrick McCabe (security)UserGroupInformation.getGroupNames() fails to return primary group first when JniBasedUnixGroupsMappingWithFallback is used

The 'du' (disk usage command from Unix) script refresh monitor is now configurable in the same way as its 'df' counterpart, via the property 'fs.du.interval', the default of which is 10 minute (in ms).

HADOOP-7344.
Major bug reported by Daryn Sharp and fixed by Colin Patrick McCabe (fs)globStatus doesn't grok groupings with a slash

Hadoop 2.2.0 Release Notes

Hadoop 2.2.0 Release Notes

These release notes include new developer and user-facing incompatibilities, features, and major improvements.

Changes since Hadoop 2.1.1-beta

YARN-1278.
Blocker bug reported by Yesha Vora and fixed by Hitesh Shah New AM does not start after rm restart

The new AM fails to start after RM restarts. It fails to start new Application master and job fails with below error.
/usr/bin/mapred job -status job_1380985373054_0001
13/10/05 15:04:04 INFO client.RMProxy: Connecting to ResourceManager at hostname
Job: job_1380985373054_0001
Job File: /user/abc/.staging/job_1380985373054_0001/job.xml
Job Tracking URL : http://hostname:8088/cluster/app/application_1380985373054_0001
Uber job : false
Number of maps: 0
Number of reduces: 0
map() completion: 0.0
reduce() completion: 0.0
Job state: FAILED
retired: false
reason for failure: There are no failed tasks for the job. Job is failed due to some other reason and reason can be found in the logs.
Counters: 0

YARN-1277.
Major sub-task reported by Suresh Srinivas and fixed by Omkar Vinit Joshi Add http policy support for YARN daemons

This YARN part of HADOOP-10022.

YARN-1274.
Blocker bug reported by Alejandro Abdelnur and fixed by Siddharth Seth (nodemanager)LCE fails to run containers that don't have resources to localize

LCE container launch assumes the usercache/USER directory exists and it is owned by the user running the container process.
But the directory is created only if there are resources to localize by the LCE localization command, if there are not resourcdes to localize, LCE localization never executes and launching fails reporting 255 exit code and the NM logs have something like:
{code}
2013-10-04 14:07:56,425 INFO org.apache.hadoop.yarn.server.nodemanager.ContainerExecutor: main : command provided 1
2013-10-04 14:07:56,425 INFO org.apache.hadoop.yarn.server.nodemanager.ContainerExecutor: main : user is llama
2013-10-04 14:07:56,425 INFO org.apache.hadoop.yarn.server.nodemanager.ContainerExecutor: Can't create directory llama in /yarn/nm/usercache/llama/appcache/application_1380853306301_0004/container_1380853306301_0004_01_000004 - Permission denied
{code}

YARN-1273.
Major bug reported by Hitesh Shah and fixed by Hitesh Shah Distributed shell does not account for start container failures reported asynchronously.

The error is shown below in the comments.
MAPREDUCE-2374 fixed this by removing "-c" when running the container launch script. It looks like the "-c" got brought back during the windows branch merge, so we should remove it again.

YARN-1262.
Major bug reported by Sandy Ryza and fixed by Karthik Kambatla TestApplicationCleanup relies on all containers assigned in a single heartbeat

TestApplicationCleanup submits container requests and waits for allocations to come in. It only sends a single node heartbeat to the node, expecting multiple containers to be assigned on this heartbeat, which not all schedulers do by default.
This is causing the test to fail when run with the Fair Scheduler.

YARN-1260.
Major sub-task reported by Yesha Vora and fixed by Omkar Vinit Joshi RM_HOME link breaks when webapp.https.address related properties are not specified

This issue happens in multiple node cluster where resource manager and node manager are running on different machines.
Steps to reproduce:
1) set yarn.resourcemanager.hostname = <resourcemanager host> in yarn-site.xml
2) set hadoop.ssl.enabled = true in core-site.xml
3) Do not specify below property in yarn-site.xml
yarn.nodemanager.webapp.https.address and yarn.resourcemanager.webapp.https.address
Here, the default value of above two property will be considered.
4) Go to nodemanager web UI "https://<nodemanager host>:8044/node"
5) Click on RM_HOME link
This link redirects to "https://<nodemanager host>:8090/cluster" instead "https://<resourcemanager host>:8090/cluster"

A container can set token service metadata for a service, say shuffle_service. If that service does not exist then the errors is silently ignored. Later, when the next container wants to access data written to shuffle_service by the first task, then it fails because the service does not have the token that was supposed to be set by the first task.

Before launching the container, NM is using the same credential object and so is polluting what container should see. We should fix this.

YARN-1251.
Major bug reported by Junping Du and fixed by Xuan Gong (applications/distributed-shell)TestDistributedShell#TestDSShell failed with timeout

TestDistributedShell#TestDSShell on trunk Jenkins are failed consistently recently.
The Stacktrace is:
{code}
java.lang.Exception: test timed out after 90000 milliseconds
at com.google.protobuf.LiteralByteString.<init>(LiteralByteString.java:234)
at com.google.protobuf.ByteString.copyFromUtf8(ByteString.java:255)
at org.apache.hadoop.ipc.protobuf.ProtobufRpcEngineProtos$RequestHeaderProto.getMethodNameBytes(ProtobufRpcEngineProtos.java:286)
at org.apache.hadoop.ipc.protobuf.ProtobufRpcEngineProtos$RequestHeaderProto.getSerializedSize(ProtobufRpcEngineProtos.java:462)
at com.google.protobuf.AbstractMessageLite.writeDelimitedTo(AbstractMessageLite.java:84)
at org.apache.hadoop.ipc.ProtobufRpcEngine$RpcMessageWithHeader.write(ProtobufRpcEngine.java:302)
at org.apache.hadoop.ipc.Client$Connection.sendRpcRequest(Client.java:989)
at org.apache.hadoop.ipc.Client.call(Client.java:1377)
at org.apache.hadoop.ipc.Client.call(Client.java:1357)
at org.apache.hadoop.ipc.ProtobufRpcEngine$Invoker.invoke(ProtobufRpcEngine.java:206)
at $Proxy70.getApplicationReport(Unknown Source)
at org.apache.hadoop.yarn.api.impl.pb.client.ApplicationClientProtocolPBClientImpl.getApplicationReport(ApplicationClientProtocolPBClientImpl.java:137)
at sun.reflect.GeneratedMethodAccessor40.invoke(Unknown Source)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:597)
at org.apache.hadoop.io.retry.RetryInvocationHandler.invokeMethod(RetryInvocationHandler.java:185)
at org.apache.hadoop.io.retry.RetryInvocationHandler.invoke(RetryInvocationHandler.java:101)
at $Proxy71.getApplicationReport(Unknown Source)
at org.apache.hadoop.yarn.client.api.impl.YarnClientImpl.getApplicationReport(YarnClientImpl.java:195)
at org.apache.hadoop.yarn.applications.distributedshell.Client.monitorApplication(Client.java:622)
at org.apache.hadoop.yarn.applications.distributedshell.Client.run(Client.java:597)
at org.apache.hadoop.yarn.applications.distributedshell.TestDistributedShell.testDSShell(TestDistributedShell.java:125)
{code}
For details, please refer:
https://builds.apache.org/job/PreCommit-YARN-Build/2039//testReport/

YARN-1247.
Major bug reported by Roman Shaposhnik and fixed by Roman Shaposhnik (nodemanager)test-container-executor has gotten out of sync with the changes to container-executor

If run under the super-user account test-container-executor.c fails in multiple different places. It would be nice to fix it so that we have better testing of LCE functionality.

Since there is no yarn history server it becomes difficult to determine what the status of an old application is. One has to be familiar with the state transition in yarn to know what means a success.
We should add a log at info level that captures what the finalStatus of an app is. This would be helpful while debugging applications if the RM has restarted and we no longer can use the UI.

YARN-1236.
Major bug reported by Sandy Ryza and fixed by Sandy Ryza (resourcemanager)FairScheduler setting queue name in RMApp is not working

The fair scheduler sometimes picks a different queue than the one an application was submitted to, such as when user-as-default-queue is turned on. It needs to update the queue name in the RMApp so that this choice will be reflected in the UI.
This isn't working because the scheduler is looking up the RMApp by application attempt id instead of app id and failing to find it.

YARN-1229.
Blocker bug reported by Tassapol Athiapinya and fixed by Xuan Gong (nodemanager)Define constraints on Auxiliary Service names. Change ShuffleHandler service name from mapreduce.shuffle to mapreduce_shuffle.

I run sleep job. If AM fails to start, this exception could occur:
13/09/20 11:00:23 INFO mapreduce.Job: Job job_1379673267098_0020 failed with state FAILED due to: Application application_1379673267098_0020 failed 1 times due to AM Container for appattempt_1379673267098_0020_000001 exited with exitCode: 1 due to: Exception from container-launch:
org.apache.hadoop.util.Shell$ExitCodeException: /myappcache/application_1379673267098_0020/container_1379673267098_0020_01_000001/launch_container.sh: line 12: export: `NM_AUX_SERVICE_mapreduce.shuffle=AAA0+gAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA=
': not a valid identifier
at org.apache.hadoop.util.Shell.runCommand(Shell.java:464)
at org.apache.hadoop.util.Shell.run(Shell.java:379)
at org.apache.hadoop.util.Shell$ShellCommandExecutor.execute(Shell.java:589)
at org.apache.hadoop.yarn.server.nodemanager.DefaultContainerExecutor.launchContainer(DefaultContainerExecutor.java:195)
at org.apache.hadoop.yarn.server.nodemanager.containermanager.launcher.ContainerLaunch.call(ContainerLaunch.java:270)
at org.apache.hadoop.yarn.server.nodemanager.containermanager.launcher.ContainerLaunch.call(ContainerLaunch.java:78)
at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:303)
at java.util.concurrent.FutureTask.run(FutureTask.java:138)
at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
at java.lang.Thread.run(Thread.java:662)
.Failing this attempt.. Failing the application.

Currently the Fair Scheduler is configured in two ways
* An allocations file that has a different format than the standard Hadoop configuration file, which makes it easier to specify hierarchical objects like queues and their properties.
* With properties like yarn.scheduler.fair.max.assign that are specified in the standard Hadoop configuration format.
The standard and default way of configuring it is to use fair-scheduler.xml as the allocations file and to put the yarn.scheduler properties in yarn-site.xml.
It is also possible to specify a different file as the allocations file, and to place the yarn.scheduler properties in fair-scheduler.xml, which will be interpreted as in the standard Hadoop configuration format. This flexibility is both confusing and unnecessary.
Additionally, the allocation file is loaded as fair-scheduler.xml from the classpath if it is not specified, but is loaded as a File if it is. This causes two problems
1. We see different behavior when not setting the yarn.scheduler.fair.allocation.file, and setting it to fair-scheduler.xml, which is its default.
2. Classloaders may choose to cache resources, which can break the reload logic when yarn.scheduler.fair.allocation.file is not specified.
We should never allow the yarn.scheduler properties to go into fair-scheduler.xml. And we should always load the allocations file as a file, not as a resource on the classpath. To preserve existing behavior and allow loading files from the classpath, we can look for files on the classpath, but strip of their scheme and interpret them as Files.

While running a Hive join operation on Yarn, I saw exception as described below. This is caused by FSDownload copy the files into a temp file and change the suffix into ".tmp" before unpacking it. In unpack(), it uses FileUtil.unTar() which will determine if the file is "gzipped" by looking at the file suffix:
{code}
boolean gzipped = inFile.toString().endsWith("gz");
{code}
To fix this problem, we can remove the ".tmp" in the temp file name.
Here is the detailed exception:
org.apache.commons.compress.archivers.tar.TarArchiveInputStream.getNextTarEntry(TarArchiveInputStream.java:240)
at org.apache.hadoop.fs.FileUtil.unTarUsingJava(FileUtil.java:676)
at org.apache.hadoop.fs.FileUtil.unTar(FileUtil.java:625)
at org.apache.hadoop.yarn.util.FSDownload.unpack(FSDownload.java:203)
at org.apache.hadoop.yarn.util.FSDownload.call(FSDownload.java:287)
at org.apache.hadoop.yarn.util.FSDownload.call(FSDownload.java:50)
at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:334)
at java.util.concurrent.FutureTask.run(FutureTask.java:166)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:334)
at java.util.concurrent.FutureTask.run(FutureTask.java:166)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1110)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603)
at java.lang.Thread.run(Thread.java:722)

YARN-1215.
Major bug reported by Chuan Liu and fixed by Chuan Liu (api)Yarn URL should include userinfo

In the {{org.apache.hadoop.yarn.api.records.URL}} class, we don't have an userinfo as part of the URL. When converting a {{java.net.URI}} object into the YARN URL object in {{ConverterUtils.getYarnUrlFromURI()}} method, we will set uri host as the url host. If the uri has a userinfo part, the userinfo is discarded. This will lead to information loss if the original uri has the userinfo, e.g. foo://username:password@example.com will be converted to foo://example.com and username/password information is lost during the conversion.

YARN-1214.
Critical sub-task reported by Jian He and fixed by Jian He (resourcemanager)Register ClientToken MasterKey in SecretManager after it is saved

Currently, app attempt ClientToken master key is registered before it is saved. This can cause problem that before the master key is saved, client gets the token and RM also crashes, RM cannot reloads the master key back after it restarts as it is not saved. As a result, client is holding an invalid token.
We can register the client token master key after it is saved in the store.

YARN-1213.
Major improvement reported by Sandy Ryza and fixed by Sandy Ryza (scheduler)Restore config to ban submitting to undeclared pools in the Fair Scheduler

There is no yarn property available to configure https port for Resource manager, nodemanager and history server. Currently, Yarn services uses the port defined for http [defined by 'mapreduce.jobhistory.webapp.address','yarn.nodemanager.webapp.address', 'yarn.resourcemanager.webapp.address'] for running services on https protocol.
Yarn should have list of property to assign https port for RM, NM and JHS.
It can be like below.
yarn.nodemanager.webapp.https.address
yarn.resourcemanager.webapp.https.address
mapreduce.jobhistory.webapp.https.address

YARN-1203.
Major sub-task reported by Yesha Vora and fixed by Omkar Vinit Joshi Application Manager UI does not appear with Https enabled

Need to add support to disable 'hadoop.ssl.enabled' for MR jobs.
A job should be able to run on http protocol by setting 'hadoop.ssl.enabled' property at job level.

YARN-1141.
Major bug reported by Zhijie Shen and fixed by Zhijie Shen Updating resource requests should be decoupled with updating blacklist

Currently, in CapacityScheduler and FifoScheduler, blacklist is updated together with resource requests, only when the incoming resource requests are not empty. Therefore, when the incoming resource requests are empty, the blacklist will not be updated even when blacklist additions and removals are not empty.

In the case when log aggregation is enabled, if a user submits MapReduce job and runs $ yarn logs -applicationId <app ID> while the YARN application is running, the command will return no message and return user back to shell. It is nice to tell the user that log aggregation is in progress.
{code}
-bash-4.1$ /usr/bin/yarn logs -applicationId application_1377900193583_0002
-bash-4.1$
{code}
At the same time, if invalid application ID is given, YARN CLI should say that the application ID is incorrect rather than throwing NoSuchElementException.
{code}
$ /usr/bin/yarn logs -applicationId application_00000
Exception in thread "main" java.util.NoSuchElementException
at com.google.common.base.AbstractIterator.next(AbstractIterator.java:75)
at org.apache.hadoop.yarn.util.ConverterUtils.toApplicationId(ConverterUtils.java:124)
at org.apache.hadoop.yarn.util.ConverterUtils.toApplicationId(ConverterUtils.java:119)
at org.apache.hadoop.yarn.logaggregation.LogDumper.run(LogDumper.java:110)
at org.apache.hadoop.yarn.logaggregation.LogDumper.main(LogDumper.java:255)
{code}

YARN-1128.
Major bug reported by Sandy Ryza and fixed by Karthik Kambatla (scheduler)FifoPolicy.computeShares throws NPE on empty list of Schedulables

FifoPolicy gives all of a queue's share to the earliest-scheduled application.
{code}
Schedulable earliest = null;
for (Schedulable schedulable : schedulables) {
if (earliest == null ||
schedulable.getStartTime() < earliest.getStartTime()) {
earliest = schedulable;
}
}
earliest.setFairShare(Resources.clone(totalResources));
{code}
If the queue has no schedulables in it, earliest will be left null, leading to an NPE on the last line.

YARN-1090.
Major bug reported by Yesha Vora and fixed by Jian He Job does not get into Pending State

When there is no resource available to run a job, next job should go in pending state. RM UI should show next job as pending app and the counter for the pending app should be incremented.
But Currently. Next job stays in ACCEPTED state and No AM has been assigned to this job.Though Pending App count is not incremented.
Running 'job status <nextjob>' shows job state=PREP.
$ mapred job -status job_1377122233385_0002
13/08/21 21:59:23 INFO client.RMProxy: Connecting to ResourceManager at host1/ip1
Job: job_1377122233385_0002
Job File: /ABC/.staging/job_1377122233385_0002/job.xml
Job Tracking URL : http://host1:port1/application_1377122233385_0002/
Uber job : false
Number of maps: 0
Number of reduces: 0
map() completion: 0.0
reduce() completion: 0.0
Job state: PREP
retired: false
reason for failure:

YARN-1070.
Major sub-task reported by Hitesh Shah and fixed by Zhijie Shen (nodemanager)ContainerImpl State Machine: Invalid event: CONTAINER_KILLED_ON_REQUEST at CONTAINER_CLEANEDUP_AFTER_KILL

We found a case where our rack resolve script was not returning rack due to problem with resolving host address. This exception was see in RackResolver.java as NPE, ultimately caught in RMContainerAllocator.
{noformat}
2013-08-01 07:11:37,708 ERROR [RMCommunicator Allocator] org.apache.hadoop.mapreduce.v2.app.rm.RMContainerAllocator: ERROR IN CONTACTING RM.
java.lang.NullPointerException
at org.apache.hadoop.yarn.util.RackResolver.coreResolve(RackResolver.java:99)
at org.apache.hadoop.yarn.util.RackResolver.resolve(RackResolver.java:92)
at org.apache.hadoop.mapreduce.v2.app.rm.RMContainerAllocator$ScheduledRequests.assignMapsWithLocality(RMContainerAllocator.java:1039)
at org.apache.hadoop.mapreduce.v2.app.rm.RMContainerAllocator$ScheduledRequests.assignContainers(RMContainerAllocator.java:925)
at org.apache.hadoop.mapreduce.v2.app.rm.RMContainerAllocator$ScheduledRequests.assign(RMContainerAllocator.java:861)
at org.apache.hadoop.mapreduce.v2.app.rm.RMContainerAllocator$ScheduledRequests.access$400(RMContainerAllocator.java:681)
at org.apache.hadoop.mapreduce.v2.app.rm.RMContainerAllocator.heartbeat(RMContainerAllocator.java:219)
at org.apache.hadoop.mapreduce.v2.app.rm.RMCommunicator$1.run(RMCommunicator.java:243)
at java.lang.Thread.run(Thread.java:722)
{noformat}

The Capacity Scheduler documents the yarn.scheduler.capacity.root.<queue-path>.acl_administer_queue config option for controlling who can administer a queue, but it is not hooked up to anything. The Fair Scheduler could make use of a similar option as well. This is a feature-parity regression from MR1.

YARN-890.
Major bug reported by Trupti Dhavle and fixed by Xuan Gong (resourcemanager)The roundup for memory values on resource manager UI is misleading

YARN-876.
Major bug reported by PengZhang and fixed by PengZhang (resourcemanager)Node resource is added twice when node comes back from unhealthy to healthy

When an unhealthy restarts, its resource maybe added twice in scheduler.
First time is at node's reconnection, while node's final state is still "UNHEALTHY".
And second time is at node's update, while node's state changing from "UNHEALTHY" to "HEALTHY".

MAPREDUCE-5533.
Major bug reported by Tassapol Athiapinya and fixed by Xuan Gong (applicationmaster)Speculative execution does not function for reduce

MAPREDUCE-5531.
Blocker sub-task reported by Robert Kanter and fixed by Robert Kanter (mrv1 , mrv2)Binary and source incompatibility in mapreduce.TaskID and mapreduce.TaskAttemptID between branch-1 and branch-2

MAPREDUCE-5530.
Blocker sub-task reported by Robert Kanter and fixed by Robert Kanter (mrv1 , mrv2)Binary and source incompatibility in mapred.lib.CombineFileInputFormat between branch-1 and branch-2

MAPREDUCE-5529.
Blocker sub-task reported by Robert Kanter and fixed by Robert Kanter (mrv1 , mrv2)Binary incompatibilities in mapred.lib.TotalOrderPartitioner between branch-1 and branch-2

During review of symbolic links, many issues were found related impact on semantics of existing APIs such FileSystem#listStatus, FileSystem#globStatus etc. There were also many issues brought up about symbolic links and the impact on security and functionality of HDFS. All these issues will be address in the upcoming release 2.3. Until then the feature is temporarily disabled.

HADOOP-10017.
Major sub-task reported by Jing Zhao and fixed by Haohui Mai Fix NPE in DFSClient#getDelegationToken when doing Distcp from a secured cluster to an insecured cluster

Changes since Hadoop 2.1.0-beta

Running TestContainerLogsPage on trunk while Native IO is enabled makes it fail

YARN-1189.
Blocker bug reported by Jason Lowe and fixed by Omkar Vinit Joshi NMTokenSecretManagerInNM is not being told when applications have finished

The {{appFinished}} method is not being called when applications have finished. This causes a couple of leaks as {{oldMasterKeys}} and {{appToAppAttemptMap}} are never being pruned.

YARN-1184.
Major bug reported by J.Andreina and fixed by Chris Douglas (capacityscheduler , resourcemanager)ClassCastException is thrown during preemption When a huge job is submitted to a queue B whose resources is used by a job in queueA

On a secure cluster, an invalid key to HMAC error is thrown when trying to get an application report for an application with an attempt that has unregistered.

YARN-1144.
Critical bug reported by Alejandro Abdelnur and fixed by Alejandro Abdelnur (resourcemanager)Unmanaged AMs registering a tracking URI should not be proxy-fied

Unmanaged AMs do not run in the cluster, their tracking URL should not be proxy-fied.

YARN-1137.
Major improvement reported by Alejandro Abdelnur and fixed by Roman Shaposhnik (nodemanager)Add support whitelist for system users to Yarn container-executor.c

Currently container-executor.c has a banned set of users (mapred, hdfs & bin) and configurable min.user.id (defaulting to 1000).
This presents a problem for systems that run as system users (below 1000) if these systems want to start containers.
Systems like Impala fit in this category. A (local) 'impala' system user is created when installing Impala on the nodes.
Note that the same thing happens when installing system like HDFS, Yarn, Oozie, from packages (Bigtop); local system users are created.
For Impala to be able to run containers in a secure cluster, the 'impala' system user must whitelisted.
For this, adding a configuration 'allowed.system.users' option in the container-executor.cfg and the logic in container-executor.c would allow the usernames in that list.
Because system users are not guaranteed to have the same UID in different machines, the 'allowed.system.users' property should use usernames and not UIDs.

YARN-1124.
Blocker bug reported by Omkar Vinit Joshi and fixed by Xuan Gong By default yarn application -list should display all the applications in a state other than FINISHED / FAILED

Today we are just listing application in RUNNING state by default for "yarn application -list". Instead we should show all the applications which are either submitted/accepted/running.

In YARN-557, we added some code to make {{ApplicationConstants.Environment.USER}} has OS-specific definition in order to fix the unit test TestUnmanagedAMLauncher. In YARN-571, the relevant test code was corrected. In YARN-602, we actually will explicitly set the environment variables for the child containers. With these changes, I think we can revert the YARN-557 change to make {{ApplicationConstants.Environment.USER}} OS neutral. The main benefit is that we can use the same method over the Enum constants. This should also fix the TestContainerLaunch#testContainerEnvVariables failure on Windows.

YARN-1117.
Major improvement reported by Tassapol Athiapinya and fixed by Xuan Gong (client)Improve help message for $ yarn applications and $yarn node

There is standardization of help message in YARN-1080. It is nice to have similar changes for $ yarn appications and yarn node

YARN-1116.
Major sub-task reported by Jian He and fixed by Jian He (resourcemanager)Populate AMRMTokens back to AMRMTokenSecretManager after RM restarts

The AMRMTokens are now only saved in RMStateStore and not populated back to AMRMTokenSecretManager after RM restarts. This is more needed now since AMRMToken also becomes used in non-secure env.

If secure RM with recovery enabled is restarted while oozie jobs are running rm fails to come up.

YARN-1101.
Major bug reported by Robert Parker and fixed by Robert Parker (resourcemanager)Active nodes can be decremented below 0

The issue is in RMNodeImpl where both RUNNING and UNHEALTHY states that transition to a deactive state (LOST, DECOMMISSIONED, REBOOTED) use the same DeactivateNodeTransition class. The DeactivateNodeTransition class naturally decrements the active node, however the in cases where the node has transition to UNHEALTHY the active count has already been decremented.

In kerberos setup it's expected for a http client to authenticate to kerberos before allowing user to browse any information.

YARN-1083.
Major bug reported by Yesha Vora and fixed by Zhijie Shen (resourcemanager)ResourceManager should fail when yarn.nm.liveness-monitor.expiry-interval-ms is set less than heartbeat interval

if 'yarn.nm.liveness-monitor.expiry-interval-ms' is set to less than heartbeat iterval, all the node managers will be added in 'Lost Nodes'
Instead, Resource Manager should validate these property and It should fail to start if combination of such property is invalid.

YARN-1080.
Major improvement reported by Tassapol Athiapinya and fixed by Xuan Gong (client)Improve help message for $ yarn logs

There are 2 parts I am proposing in this jira. They can be fixed together in one patch.
1. Standardize help message for required parameter of $ yarn logs
YARN CLI has a command "logs" ($ yarn logs). The command always requires a parameter of "-applicationId <arg>". However, help message of the command does not make it clear. It lists -applicationId as optional parameter. If I don't set it, YARN CLI will complain this is missing. It is better to use standard required notation used in other Linux command for help message. Any user familiar to the command can understand that this parameter is needed more easily.
{code:title=current help message}
-bash-4.1$ yarn logs
usage: general options are:
-applicationId <arg> ApplicationId (required)
-appOwner <arg> AppOwner (assumed to be current user if not
specified)
-containerId <arg> ContainerId (must be specified if node address is
specified)
-nodeAddress <arg> NodeAddress in the format nodename:port (must be
specified if container id is specified)
{code}
{code:title=proposed help message}
-bash-4.1$ yarn logs
usage: yarn logs -applicationId <application ID> [OPTIONS]
general options are:
-appOwner <arg> AppOwner (assumed to be current user if not
specified)
-containerId <arg> ContainerId (must be specified if node address is
specified)
-nodeAddress <arg> NodeAddress in the format nodename:port (must be
specified if container id is specified)
{code}
2. Add description for help command. As far as I know, a user cannot get logs for running job. Since I spent some time trying to get logs of running applications, it should be nice to say this in command description.
{code:title=proposed help}
Retrieve logs for completed/killed YARN application
usage: general options are...
{code}

The three unit tests fail on Windows due to host name resolution differences on Windows, i.e. 127.0.0.1 does not resolve to host name "localhost".
{noformat}
org.apache.hadoop.security.token.SecretManager$InvalidToken: Given Container container_0_0000_01_000000 identifier is not valid for current Node manager. Expected : 127.0.0.1:12345 Found : localhost:12345
{noformat}
{noformat}
testNMConnectionToRM(org.apache.hadoop.yarn.server.nodemanager.TestNodeStatusUpdater) Time elapsed: 8343 sec <<< FAILURE!
org.junit.ComparisonFailure: expected:<[localhost]:12345> but was:<[127.0.0.1]:12345>
at org.junit.Assert.assertEquals(Assert.java:125)
at org.junit.Assert.assertEquals(Assert.java:147)
at org.apache.hadoop.yarn.server.nodemanager.TestNodeStatusUpdater$MyResourceTracker6.registerNodeManager(TestNodeStatusUpdater.java:712)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:597)
at org.apache.hadoop.io.retry.RetryInvocationHandler.invokeMethod(RetryInvocationHandler.java:187)
at org.apache.hadoop.io.retry.RetryInvocationHandler.invoke(RetryInvocationHandler.java:101)
at $Proxy26.registerNodeManager(Unknown Source)
at org.apache.hadoop.yarn.server.nodemanager.NodeStatusUpdaterImpl.registerWithRM(NodeStatusUpdaterImpl.java:212)
at org.apache.hadoop.yarn.server.nodemanager.NodeStatusUpdaterImpl.serviceStart(NodeStatusUpdaterImpl.java:149)
at org.apache.hadoop.yarn.server.nodemanager.TestNodeStatusUpdater$MyNodeStatusUpdater4.serviceStart(TestNodeStatusUpdater.java:369)
at org.apache.hadoop.service.AbstractService.start(AbstractService.java:193)
at org.apache.hadoop.service.CompositeService.serviceStart(CompositeService.java:101)
at org.apache.hadoop.yarn.server.nodemanager.NodeManager.serviceStart(NodeManager.java:213)
at org.apache.hadoop.service.AbstractService.start(AbstractService.java:193)
at org.apache.hadoop.yarn.server.nodemanager.TestNodeStatusUpdater.testNMConnectionToRM(TestNodeStatusUpdater.java:985)
{noformat}

Several cases in this unit tests fail on Windows. (Append error log at the end.)
testInvalidEnvSyntaxDiagnostics fails because the difference between cmd and bash script error handling. If some command fails in the cmd script, cmd will continue execute the the rest of the script command. Error handling needs to be explicitly carried out in the script file. The error code of the last command will be returned as the error code of the whole script. In this test, some error happened in the middle of the cmd script, the test expect an exception and non-zero error code. In the cmd script, the intermediate errors are ignored. The last command "call" succeeded and there is no exception.
testContainerLaunchStdoutAndStderrDiagnostics fails due to wrong cmd commands used by the test.
testContainerEnvVariables and testDelayedKill fail due to a regression from YARN-906.
{noformat}
-------------------------------------------------------------------------------
Test set: org.apache.hadoop.yarn.server.nodemanager.containermanager.launcher.TestContainerLaunch
-------------------------------------------------------------------------------
Tests run: 7, Failures: 4, Errors: 0, Skipped: 0, Time elapsed: 11.526 sec <<< FAILURE!
testInvalidEnvSyntaxDiagnostics(org.apache.hadoop.yarn.server.nodemanager.containermanager.launcher.TestContainerLaunch) Time elapsed: 583 sec <<< FAILURE!
junit.framework.AssertionFailedError: Should catch exception
at junit.framework.Assert.fail(Assert.java:50)
at org.apache.hadoop.yarn.server.nodemanager.containermanager.launcher.TestContainerLaunch.testInvalidEnvSyntaxDiagnostics(TestContainerLaunch.java:269)
...
testContainerLaunchStdoutAndStderrDiagnostics(org.apache.hadoop.yarn.server.nodemanager.containermanager.launcher.TestContainerLaunch) Time elapsed: 561 sec <<< FAILURE!
junit.framework.AssertionFailedError: Should catch exception
at junit.framework.Assert.fail(Assert.java:50)
at org.apache.hadoop.yarn.server.nodemanager.containermanager.launcher.TestContainerLaunch.testContainerLaunchStdoutAndStderrDiagnostics(TestContainerLaunch.java:314)
...
testContainerEnvVariables(org.apache.hadoop.yarn.server.nodemanager.containermanager.launcher.TestContainerLaunch) Time elapsed: 4136 sec <<< FAILURE!
junit.framework.AssertionFailedError: expected:<137> but was:<143>
at junit.framework.Assert.fail(Assert.java:50)
at junit.framework.Assert.failNotEquals(Assert.java:287)
at junit.framework.Assert.assertEquals(Assert.java:67)
at junit.framework.Assert.assertEquals(Assert.java:199)
at junit.framework.Assert.assertEquals(Assert.java:205)
at org.apache.hadoop.yarn.server.nodemanager.containermanager.launcher.TestContainerLaunch.testContainerEnvVariables(TestContainerLaunch.java:500)
...
testDelayedKill(org.apache.hadoop.yarn.server.nodemanager.containermanager.launcher.TestContainerLaunch) Time elapsed: 2744 sec <<< FAILURE!
junit.framework.AssertionFailedError: expected:<137> but was:<143>
at junit.framework.Assert.fail(Assert.java:50)
at junit.framework.Assert.failNotEquals(Assert.java:287)
at junit.framework.Assert.assertEquals(Assert.java:67)
at junit.framework.Assert.assertEquals(Assert.java:199)
at junit.framework.Assert.assertEquals(Assert.java:205)
at org.apache.hadoop.yarn.server.nodemanager.containermanager.launcher.TestContainerLaunch.testDelayedKill(TestContainerLaunch.java:601)
...
{noformat}

YARN-1074.
Major improvement reported by Tassapol Athiapinya and fixed by Xuan Gong (client)Clean up YARN CLI app list to show only running apps.

YARN-1049.
Blocker bug reported by Alejandro Abdelnur and fixed by Alejandro Abdelnur (api)ContainerExistStatus should define a status for preempted containers

With the current behavior is impossible to determine if a container has been preempted or lost due to a NM crash.
Adding a PREEMPTED exit status (-102) will help an AM determine that a container has been preempted.
Note the change of scope from the original summary/description. The original scope proposed API/behavior changes. Because we are passed 2.1.0-beta I'm reducing the scope of this JIRA.

The YARN Fair Scheduler is largely stable now, and should no longer be declared experimental.

YARN-1025.
Major bug reported by Chris Nauroth and fixed by Chris Nauroth (nodemanager , resourcemanager)ResourceManager and NodeManager do not load native libraries on Windows.

ResourceManager and NodeManager do not have the correct setting for java.library.path when launched on Windows. This prevents the processes from loading native code from hadoop.dll. The native code is required for correct functioning on Windows (not optional), so this ultimately can cause failures.

YARN-1008.
Major bug reported by Alejandro Abdelnur and fixed by Alejandro Abdelnur (nodemanager)MiniYARNCluster with multiple nodemanagers, all nodes have same key for allocations

While the NMs are keyed using the NodeId, the allocation is done based on the hostname.
This makes the different nodes indistinguishable to the scheduler.
There should be an option to enabled the host:port instead just port for allocations. The nodes reported to the AM should report the 'key' (host or host:port).

YARN-1006.
Major bug reported by Jian He and fixed by Xuan Gong Nodes list web page on the RM web UI is broken

The nodes web page which list all the connected nodes of the cluster is broken.
1. The page is not showing in correct format/style.
2. If we restart the NM, the node list is not refreshed, but just add the new started NM to the list. The old NMs information still remain.

YARN-1001.
Blocker task reported by Srimanth Gunturi and fixed by Zhijie Shen (api)YARN should provide per application-type and state statistics

In Ambari we plan to show for MR2 the number of applications finished, running, waiting, etc. It would be efficient if YARN could provide per application-type and state aggregated counts.

YARN-994.
Major bug reported by Xuan Gong and fixed by Xuan Gong HeartBeat thread in AMRMClientAsync does not handle runtime exception correctly

YARN-654 performs sanity checks for parameters of public methods in AMRMClient. Those may create runtime exception.
Currently, heartBeat thread in AMRMClientAsync only captures IOException and YarnException, and will not handle Runtime Exception properly.
Possible solution can be: heartbeat thread will catch throwable and notify the callbackhandler thread via existing savedException

YARN-981.
Major bug reported by Xuan Gong and fixed by Jian He YARN/MR2/Job-history /logs link does not have correct content

YARN-966.
Major bug reported by Zhijie Shen and fixed by Zhijie Shen The thread of ContainerLaunch#call will fail without any signal if getLocalizedResources() is called when the container is not at LOCALIZED

In ContainerImpl.getLocalizedResources(), there's:
{code}
assert ContainerState.LOCALIZED == getContainerState(); // TODO: FIXME!!
{code}
ContainerImpl.getLocalizedResources() is called in ContainerLaunch.call(), which is scheduled on a separate thread. If the container is not at LOCALIZED (e.g. it is at KILLING, see YARN-906), an AssertError will be thrown and fails the thread without notifying NM. Therefore, the container cannot receive more events, which are supposed to be sent from ContainerLaunch.call(), and move towards completion.

I have 2 node managers.
* one with 1024 MB memory.(nm1)
* second with 2048 MB memory.(nm2)
I am submitting simple map reduce application with 1 mapper and one reducer with 1024mb each. The steps to reproduce this are
* stop nm2 with 2048MB memory.( This I am doing to make sure that this node's heartbeat doesn't reach RM first).
* now submit application. As soon as it receives first node's (nm1) heartbeat it will try to reserve memory for AM-container (2048MB). However it has only 1024MB of memory.
* now start nm2 with 2048 MB memory.
It hangs forever... Ideally this has two potential issues.
* It should not try to reserve memory on a node manager which is never going to give requested memory. i.e. Current max capability of node manager is 1024MB but 2048MB is reserved on it. But it still does that.
* Say 2048MB is reserved on nm1 but nm2 comes back with 2048MB available memory. In this case if the original request was made without any locality then scheduler should unreserve memory on nm1 and allocate requested 2048MB container on nm2.

YARN-948.
Major bug reported by Omkar Vinit Joshi and fixed by Omkar Vinit Joshi RM should validate the release container list before actually releasing them

YARN-942.
Major bug reported by Sandy Ryza and fixed by Akira AJISAKA (scheduler)In Fair Scheduler documentation, inconsistency on which properties have prefix

locality.threshold.node and locality.threshold.rack should have the yarn.scheduler.fair prefix like the items before them
http://hadoop.apache.org/docs/current/hadoop-yarn/hadoop-yarn-site/FairScheduler.html

YARN-910.
Major improvement reported by Sandy Ryza and fixed by Alejandro Abdelnur (nodemanager)Allow auxiliary services to listen for container starts and completions

Making container start and completion events available to auxiliary services would allow them to be resource-aware. The auxiliary service would be able to notify a co-located service that is opportunistically using free capacity of allocation changes.

YARN-906.
Major sub-task reported by Zhijie Shen and fixed by Zhijie Shen Cancelling ContainerLaunch#call at KILLING causes that the container cannot be completed

See https://builds.apache.org/job/PreCommit-YARN-Build/1435//testReport/org.apache.hadoop.yarn.client.api.impl/TestNMClient/testNMClientNoCleanupOnStop/

If user info is present in the client token then it can be used to do limited authz in the AM.

YARN-696.
Major improvement reported by Trevor Lorimer and fixed by Trevor Lorimer (resourcemanager)Enable multiple states to to be specified in Resource Manager apps REST call

Within the YARN Resource Manager REST API the GET call which returns all Applications can be filtered by a single State query parameter (http://<rm http address:port>/ws/v1/cluster/apps).
There are 8 possible states (New, Submitted, Accepted, Running, Finishing, Finished, Failed, Killed), if no state parameter is specified all states are returned, however if a sub-set of states is required then multiple REST calls are required (max. of 7).
The proposal is to be able to specify multiple states in a single REST call.

YARN-643.
Major bug reported by Jian He and fixed by Xuan Gong WHY appToken is removed both in BaseFinalTransition and AMUnregisteredTransition AND clientToken is removed in FinalTransition and not BaseFinalTransition

The jira is tracking why appToken and clientToAMToken is removed separately, and why they are distributed in different transitions, ideally there may be a common place where these two tokens can be removed at the same time.

YARN-602.
Major bug reported by Xuan Gong and fixed by Kenji Kikushima NodeManager should mandatorily set some Environment variables into every containers that it launches

NodeManager should mandatorily set some Environment variables into every containers that it launches, such as Environment.user, Environment.pwd. If both users and NodeManager set those variables, the value set by NM should be used

YARN-589.
Major improvement reported by Sandy Ryza and fixed by Sandy Ryza (scheduler)Expose a REST API for monitoring the fair scheduler

The fair scheduler should have an HTTP interface that exposes information such as applications per queue, fair shares, demands, current allocations.

PublicLocalizer
1) pending accessed by addResource (part of event handling) and run method (as a part of PublicLocalizer.run() ).
PrivateLocalizer
1) pending accessed by addResource (part of event handling) and findNextResource (i.remove()). Also update method should be fixed. It too is sharing pending list.

YARN-540.
Major sub-task reported by Jian He and fixed by Jian He (resourcemanager)Race condition causing RM to potentially relaunch already unregistered AMs on RM restart

When job succeeds and successfully call finishApplicationMaster, RM shutdown and restart-dispatcher is stopped before it can process REMOVE_APP event. The next time RM comes back, it will reload the existing state files even though the job is succeeded

YARN-502.
Major sub-task reported by Lohit Vijayarenu and fixed by Mayank Bansal RM crash with NPE on NODE_REMOVED event with FairScheduler

While running some test and adding/removing nodes, we see RM crashed with the below exception. We are testing with fair scheduler and running hadoop-2.0.3-alpha
{noformat}
2013-03-22 18:54:27,015 INFO org.apache.hadoop.yarn.server.resourcemanager.rmnode.RMNodeImpl: Deactivating Node YYYY:55680 as it is now LOST
2013-03-22 18:54:27,015 INFO org.apache.hadoop.yarn.server.resourcemanager.rmnode.RMNodeImpl: YYYY:55680 Node Transitioned from UNHEALTHY to LOST
2013-03-22 18:54:27,015 FATAL org.apache.hadoop.yarn.server.resourcemanager.ResourceManager: Error in handling event type NODE_REMOVED to the scheduler
java.lang.NullPointerException
at org.apache.hadoop.yarn.server.resourcemanager.scheduler.fair.FairScheduler.removeNode(FairScheduler.java:619)
at org.apache.hadoop.yarn.server.resourcemanager.scheduler.fair.FairScheduler.handle(FairScheduler.java:856)
at org.apache.hadoop.yarn.server.resourcemanager.scheduler.fair.FairScheduler.handle(FairScheduler.java:98)
at org.apache.hadoop.yarn.server.resourcemanager.ResourceManager$SchedulerEventDispatcher$EventProcessor.run(ResourceManager.java:375)
at java.lang.Thread.run(Thread.java:662)
2013-03-22 18:54:27,016 INFO org.apache.hadoop.yarn.server.resourcemanager.ResourceManager: Exiting, bbye..
2013-03-22 18:54:27,020 INFO org.mortbay.log: Stopped SelectChannelConnector@XXXX:50030
{noformat}

When the ResourceManager kills an application, it leaves the proxy URL redirecting to the original tracking URL for the application even though the ApplicationMaster is no longer there to service it. It should redirect it somewhere more useful, like the RM's web page for the application, where the user can find that the application was killed and links to the AM logs.
In addition, sometimes the AM during teardown from the kill can attempt to unregister and provide an updated tracking URL, but unfortunately the RM has "forgotten" the AM due to the kill and refuses to process the unregistration. Instead it logs:
{noformat}
2013-01-09 17:37:49,671 [IPC Server handler 2 on 8030] ERROR
org.apache.hadoop.yarn.server.resourcemanager.ApplicationMasterService: AppAttemptId doesnt exist in cache appattempt_1357575694478_28614_000001
{noformat}
It should go ahead and process the unregistration to update the tracking URL since the application offered it.

HDFS-5118.
Major new feature reported by Jing Zhao and fixed by Jing Zhao Provide testing support for DFSClient to drop RPC responses

Used for testing when NameNode HA is enabled. Users can use a new configuration property "dfs.client.test.drop.namenode.response.number" to specify the number of responses that DFSClient will drop in each RPC call. This feature can help testing functionalities such as NameNode retry cache.

HDFS-4632.
Major bug reported by Chris Nauroth and fixed by Chuan Liu (test)globStatus using backslash for escaping does not work on Windows

HDFS-4594.
Minor bug reported by Arpit Gupta and fixed by Chris Nauroth (webhdfs)WebHDFS open sets Content-Length header to what is specified by length parameter rather than how much data is actually returned.

HDFS-4329.
Major bug reported by Andy Isaacson and fixed by Cristina L. Abad (hdfs-client)DFSShell issues with directories with spaces in name

HDFS-3245.
Major improvement reported by Todd Lipcon and fixed by Ravi Prakash (namenode)Add metrics and web UI for cluster version summary

Changes since Hadoop 2.0.5-alpha

Fix configs yarn.resourcemanager.resourcemanager.connect.{max.wait.secs|retry_interval.secs} to have a *resourcemanager* only once, make them consistent with other such yarn configs and add entries in yarn-default.xml

YARN-1046.
Major bug reported by Karthik Kambatla and fixed by Karthik Kambatla Disable mem monitoring by default in MiniYARNCluster

Have been running into this frequently inspite of MAPREDUCE-3709 on centos6 machines. However, when I try to run it independently on the machines, I have not been able to reproduce it.
{noformat}
2013-08-07 19:17:35,048 WARN [Container Monitor] monitor.ContainersMonitorImpl (ContainersMonitorImpl.java:run(444)) - Container [pid=16556,containerID=container_1375928243488_0001_01_000001] is running beyond virtual memory limits. Current usage: 132.4 MB of 512 MB physical memory used; 1.2 GB of 1.0 GB virtual memory used. Killing container.
{noformat}

YARN-1045.
Major improvement reported by Siddharth Seth and fixed by Jian He Improve toString implementation for PBImpls

The generic toString implementation that is used in most of the PBImpls {code}getProto().toString().replaceAll("\\n", ", ").replaceAll("\\s+", " ");{code} is rather inefficient - replacing "\n" and "\s" to generate a one line string. Instead, we can use {code}TextFormat.shortDebugString(getProto());{code}.
If we can get this into 2.1.0 - great, otherwise the next release.

YARN-1043.
Major bug reported by Yusaku Sako and fixed by Jian He YARN Queue metrics are getting pushed to neither file nor Ganglia

If an RM admin command is issued using CLI, I get something like following:
13/07/24 17:19:40 INFO client.RMProxy: Connecting to ResourceManager at xxxx.com/1.2.3.4:1234
refreshQueues: Unknown protocol: org.apache.hadoop.yarn.api.ResourceManagerAdministrationProtocolPB

Fix unmanaged AM in non-secure/secure setup post YARN-701 since app-tokens will be used in both scenarios.

YARN-932.
Major bug reported by Sandy Ryza and fixed by Karthik Kambatla TestResourceLocalizationService.testLocalizationInit can fail on JDK7

It looks like this is occurring when testLocalizationInit doesn't run first. Somehow yarn.nodemanager.log-dirs is getting set by one of the other tests (to ${yarn.log.dir}/userlogs), but yarn.log.dir isn't being set.

YARN-927.
Major task reported by Bikas Saha and fixed by Bikas Saha Change ContainerRequest to not have more than 1 container count and remove StoreContainerRequest

The downside is having to use more than 1 container request when requesting more than 1 container at * priority. For most other use cases that have specific locations we anyways need to make multiple container requests. This will also remove unnecessary duplication caused by StoredContainerRequest. It will make the getMatchingRequest() always available and easy to use removeContainerRequest().

YARN-926.
Blocker bug reported by Vinod Kumar Vavilapalli and fixed by Jian He ContainerManagerProtcol APIs should take in requests for multiple containers

AMs typically have to launch multiple containers on a node and the current single container APIs aren't helping. We should have all the APIs take in multiple requests and return multiple responses.
The client libraries could expose both the single and multi-container requests.

YARN-922.
Major sub-task reported by Jian He and fixed by Jian He (resourcemanager)Change FileSystemRMStateStore to use directories

Store each app and its attempts in the same directory so that removing application state is only one operation

Right now there are no defaults in yarn env scripts for resource manager nad node manager and if user wants to override that, then user has to go to documentation and find the variables and change the script.
There is no straight forward way to change it in script. Just updating the variables with defaults.

The childQueues of a ParentQueue are stored in a TreeSet where UsedCapacity defines the sort order. This ensures the queue with least UsedCapacity to receive resources next. On containerAssignment we correctly update the order, but we miss to do so on container completions. This corrupts the TreeSet structure, and under-capacity queues might starve for resources.

In {{NodeHealthScriptRunner}} method, we will set HealthChecker status based on the Shell execution results. Some status are based on the exception thrown during the Shell script execution.
Currently, we will catch a non-ExitCodeException from ShellCommandExecutor, and if Shell has the timeout status set at the same time, we will also set HealthChecker status to timeout.
We have following execution sequence in Shell:
1) In main thread, schedule a delayed timer task that will kill the original process upon timeout.
2) In main thread, open a buffered reader and feed in the process's standard input stream.
3) When timeout happens, the timer task will call {{Process#destroy()}}
to kill the main process.
On Linux, when timeout happened and process killed, the buffered reader will thrown an IOException with message: "Stream closed" in main thread.
On Windows, we don't have the IOException. Only "-1" was returned from the reader that indicates the buffer is finished. As a result, the timeout status is not set on Windows, and {{TestNodeHealthService}} fails on Windows because of this.

YARN-853.
Major bug reported by Devaraj K and fixed by Devaraj K (capacityscheduler)maximum-am-resource-percent doesn't work after refreshQueues command

If we update yarn.scheduler.capacity.maximum-am-resource-percent / yarn.scheduler.capacity.<queue-path>.maximum-am-resource-percent configuration and then do the refreshNodes, it uses the new config value to calculate Max Active Applications and Max Active Application Per User. If we add new node after issuing 'rmadmin -refreshQueues' command, it uses the old maximum-am-resource-percent config value to calculate Max Active Applications and Max Active Application Per User.

The YARN unit test case fails on Windows when comparing expected message with log message in the file. The expected message constructed in the test case has two problems: 1) it uses Path.separator to concatenate path string. Path.separator is always a forward slash, which does not match the backslash used in the log message. 2) On Windows, the default file owner is Administrators group if the file is created by an Administrators user. The test expect the user to be the current user.

YARN-851.
Major bug reported by Omkar Vinit Joshi and fixed by Omkar Vinit Joshi Share NMTokens using NMTokenCache (api-based) instead of memory based approach which is used currently.

It is a follow up ticket for YARN-694. Changing the way NMTokens are shared.

YARN-850.
Major sub-task reported by Jian He and fixed by Jian He Rename getClusterAvailableResources to getAvailableResources in AMRMClients

YARN-848.
Major bug reported by Hitesh Shah and fixed by Hitesh Shah Nodemanager does not register with RM using the fully qualified hostname

If the hostname is misconfigured to not be fully qualified ( i.e. hostname returns foo and hostname -f returns foo.bar.xyz ), the NM ends up registering with the RM using only "foo". This can create problems if DNS cannot resolve the hostname properly.
Furthermore, HDFS uses fully qualified hostnames which can end up affecting locality matches when allocating containers based on block locations.

YARN-846.
Major sub-task reported by Jian He and fixed by Jian He Move pb Impl from yarn-api to yarn-common

YARN-845.
Major sub-task reported by Arpit Gupta and fixed by Mayank Bansal (resourcemanager)RM crash with NPE on NODE_UPDATE

The unit test case fails on Windows due to job id or container id was not printed out as part of the container script. Later, the test tries to read the pid from output of the file, and fails.
Exception in trunk:
{noformat}
Running org.apache.hadoop.yarn.server.nodemanager.containermanager.launcher.TestContainerLaunch
Tests run: 3, Failures: 0, Errors: 1, Skipped: 0, Time elapsed: 9.903 sec <<< FAILURE!
testContainerEnvVariables(org.apache.hadoop.yarn.server.nodemanager.containermanager.launcher.TestContainerLaunch) Time elapsed: 1307 sec <<< ERROR!
java.lang.NullPointerException
at org.apache.hadoop.yarn.server.nodemanager.containermanager.launcher.TestContainerLaunch.testContainerEnvVariables(TestContainerLaunch.java:278)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:597)
at org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:45)
at org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:15)
at org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:42)
at org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:20)
at org.junit.internal.runners.statements.FailOnTimeout$StatementThread.run(FailOnTimeout.java:62)
{noformat}

YARN-837.
Major sub-task reported by Zhijie Shen and fixed by Zhijie Shen ClusterInfo.java doesn't seem to belong to org.apache.hadoop.yarn

YarnVersionAnnotation is not used at all, and the version information can be accessed through YarnVersionInfo instead.

YARN-827.
Critical sub-task reported by Bikas Saha and fixed by Jian He Need to make Resource arithmetic methods accessible

org.apache.hadoop.yarn.server.resourcemanager.resource has stuff like Resources and Calculators that help compare/add resources etc. Without these users will be forced to replicate the logic, potentially incorrectly.

API change. At present this token is getting used on scheduler api AMRMProtocol. Right now name wise it is little confusing as it might be useful for the application to talk to complete yarn system (RM/NM) but that is not the case after YARN-694. NM will have specific NMToken so it is better to name it as AMRMToken.

YARN-821.
Major sub-task reported by Jian He and fixed by Jian He Rename FinishApplicationMasterRequest.setFinishApplicationStatus to setFinalApplicationStatus to be consistent with getter

YARN-820.
Major sub-task reported by Bikas Saha and fixed by Mayank Bansal NodeManager has invalid state transition after error in resource localization

YARN-814.
Major sub-task reported by Hitesh Shah and fixed by Jian He Difficult to diagnose a failed container launch when error due to invalid environment variable

The container's launch script sets up environment variables, symlinks etc.
If there is any failure when setting up the basic context ( before the actual user's process is launched ), nothing is captured by the NM. This makes it impossible to diagnose the reason for the failure.
To reproduce, set an env var where the value contains characters that throw syntax errors in bash.

YARN-806.
Major sub-task reported by Jian He and fixed by Jian He Move ContainerExitStatus from yarn.api to yarn.api.records

YARN-805.
Blocker sub-task reported by Jian He and fixed by Jian He Fix yarn-api javadoc annotations

YARN-803.
Major improvement reported by Alejandro Abdelnur and fixed by Alejandro Abdelnur (resourcemanager , scheduler)factor out scheduler config validation from the ResourceManager to each scheduler implementation

Per discussion in YARN-789 we should factor out from the ResourceManager class the scheduler config validations.

YARN-799.
Major bug reported by Chris Riccomini and fixed by Chris Riccomini (nodemanager)CgroupsLCEResourcesHandler tries to write to cgroup.procs

YARN-795.
Major bug reported by Wei Yan and fixed by Wei Yan (scheduler)Fair scheduler queue metrics should subtract allocated vCores from available vCores

The queue metrics of fair scheduler doesn't subtract allocated vCores from available vCores, causing the available vCores returned is incorrect.
This is happening because {code}QueueMetrics.getAllocateResources(){code} doesn't return the allocated vCores.

YARN-792.
Major sub-task reported by Jian He and fixed by Jian He Move NodeHealthStatus from yarn.api.record to yarn.server.api.record

Per discussion in YARN-689, reposting updated use case:
1. I have a set of services co-existing with a Yarn cluster.
2. These services run out of band from Yarn. They are not started as yarn containers and they don't use Yarn containers for processing.
3. These services use, dynamically, different amounts of CPU and memory based on their load. They manage their CPU and memory requirements independently. In other words, depending on their load, they may require more CPU but not memory or vice-versa.
By using YARN as RM for these services I'm able share and utilize the resources of the cluster appropriately and in a dynamic way. Yarn keeps tab of all the resources.
These services run an AM that reserves resources on their behalf. When this AM gets the requested resources, the services bump up their CPU/memory utilization out of band from Yarn. If the Yarn allocations are released/preempted, the services back off on their resources utilization. By doing this, Yarn and these service correctly share the cluster resources, being Yarn RM the only one that does the overall resource bookkeeping.
The services AM, not to break the lifecycle of containers, start containers in the corresponding NMs. These container processes do basically a sleep forever (i.e. sleep 10000d). They are almost not using any CPU nor memory (less than 1MB). Thus it is reasonable to assume their required CPU and memory utilization is NIL (more on hard enforcement later). Because of this almost NIL utilization of CPU and memory, it is possible to specify, when doing a request, zero as one of the dimensions (CPU or memory).
The current limitation is that the increment is also the minimum.
If we set the memory increment to 1MB. When doing a pure CPU request, we would have to specify 1MB of memory. That would work. However it would allow discretionary memory requests without a desired normalization (increments of 256, 512, etc).
If we set the CPU increment to 1CPU. When doing a pure memory request, we would have to specify 1CPU. CPU amounts a much smaller than memory amounts, and because we don't have fractional CPUs, it would mean that all my pure memory requests will be wasting 1 CPU thus reducing the overall utilization of the cluster.
Finally, on hard enforcement.
* For CPU. Hard enforcement can be done via a cgroup cpu controller. Using an absolute minimum of a few CPU shares (ie 10) in the LinuxContainerExecutor we ensure there is enough CPU cycles to run the sleep process. This absolute minimum would only kick-in if zero is allowed, otherwise will never kick in as the shares for 1 CPU are 1024.
* For Memory. Hard enforcement is currently done by the ProcfsBasedProcessTree.java, using a minimum absolute of 1 or 2 MBs would take care of zero memory resources. And again, this absolute minimum would only kick-in if zero is allowed, otherwise will never kick in as the increment memory is in several MBs if not 1GB.

The vcores-pcores ratio functions differently from the vmem-pmem ratio in the sense that the vcores-pcores ratio has an impact on allocations and the vmem-pmem ratio does not.
If I double my vmem-pmem ratio, the only change that occurs is that my containers, after being scheduled, are less likely to be killed for using too much virtual memory. But if I double my vcore-pcore ratio, my nodes will appear to the ResourceManager to contain double the amount of CPU space, which will affect scheduling decisions.
The lack of consistency will exacerbate the already difficult problem of resource configuration.

YARN-781.
Major sub-task reported by Devaraj Das and fixed by Jian He Expose LOGDIR that containers should use for logging

The LOGDIR is known. We should expose this to the container's environment.

YARN-777.
Major sub-task reported by Jian He and fixed by Jian He Remove unreferenced objects from proto

YARN-773.
Major sub-task reported by Jian He and fixed by Jian He Move YarnRuntimeException from package api.yarn to api.yarn.exceptions

YARN-767.
Major bug reported by Jian He and fixed by Jian He Initialize Application status metrics when QueueMetrics is initialized

Applications: ResourceManager.QueueMetrics.AppsSubmitted, ResourceManager.QueueMetrics.AppsRunning, ResourceManager.QueueMetrics.AppsPending, ResourceManager.QueueMetrics.AppsCompleted, ResourceManager.QueueMetrics.AppsKilled, ResourceManager.QueueMetrics.AppsFailed
For now these metrics are created only when they are needed, we want to make them be seen when QueueMetrics is initialized

YARN-764.
Major bug reported by nemon lou and fixed by nemon lou (resourcemanager)blank Used Resources on Capacity Scheduler page

Even when there are jobs running,used resources is empty on Capacity Scheduler page for leaf queue.(I use google-chrome on windows 7.)
After changing resource.java's toString method by replacing "<>" with "{}",this bug gets fixed.

YARN-763.
Major bug reported by Bikas Saha and fixed by Xuan Gong AMRMClientAsync should stop heartbeating after receiving shutdown from RM

YARN-756.
Major sub-task reported by Jian He and fixed by Jian He Move PreemptionContainer/PremptionContract/PreemptionMessage/StrictPreemptionContract/PreemptionResourceRequest to api.records

YARN-755.
Major sub-task reported by Bikas Saha and fixed by Bikas Saha Rename AllocateResponse.reboot to AllocateResponse.resync

For work preserving rm restart the am's will be resyncing instead of rebooting. rebooting is an action that currently satisfies the resync requirement. Changing the name now so that it continues to make sense in the real resync case.

YARN-753.
Major sub-task reported by Jian He and fixed by Jian He Add individual factory method for api protocol records

A ContainerRequest that includes node-level requests must also include matching rack-level requests for the racks that those nodes are on. When a node is present without its rack, it makes sense for the client to automatically add the node's rack.

YARN-750.
Major sub-task reported by Arun C Murthy and fixed by Arun C Murthy Allow for black-listing resources in YARN API and Impl in CS

YARN-392 and YARN-398 enhance scheduler api to allow for white-lists of resources.
This jira is a companion to allow for black-listing (in CS).

make it clear what you are registering on a {{Service}} by naming the methods {{registerServiceListener()}} & {{unregisterServiceListener()}} respectively.
This only affects a couple of production classes; {{Service.register()}} and is used in some of the lifecycle tests of the YARN-530. There are no tests of {{Service.unregister()}}, which is something that could be corrected.

In one of our clusters, namenode RPC is spending 45% of its time on serving setPermission calls. Further investigation has revealed that most calls are redundantly made on /mapred/logs/<user>/logs. Also mkdirs calls are made before this.

YARN-739.
Major sub-task reported by Siddharth Seth and fixed by Omkar Vinit Joshi NM startContainer should validate the NodeId

The NM validates certain fields from the ContainerToken on a startContainer call. It shoudl also validate the NodeId (which needs to be added to the ContianerToken).

YARN-737.
Major sub-task reported by Jian He and fixed by Jian He Some Exceptions no longer need to be wrapped by YarnException and can be directly thrown out after YARN-142

Currently, at a regular interval, the fair scheduler computes a fair memory share for each queue and application inside it. This fair share is not used for scheduling decisions, but is displayed in the web UI, exposed as a metric, and used for preemption decisions.
With DRF and multi-resource scheduling, assigning a memory share as the fair share metric to every queue no longer makes sense. It's not obvious what the replacement should be, but probably something like fractional fairness within a queue, or distance from an ideal cluster state.

YARN-735.
Major sub-task reported by Jian He and fixed by Jian He Make ApplicationAttemptID, ContainerID, NodeID immutable

The queue shows up as "Invalid Date"
Finish Time shows up as a Long value.

YARN-724.
Major sub-task reported by Jian He and fixed by Jian He Move ProtoBase from api.records to api.records.impl.pb

Simply move ProtoBase to records.impl.pb

YARN-720.
Major sub-task reported by Siddharth Seth and fixed by Zhijie Shen container-log4j.properties should not refer to mapreduce properties

This refers to yarn.app.mapreduce.container.log.dir and yarn.app.mapreduce.container.log.filesize. This should either be moved into the MR codebase. Alternately the parameters should be renamed.

YARN-719.
Major sub-task reported by Vinod Kumar Vavilapalli and fixed by Vinod Kumar Vavilapalli Move RMIdentifier from Container to ContainerTokenIdentifier

This needs to be done for YARN-684 to happen.

YARN-717.
Major sub-task reported by Jian He and fixed by Jian He Copy BuilderUtil methods into token-related records

This is separated from YARN-711,as after changing yarn.api.token from interface to abstract class, eg: ClientTokenPBImpl has to extend two classes: both TokenPBImpl and ClientToken abstract class, which is not allowed in JAVA.
We may remove the ClientToken/ContainerToken/DelegationToken interface and just use the common Token interface

YARN-716.
Major task reported by Siddharth Seth and fixed by Siddharth Seth Make ApplicationID immutable

YARN-715.
Major bug reported by Siddharth Seth and fixed by Vinod Kumar Vavilapalli TestDistributedShell and TestUnmanagedAMLauncher are failing

NMToken will be sent to AM on allocate call if
1) AM doesn't already have NMToken for the underlying NM
2) Key rolled over on RM and AM gets new container on the same NM.
On allocate call RM will send a consolidated list of all required NMTokens.

YARN-711.
Major sub-task reported by Vinod Kumar Vavilapalli and fixed by Jian He Copy BuilderUtil methods into individual records

BuilderUtils is one giant utils class which has all the factory methods needed for creating records. It is painful for users to figure out how to create records. We are better off having the factories in each record, that way users can easily create records.
As a first step, we should just copy all the factory methods into individual classes, deprecate BuilderUtils and then slowly move all code off BuilderUtils.

YARN-708.
Major task reported by Siddharth Seth and fixed by Siddharth Seth Move RecordFactory classes to hadoop-yarn-api, miscellaneous fixes to the interfaces

This is required for additional changes in YARN-528.
Some of the interfaces could use some cleanup as well - they shouldn't be declaring YarnException (Runtime) in their signature.

YARN-706.
Major bug reported by Zhijie Shen and fixed by Zhijie Shen Race Condition in TestFSDownload

See the test failure in YARN-695
https://builds.apache.org/job/PreCommit-YARN-Build/957//testReport/org.apache.hadoop.yarn.util/TestFSDownload/testDownloadPatternJar/

YARN-701.
Blocker sub-task reported by Vinod Kumar Vavilapalli and fixed by Vinod Kumar Vavilapalli ApplicationTokens should be used irrespective of kerberos

- Single code path for secure and non-secure cases is useful for testing, coverage.
- Having this in non-secure mode will help us avoid accidental bugs in AMs DDos'ing and bringing down RM.

YARN-700.
Major bug reported by Ivan Mitic and fixed by Ivan Mitic TestInfoBlock fails on Windows because of line ending missmatch

YARN-695.
Major sub-task reported by Zhijie Shen and fixed by Zhijie Shen masterContainer and status are in ApplicationReportProto but not in ApplicationReport

If masterContainer and status are no longer part of ApplicationReport, they should be removed from proto as well.

YARN-694.
Major bug reported by Omkar Vinit Joshi and fixed by Omkar Vinit Joshi Start using NMTokens to authenticate all communication with NM

AM uses the NMToken to authenticate all the AM-NM communication.
NM will validate NMToken in below manner
* If NMToken is using current or previous master key then the NMToken is valid. In this case it will update its cache with this key corresponding to appId.
* If NMToken is using the master key which is present in NM's cache corresponding to AM's appId then it will be validated based on this.
* If NMToken is invalid then NM will reject AM calls.
Modification for ContainerToken
* At present RPC validates AM-NM communication based on ContainerToken. It will be replaced with NMToken. Also now onwards AM will use NMToken per NM (replacing earlier behavior of ContainerToken per container per NM).
* startContainer in case of Secured environment is using ContainerToken from UGI YARN-617; however after this it will use it from the payload (Container).
* ContainerToken will exist and it will only be used to validate the AM's container start request.

This is part of YARN-613.
As per the updated design, AM will receive per NM, NMToken in following scenarios
* AM is receiving first container on underlying NM.
* AM is receiving container on underlying NM after either NM or RM rebooted.
** After RM reboot, as RM doesn't remember (persist) the information about keys issued per AM per NM, it will reissue tokens in case AM gets new container on underlying NM. However on NM side NM will still retain older token until it receives new token to support long running jobs (in work preserving environment).
** After NM reboot, RM will delete the token information corresponding to that AM for all AMs.
* AM is receiving container on underlying NM after NMToken master key is rolled over on RM side.
In all the cases if AM receives new NMToken then it is suppose to store it for future NM communication until it receives a new one.
AMRMClient should expose these NMToken to client.

YARN-692.
Major bug reported by Omkar Vinit Joshi and fixed by Omkar Vinit Joshi Creating NMToken master key on RM and sharing it with NM as a part of RM-NM heartbeat.

This is related to YARN-613 . Here we will be implementing NMToken generation on RM side and sharing it with NM during RM-NM heartbeat. As a part of this JIRA mater key will only be made available to NM but there will be no validation done until AM-NM communication is fixed.

The DelegationTokenRenewer thread is critical to the RM. When a non-IOException occurs, the thread calls System.exit to prevent the RM from running w/o the thread. It should be exiting only on non-RuntimeExceptions.
The problem is especially bad in 23 because the yarn protobuf layer converts IOExceptions into UndeclaredThrowableExceptions (RuntimeException) which causes the renewer to abort the process. An UnknownHostException takes down the RM...

YARN-688.
Major bug reported by Jian He and fixed by Jian He Containers not cleaned up when NM received SHUTDOWN event from NodeStatusUpdater

Currently, both SHUTDOWN event from nodeStatusUpdater and CleanupContainers event happens to be on the same dispatcher thread, CleanupContainers Event will not be processed until SHUTDOWN event is processed. see similar problem on YARN-495.
On normal NM shutdown, this is not a problem since normal stop happens on shutdownHook thread.

The NodeReport returned by getClusterNodes or given to AMs in heartbeat responses includes both a NodeState (enum) and a NodeHealthStatus (object). As UNHEALTHY is already NodeState, a separate NodeHealthStatus doesn't seem necessary. I propose eliminating NodeHealthStatus#getIsNodeHealthy and moving its two other methods, getHealthReport and getLastHealthReportTime, into NodeReport.

YARN-684.
Major sub-task reported by Vinod Kumar Vavilapalli and fixed by Vinod Kumar Vavilapalli ContainerManager.startContainer needs to only have ContainerTokenIdentifier instead of the whole Container

The NM only needs the token, the whole Container is unnecessary.

YARN-663.
Major sub-task reported by Xuan Gong and fixed by Xuan Gong Change ResourceTracker API and LocalizationProtocol API to throw YarnRemoteException and IOException

YARN-661.
Major bug reported by Jason Lowe and fixed by Omkar Vinit Joshi (nodemanager)NM fails to cleanup local directories for users

YARN-71 added deletion of local directories on startup, but in practice it fails to delete the directories because of permission problems. The top-level usercache directory is owned by the user but is in a directory that is not writable by the user. Therefore the deletion of the user's usercache directory, as the user, fails due to lack of permissions.

Issues are found in the doc page for Fair Scheduler http://hadoop.apache.org/docs/current/hadoop-yarn/hadoop-yarn-site/FairScheduler.html:
1.In the section “Configuration”, It contains two properties named “yarn.scheduler.fair.minimum-allocation-mb”, the second one should be “yarn.scheduler.fair.maximum-allocation-mb”
2.In the section “Allocation file format”, the document tells “ The format contains three types of elements”, but it lists four types of elements following that.

YARN-645.
Major bug reported by Jian He and fixed by Jian He Move RMDelegationTokenSecretManager from yarn-server-common to yarn-server-resourcemanager

RMDelegationTokenSecretManager is specific to resource manager, should not belong to server-common

YARN-642.
Major bug reported by Sandy Ryza and fixed by Sandy Ryza (api , resourcemanager)Fix up /nodes REST API to have 1 param and be consistent with the Java API

The code behind the /nodes RM REST API is unnecessarily muddled, logs the same misspelled INFO message repeatedly, and does not return unhealthy nodes, even when asked.

YARN-639.
Major bug reported by Zhijie Shen and fixed by Zhijie Shen (applications/distributed-shell)Make AM of Distributed Shell Use NMClient

YARN-422 adds NMClient. AM of Distributed Shell should use it instead of using ContainerManager directly.

YARN-638.
Major sub-task reported by Jian He and fixed by Jian He (resourcemanager)Restore RMDelegationTokens after RM Restart

This is missed in YARN-581. After RM restart, RMDelegationTokens need to be added both in DelegationTokenRenewer (addressed in YARN-581), and delegationTokenSecretManager

YARN-637.
Major bug reported by Karthik Kambatla and fixed by Karthik Kambatla (scheduler)FS: maxAssign is not honored

maxAssign limits the number of containers that can be assigned in a single heartbeat. Currently, FS doesn't keep track of number of assigned containers to check this.

YARN-635.
Major sub-task reported by Xuan Gong and fixed by Siddharth Seth Rename YarnRemoteException to YarnException

YARN-634.
Major sub-task reported by Siddharth Seth and fixed by Siddharth Seth Make YarnRemoteException not backed by PB and introduce a SerializedException

LocalizationProtocol sends an exception over the wire. This currently uses YarnRemoteException. Post YARN-627, this needs to be changed and a new serialized exception is required.

YARN-633.
Major sub-task reported by Xuan Gong and fixed by Xuan Gong Change RMAdminProtocol api to throw IOException and YarnRemoteException

YARN-632.
Major sub-task reported by Xuan Gong and fixed by Xuan Gong Change ContainerManager api to throw IOException and YarnRemoteException

YARN-631.
Major sub-task reported by Xuan Gong and fixed by Xuan Gong Change ClientRMProtocol api to throw IOException and YarnRemoteException

YARN-630.
Major sub-task reported by Xuan Gong and fixed by Xuan Gong Change AMRMProtocol api to throw IOException and YarnRemoteException

YARN-629.
Major sub-task reported by Xuan Gong and fixed by Xuan Gong Make YarnRemoteException not be rooted at IOException

After HADOOP-9343, it should be possible for YarnException to not be rooted at IOException

Without security, it is impossible to completely avoid AMs faking resources. We can at the least make it as difficult as possible by using the same container tokens and the RM-NM shared key mechanism over unauthenticated RM-NM channel.
In the minimum, this will avoid accidental bugs in AMs in unsecure mode.

YARN-615.
Major sub-task reported by Vinod Kumar Vavilapalli and fixed by Vinod Kumar Vavilapalli ContainerLaunchContext.containerTokens should simply be called tokens

ContainerToken is the name of the specific token that AMs use to launch containers on NMs, so we should rename CLC.containerTokens to be simply tokens.

YARN-613.
Major sub-task reported by Bikas Saha and fixed by Omkar Vinit Joshi Create NM proxy per NM instead of per container

Currently a new NM proxy has to be created per container since the secure authentication is using a containertoken from the container.

YARN-610.
Blocker sub-task reported by Siddharth Seth and fixed by Omkar Vinit Joshi ClientToken (ClientToAMToken) should not be set in the environment

Similar to YARN-579, this can be set via ContainerTokens

YARN-605.
Major bug reported by Hitesh Shah and fixed by Hitesh Shah Failing unit test in TestNMWebServices when using git for source control

YARN-600.
Major improvement reported by Sandy Ryza and fixed by Sandy Ryza (resourcemanager , scheduler)Hook up cgroups CPU settings to the number of virtual cores allocated

YARN-3 introduced CPU isolation and monitoring through cgroups. YARN-2 and introduced CPU scheduling in the capacity scheduler, and YARN-326 will introduce it in the fair scheduler. The number of virtual cores allocated to a container should be used to weight the number of cgroups CPU shares given to it.

YARN-599.
Major bug reported by Zhijie Shen and fixed by Zhijie Shen Refactoring submitApplication in ClientRMService and RMAppManager

Currently, ClientRMService#submitApplication call RMAppManager#handle, and consequently call RMAppMangager#submitApplication directly, though the code looks like scheduling an APP_SUBMIT event.
In addition, the validation code before creating an RMApp instance is not well organized. Ideally, the dynamic validation, which depends on the RM's configuration, should be put in RMAppMangager#submitApplication. RMAppMangager#submitApplication is called by ClientRMService#submitApplication and RMAppMangager#recover. Since the configuration may be changed after RM restarts, the validation needs to be done again even in recovery mode. Therefore, resource request validation, which based on min/max resource limits, should be moved from ClientRMService#submitApplication to RMAppMangager#submitApplication. On the other hand, the static validation, which is independent of the RM's configuration should be put in ClientRMService#submitApplication, because it is only need to be done once during the first submission.
Furthermore, try-catch flow in RMAppMangager#submitApplication has a flaw. RMAppMangager#submitApplication has a flaw is not synchronized. If two application submissions with the same application ID enter the function, and one progresses to the completion of RMApp instantiation, and the other progresses the completion of putting the RMApp instance into rmContext, the slower submission will cause an exception due to the duplicate application ID. However, the exception will cause the RMApp instance already in rmContext (belongs to the faster submission) being rejected with the current code flow.

QueueMetrics includes allocatedMB, availableMB, pendingMB, reservedMB. It should have equivalents for CPU.

YARN-597.
Major bug reported by Ivan Mitic and fixed by Ivan Mitic TestFSDownload fails on Windows because of dependencies on tar/gzip/jar tools

{{testDownloadArchive}}, {{testDownloadPatternJar}} and {{testDownloadArchiveZip}} fail with the similar Shell ExitCodeException:
{code}
testDownloadArchiveZip(org.apache.hadoop.yarn.util.TestFSDownload) Time elapsed: 480 sec <<< ERROR!
org.apache.hadoop.util.Shell$ExitCodeException: bash: line 0: cd: /D:/svn/t/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/target/TestFSDownload: No such file or directory
gzip: 1: No such file or directory
at org.apache.hadoop.util.Shell.runCommand(Shell.java:377)
at org.apache.hadoop.util.Shell.run(Shell.java:292)
at org.apache.hadoop.util.Shell$ShellCommandExecutor.execute(Shell.java:497)
at org.apache.hadoop.yarn.util.TestFSDownload.createZipFile(TestFSDownload.java:225)
at org.apache.hadoop.yarn.util.TestFSDownload.testDownloadArchiveZip(TestFSDownload.java:503)
{code}

YARN-595.
Major sub-task reported by Sandy Ryza and fixed by Sandy Ryza (scheduler)Refactor fair scheduler to use common Resources

resourcemanager.fair and resourcemanager.resources have two copies of basically the same code for operations on Resource objects

YARN-594.
Major bug reported by Jian He and fixed by Jian He Update test and add comments in YARN-534

This jira is simply to add some comments in the patch YARN-534 and update the test case

YARN-593.
Major bug reported by Chris Nauroth and fixed by Chris Nauroth (nodemanager)container launch on Windows does not correctly populate classpath with new process's environment variables and localized resources

On Windows, we must bundle the classpath of a launched container in an intermediate jar with a manifest. Currently, this logic incorrectly uses the nodemanager process's environment variables for substitution. Instead, it needs to use the new environment for the launched process. Also, the bundled classpath is missing some localized resources for directories, due to a quirk in the way {{File#toURI}} decides whether or not to append a trailing '/'.

YARN-591.
Major sub-task reported by Vinod Kumar Vavilapalli and fixed by Vinod Kumar Vavilapalli RM recovery related records do not belong to the API

We need to move out AppliationStateData and ApplicationAttemptStateData into resourcemanager module. They are not part of the public API..

YARN-590.
Major improvement reported by Vinod Kumar Vavilapalli and fixed by Mayank Bansal Add an optional mesage to RegisterNodeManagerResponse as to why NM is being asked to resync or shutdown

We should log such message in NM itself. Helps in debugging issues on NM directly instead of distributed debugging between RM and NM when such an action is received from RM.

YARN-585.
Major bug reported by Zhijie Shen and fixed by Zhijie Shen TestFairScheduler#testNotAllowSubmitApplication is broken due to YARN-514

TestFairScheduler#testNotAllowSubmitApplication is broken due to YARN-514. See the discussions in YARN-514.

YARN-583.
Major sub-task reported by Omkar Vinit Joshi and fixed by Omkar Vinit Joshi Application cache files should be localized under local-dir/usercache/userid/appcache/appid/filecache

Currently application cache files are getting localized under local-dir/usercache/userid/appcache/appid/. however they should be localized under filecache sub directory.

YARN-582.
Major sub-task reported by Bikas Saha and fixed by Jian He (resourcemanager)Restore appToken and clientToken for app attempt after RM restart

These need to be saved and restored on a per app attempt basis. This is required only when work preserving restart is implemented for secure clusters. In non-preserving restart app attempts are killed and so this does not matter.

YARN-581.
Major sub-task reported by Bikas Saha and fixed by Jian He (resourcemanager)Test and verify that app delegation tokens are added to tokenRenewer after RM restart

The code already saves the delegation tokens in AppSubmissionContext. Upon restart the AppSubmissionContext is used to submit the application again and so restores the delegation tokens. This jira tracks testing and verifying this functionality in a secure setup.

YARN-579.
Major sub-task reported by Vinod Kumar Vavilapalli and fixed by Vinod Kumar Vavilapalli Make ApplicationToken part of Container's token list to help RM-restart

Container is already persisted for helping RM restart. Instead of explicitly setting ApplicationToken in AM's env, if we change it to be in Container, we can avoid env and can also help restart.

YARN-578.
Major sub-task reported by Vinod Kumar Vavilapalli and fixed by Omkar Vinit Joshi (nodemanager)NodeManager should use SecureIOUtils for serving and aggregating logs

Log servlets for serving logs and the ShuffleService for serving intermediate outputs both should use SecureIOUtils for avoiding symlink attacks.

YARN-577.
Major sub-task reported by Hitesh Shah and fixed by Hitesh Shah ApplicationReport does not provide progress value of application

An application sends its progress % to the RM via AllocateRequest. This should be able to be retrieved by a client via the ApplicationReport.

YARN-576.
Major bug reported by Hitesh Shah and fixed by Kenji Kikushima RM should not allow registrations from NMs that do not satisfy minimum scheduler allocations

If the minimum resource allocation configured for the RM scheduler is 1 GB, the RM should drop all NMs that register with a total capacity of less than 1 GB.

YARN-571.
Major sub-task reported by Hitesh Shah and fixed by Omkar Vinit Joshi User should not be part of ContainerLaunchContext

Today, a user is expected to set the user name in the CLC when either submitting an application or launching a container from the AM. This does not make sense as the user can/has been identified by the RM as part of the RPC layer.
Solution would be to move the user information into either the Container object or directly into the ContainerToken which can then be used by the NM to launch the container. This user information would set into the container by the RM.

YARN-569.
Major sub-task reported by Carlo Curino and fixed by Carlo Curino (capacityscheduler)CapacityScheduler: support for preemption (using a capacity monitor)

There is a tension between the fast-pace reactive role of the CapacityScheduler, which needs to respond quickly to
applications resource requests, and node updates, and the more introspective, time-based considerations
needed to observe and correct for capacity balance. To this purpose we opted instead of hacking the delicate
mechanisms of the CapacityScheduler directly to add support for preemption by means of a "Capacity Monitor",
which can be run optionally as a separate service (much like the NMLivelinessMonitor).
The capacity monitor (similarly to equivalent functionalities in the fairness scheduler) operates running on intervals
(e.g., every 3 seconds), observe the state of the assignment of resources to queues from the capacity scheduler,
performs off-line computation to determine if preemption is needed, and how best to "edit" the current schedule to
improve capacity, and generates events that produce four possible actions:
# Container de-reservations
# Resource-based preemptions
# Container-based preemptions
# Container killing
The actions listed above are progressively more costly, and it is up to the policy to use them as desired to achieve the rebalancing goals.
Note that due to the "lag" in the effect of these actions the policy should operate at the macroscopic level (e.g., preempt tens of containers
from a queue) and not trying to tightly and consistently micromanage container allocations.
------------- Preemption policy (ProportionalCapacityPreemptionPolicy): -------------
Preemption policies are by design pluggable, in the following we present an initial policy (ProportionalCapacityPreemptionPolicy) we have been experimenting with. The ProportionalCapacityPreemptionPolicy behaves as follows:
# it gathers from the scheduler the state of the queues, in particular, their current capacity, guaranteed capacity and pending requests (*)
# if there are pending requests from queues that are under capacity it computes a new ideal balanced state (**)
# it computes the set of preemptions needed to repair the current schedule and achieve capacity balance (accounting for natural completion rates, and
respecting bounds on the amount of preemption we allow for each round)
# it selects which applications to preempt from each over-capacity queue (the last one in the FIFO order)
# it remove reservations from the most recently assigned app until the amount of resource to reclaim is obtained, or until no more reservations exits
# (if not enough) it issues preemptions for containers from the same applications (reverse chronological order, last assigned container first) again until necessary or until no containers except the AM container are left,
# (if not enough) it moves onto unreserve and preempt from the next application.
# containers that have been asked to preempt are tracked across executions. If a containers is among the one to be preempted for more than a certain time, the container is moved in a the list of containers to be forcibly killed.
Notes:
(*) at the moment, in order to avoid double-counting of the requests, we only look at the "ANY" part of pending resource requests, which means we might not preempt on behalf of AMs that ask only for specific locations but not any.
(**) The ideal balance state is one in which each queue has at least its guaranteed capacity, and the spare capacity is distributed among queues (that wants some) as a weighted fair share. Where the weighting is based on the guaranteed capacity of a queue, and the function runs to a fix point.
Tunables of the ProportionalCapacityPreemptionPolicy:
# observe-only mode (i.e., log the actions it would take, but behave as read-only)
# how frequently to run the policy
# how long to wait between preemption and kill of a container
# which fraction of the containers I would like to obtain should I preempt (has to do with the natural rate at which containers are returned)
# deadzone size, i.e., what % of over-capacity should I ignore (if we are off perfect balance by some small % we ignore it)
# overall amount of preemption we can afford for each run of the policy (in terms of total cluster capacity)
In our current experiments this set of tunables seem to be a good start to shape the preemption action properly. More sophisticated preemption policies could take into account different type of applications running, job priorities, cost of preemption, integral of capacity imbalance. This is very much a control-theory kind of problem, and some of the lessons on designing and tuning controllers are likely to apply.
Generality:
The monitor-based scheduler edit, and the preemption mechanisms we introduced here are designed to be more general than enforcing capacity/fairness, in fact, we are considering other monitors that leverage the same idea of "schedule edits" to target different global properties (e.g., allocate enough resources to guarantee deadlines for important jobs, or data-locality optimizations, IO-balancing among nodes, etc...).
Note that by default the preemption policy we describe is disabled in the patch.
Depends on YARN-45 and YARN-567, is related to YARN-568

YARN-568.
Major improvement reported by Carlo Curino and fixed by Carlo Curino (scheduler)FairScheduler: support for work-preserving preemption

In the attached patch, we modified the FairScheduler to substitute its preemption-by-killling with a work-preserving version of preemption (followed by killing if the AMs do not respond quickly enough). This should allows to run preemption checking more often, but kill less often (proper tuning to be investigated). Depends on YARN-567 and YARN-45, is related to YARN-569.

YARN-567.
Major sub-task reported by Carlo Curino and fixed by Carlo Curino (resourcemanager)RM changes to support preemption for FairScheduler and CapacityScheduler

A common tradeoff in scheduling jobs is between keeping the cluster busy and enforcing capacity/fairness properties. FairScheduler and CapacityScheduler takes opposite stance on how to achieve this.
The FairScheduler, leverages task-killing to quickly reclaim resources from currently running jobs and redistributing them among new jobs, thus keeping the cluster busy but waste useful work. The CapacityScheduler is typically tuned
to limit the portion of the cluster used by each queue so that the likelihood of violating capacity is low, thus never wasting work, but risking to keep the cluster underutilized or have jobs waiting to obtain their rightful capacity.
By introducing the notion of a work-preserving preemption we can remove this tradeoff. This requires a protocol for preemption (YARN-45), and ApplicationMasters that can answer to preemption efficiently (e.g., by saving their intermediate state, this will be posted for MapReduce in a separate JIRA soon), together with a scheduler that can issues preemption requests (discussed in separate JIRAs YARN-568 and YARN-569).
The changes we track with this JIRA are common to FairScheduler and CapacityScheduler, and are mostly propagation of preemption decisions through the ApplicationMastersService.

This field is needed to distinguish different types of applications (app master implementations). For example, we may run applications of type XYZ in a cluster alongside MR and would like to filter applications by type.

YARN-562.
Major sub-task reported by Jian He and fixed by Jian He (resourcemanager)NM should reject containers allocated by previous RM

Its possible that after RM shutdown, before AM goes down,AM still call startContainer on NM with containers allocated by previous RM. When RM comes back, NM doesn't know whether this container launch request comes from previous RM or the current RM. we should reject containers allocated by previous RM

YARN-561.
Major sub-task reported by Hitesh Shah and fixed by Xuan Gong Nodemanager should set some key information into the environment of every container that it launches.

Information such as containerId, nodemanager hostname, nodemanager port is not set in the environment when any container is launched.
For an AM, the RM does all of this for it but for a container launched by an application, all of the above need to be set by the ApplicationMaster.
At the minimum, container id would be a useful piece of information. If the container wishes to talk to its local NM, the nodemanager related information would also come in handy.

YARN-557.
Major bug reported by Chris Nauroth and fixed by Chris Nauroth (applications)TestUnmanagedAMLauncher fails on Windows

{{TestUnmanagedAMLauncher}} fails on Windows due to attempting to run a Unix-specific command in distributed shell and use of a Unix-specific environment variable to determine username for the {{ContainerLaunchContext}}.

Right now, we're doing multiple steps to create a relevant ApplicationSubmissionContext for a pre-received GetNewApplicationResponse.
{code}
GetNewApplicationResponse newApp = yarnClient.getNewApplication();
ApplicationId appId = newApp.getApplicationId();
ApplicationSubmissionContext appContext = Records.newRecord(ApplicationSubmissionContext.class);
appContext.setApplicationId(appId);
{code}
A simplified way may be to have the GetNewApplicationResponse itself provide a helper method that builds a usable ApplicationSubmissionContext for us. Something like:
{code}
GetNewApplicationResponse newApp = yarnClient.getNewApplication();
ApplicationSubmissionContext appContext = newApp.generateApplicationSubmissionContext();
{code}
[The above method can also take an arg for the container launch spec, or perhaps pre-load defaults like min-resource, etc. in the returned object, aside of just associating the application ID automatically.]

YARN-549.
Major sub-task reported by Zhijie Shen and fixed by Zhijie Shen YarnClient.submitApplication should wait for application to be accepted by the RM

Currently, when submitting an application, storeApplication will be called for recovery. However, it is a blocking API, and is likely to block concurrent application submissions. Therefore, it is good to make application submission asynchronous, and postpone storeApplication. YarnClient needs to change to wait for the whole operation to complete so that clients can be notified after the application is really submitted. YarnClient needs to wait for application to reach SUBMITTED state or beyond.

YARN-548.
Major sub-task reported by Vadim Bondarev and fixed by Vadim Bondarev Add tests for YarnUncaughtExceptionHandler

YARN-547.
Major sub-task reported by Omkar Vinit Joshi and fixed by Omkar Vinit Joshi Race condition in Public / Private Localizer may result into resource getting downloaded again

Public Localizer :
At present when multiple containers try to request a localized resource
* If the resource is not present then first it is created and Resource Localization starts ( LocalizedResource is in DOWNLOADING state)
* Now if in this state multiple ResourceRequestEvents arrive then ResourceLocalizationEvents are sent for all of them.
Most of the times it is not resulting into a duplicate resource download but there is a race condition present there. Inside ResourceLocalization (for public download) all the requests are added to local attempts map. If a new request comes in then first it is checked in this map before a new download starts for the same. For the current download the request will be there in the map. Now if a same resource request comes in then it will rejected (i.e. resource is getting downloaded already). However if the current download completes then the request will be removed from this local map. Now after this removal if the LocalizerRequestEvent comes in then as it is not present in local map the resource will be downloaded again.
PrivateLocalizer :
Here a different but similar race condition is present.
* Here inside findNextResource method call; each LocalizerRunner tries to grab a lock on LocalizerResource. If the lock is not acquired then it will keep trying until the resource state changes to LOCALIZED. This lock will be released by the LocalizerRunner when download completes.
* Now if another ContainerLocalizer tries to grab the lock on a resource before LocalizedResource state changes to LOCALIZED then resource will be downloaded again.
At both the places the root cause of this is that all the threads try to acquire the lock on resource however current state of the LocalizedResource is not taken into consideration.

YARN-542.
Major bug reported by Vinod Kumar Vavilapalli and fixed by Zhijie Shen Change the default global AM max-attempts value to be not one

Today, the global AM max-attempts is set to 1 which is a bad choice. AM max-attempts accounts for both AM level failures as well as container crashes due to localization issue, lost nodes etc. To account for AM crashes due to problems that are not caused by user code, mainly lost nodes, we want to give AMs some retires.
I propose we change it to atleast two. Can change it to 4 to match other retry-configs.

YARN-541.
Blocker bug reported by Krishna Kishore Bonagiri and fixed by Bikas Saha (resourcemanager)getAllocatedContainers() is not returning all the allocated containers

I am running an application that was written and working well with the hadoop-2.0.0-alpha but when I am running the same against 2.0.3-alpha, the getAllocatedContainers() method called on AMResponse is not returning all the containers allocated sometimes. For example, I request for 10 containers and this method gives me only 9 containers sometimes, and when I looked at the log of Resource Manager, the 10th container is also allocated. It happens only sometimes randomly and works fine all other times. If I send one more request for the remaining container to RM after it failed to give them the first time(and before releasing already acquired ones), it could allocate that container. I am running only one application at a time, but 1000s of them one after another.
My main worry is, even though the RM's log is saying that all 10 requested containers are allocated, the getAllocatedContainers() method is not returning me all of them, it returned only 9 surprisingly. I never saw this kind of issue in the previous version, i.e. hadoop-2.0.0-alpha.
Thanks,
Kishore

If resource localization fails then resource remains in memory and is
1) Either cleaned up when next time cache cleanup runs and there is space crunch. (If sufficient space in cache is available then it will remain in memory).
2) reused if LocalizationRequest comes again for the same resource.
I think when resource localization fails then that event should be sent to LocalResourceTracker which will then remove it from its cache.

When I run the job history server locally, every page load takes in the 10s of seconds. I profiled the process and discovered that all the extra time was spent inside YarnConfiguration#getRMWebAppURL, trying to resolve 0.0.0.0 to a hostname. When I changed my yarn.resourcemanager.address to localhost, the page load times decreased drastically.
There's no that we need to perform this resolution on every page load.

YARN-536.
Major sub-task reported by Xuan Gong and fixed by Xuan Gong Remove ContainerStatus, ContainerState from Container api interface as they will not be called by the container object

Remove containerstate, containerStatus from container interface. They will not be called by container object

YARN-534.
Major sub-task reported by Jian He and fixed by Jian He (resourcemanager)AM max attempts is not checked when RM restart and try to recover attempts

Currently,AM max attempts is only checked if the current attempt fails and check to see whether to create new attempt. If the RM restarts before the max-attempt fails, it'll not clean the state store, when RM comes back, it will retry attempt again.

YARN-532.
Major bug reported by Siddharth Seth and fixed by Siddharth Seth RMAdminProtocolPBClientImpl should implement Closeable

Required for RPC.stopProxy to work. Already done in most of the other protocols. (MAPREDUCE-5117 addressing the one other protocol missing this)

# Extend the YARN {{Service}} interface as discussed in YARN-117
# Implement the changes in {{AbstractService}} and {{FilterService}}.
# Migrate all services in yarn-common to the more robust service model, test.

the config yarn.scheduler.capacity.node-locality-delay doesn't change when you change the value in capacity_scheduler.xml and then run yarn rmadmin -refreshQueues.

YARN-523.
Major sub-task reported by Vinod Kumar Vavilapalli and fixed by Jian He Container localization failures aren't reported from NM to RM

This is mainly a pain on crashing AMs, but once we fix this, containers also can benefit - same fix for both.

YARN-521.
Major sub-task reported by Sandy Ryza and fixed by Sandy Ryza (api)Augment AM - RM client module to be able to request containers only at specific locations

When YARN-392 and YARN-398 are completed, it would be good for AMRMClient to offer an easy way to access their functionality

YARN-518.
Major improvement reported by Dapeng Sun and fixed by Sandy Ryza (documentation)Fair Scheduler's document link could be added to the hadoop 2.x main doc page

Currently the doc page for Fair Scheduler looks good and it’s here, http://hadoop.apache.org/docs/current/hadoop-yarn/hadoop-yarn-site/FairScheduler.html.
It would be better to add the document link to the YARN section in the Hadoop 2.x main doc page, so that users can easily find the doc to experimentally try Fair Scheduler as Capacity Scheduler.

YARN-515.
Blocker bug reported by Robert Joseph Evans and fixed by Robert Joseph Evans Node Manager not getting the master key

On branch-2 the latest version I see the following on a secure cluster.
{noformat}
2013-03-28 19:21:06,243 [main] INFO org.apache.hadoop.yarn.server.nodemanager.NodeStatusUpdaterImpl: Security enabled - updating secret keys now
2013-03-28 19:21:06,243 [main] INFO org.apache.hadoop.yarn.server.nodemanager.NodeStatusUpdaterImpl: Registered with ResourceManager as RM:PORT with total resource of <me
mory:12288, vCores:16>
2013-03-28 19:21:06,244 [main] INFO org.apache.hadoop.yarn.service.AbstractService: Service:org.apache.hadoop.yarn.server.nodemanager.NodeStatusUpdaterImpl is started.
2013-03-28 19:21:06,245 [main] INFO org.apache.hadoop.yarn.service.AbstractService: Service:org.apache.hadoop.yarn.server.nodemanager.NodeManager is started.
2013-03-28 19:21:07,257 [Node Status Updater] ERROR org.apache.hadoop.yarn.server.nodemanager.NodeStatusUpdaterImpl: Caught exception in status-updater
java.lang.NullPointerException
at org.apache.hadoop.yarn.server.security.BaseContainerTokenSecretManager.getCurrentKey(BaseContainerTokenSecretManager.java:121)
at org.apache.hadoop.yarn.server.nodemanager.NodeStatusUpdaterImpl$1.run(NodeStatusUpdaterImpl.java:407)
{noformat}
The Null pointer exception just keeps repeating and all of the nodes end up being lost. It looks like it never gets the secret key when it registers.

YARN-514.
Major sub-task reported by Bikas Saha and fixed by Zhijie Shen (resourcemanager)Delayed store operations should not result in RM unavailability for app submission

Currently, app submission is the only store operation performed synchronously because the app must be stored before the request returns with success. This makes the RM susceptible to blocking all client threads on slow store operations, resulting in RM being perceived as unavailable by clients.

YARN-513.
Major sub-task reported by Bikas Saha and fixed by Jian He (resourcemanager)Create common proxy client for communicating with RM

When the RM is restarting, the NM, AM and Clients should wait for some time for the RM to come back up.

YARN-512.
Minor bug reported by Jason Lowe and fixed by Maysam Yabandeh (nodemanager)Log aggregation root directory check is more expensive than it needs to be

The log aggregation root directory check first does an {{exists}} call followed by a {{getFileStatus}} call. That effectively stats the file twice. It should just use {{getFileStatus}} and catch {{FileNotFoundException}} to handle the non-existent case.
In addition we may consider caching the presence of the directory rather than checking it each time a node aggregates logs for an application.

When FairScheduler#reinitialize is called, some of the scheduler-wide configs are refreshed and others aren't. They should all be refreshed.
Ones that are refreshed: userAsDefaultQueue, nodeLocalityThreshold, rackLocalityThreshold, preemptionEnabled
Ones that aren't: minimumAllocation, maximumAllocation, assignMultiple, maxAssign

YARN-495.
Major bug reported by Jian He and fixed by Jian He Change NM behavior of reboot to resync

When a reboot command is sent from RM, the node manager doesn't clean up the containers while its stopping.

YARN-493.
Major bug reported by Chris Nauroth and fixed by Chris Nauroth (nodemanager)NodeManager job control logic flaws on Windows

Both product and test code contain some platform-specific assumptions, such as availability of bash for executing a command in a container and signals to check existence of a process and terminate it.

YARN-491.
Major bug reported by Chris Nauroth and fixed by Chris Nauroth (nodemanager)TestContainerLogsPage fails on Windows

{{TestContainerLogsPage}} contains some code for initializing a log directory that doesn't work correctly on Windows.

YARN-490.
Major bug reported by Chris Nauroth and fixed by Chris Nauroth (applications/distributed-shell)TestDistributedShell fails on Windows

There are a few platform-specific assumption in distributed shell (both main code and test code) that prevent it from working correctly on Windows.

YARN-488.
Major bug reported by Chris Nauroth and fixed by Chris Nauroth (nodemanager)TestContainerManagerSecurity fails on Windows

These tests are failing to launch containers correctly when running on Windows.

YARN-487.
Major bug reported by Chris Nauroth and fixed by Chris Nauroth (nodemanager)TestDiskFailures fails on Windows due to path mishandling

{{TestDiskFailures#testDirFailuresOnStartup}} fails due to insertion of an extra leading '/' on the path within {{LocalDirsHandlerService}} when running on Windows. The test assertions also fail to account for the fact that {{Path}} normalizes '\' to '/'.

YARN-486.
Major sub-task reported by Bikas Saha and fixed by Xuan Gong Change startContainer NM API to accept Container as a parameter and make ContainerLaunchContext user land

Currently, id, resource request etc need to be copied over from Container to ContainerLaunchContext. This can be brittle. Also it leads to duplication of information (such as Resource from CLC and Resource from Container and Container.tokens). Sending Container directly to startContainer solves these problems. It also makes CLC clean by only having stuff in it that it set by the client/AM.

YARN-485.
Major bug reported by Karthik Kambatla and fixed by Karthik Kambatla TestProcfsProcessTree#testProcessTree() doesn't wait long enough for the process to die

TestProcfsProcessTree#testProcessTree fails occasionally with the following stack trace
{noformat}
Stack Trace:
junit.framework.AssertionFailedError: expected:<false> but was:<true>
at org.apache.hadoop.util.TestProcfsBasedProcessTree.testProcessTree(TestProcfsBasedProcessTree.java)
{noformat}
kill -9 is executed asynchronously, the signal is delivered when the process comes out of the kernel (sys call). Checking if the process died immediately after can fail at times.

FS allows setting {{SchedulingMode}} for leaf queues. Extending this to non-leaf queues allows using different kinds of fairness: e.g., root can have three child queues - fair-mem, drf-cpu-mem, drf-cpu-disk-mem taking different number of resources into account. In turn, this allows users to decide on the scheduling latency vs sophistication of the scheduling mode.

Hey Guys,
I noticed that the ApplicationCLI is just randomly not printing some of the values in the ApplicationReport. I've added the getHost and getRpcPort. These are useful for me, since I want to make an RPC call to the AM (not the tracker call).
Thanks!
Chris

YARN-479.
Major bug reported by Hitesh Shah and fixed by Jian He NM retry behavior for connection to RM should be similar for lost heartbeats

Regardless of connection loss at the start or at an intermediate point, NM's retry behavior to the RM should follow the same flow.

ProcfsBasedProcessTree has a habit of emitting not-so-helpful messages such as the following:
{noformat}
2013-03-13 12:41:51,957 INFO [communication thread] org.apache.hadoop.yarn.util.ProcfsBasedProcessTree: The process 28747 may have finished in the interim.
2013-03-13 12:41:51,958 INFO [communication thread] org.apache.hadoop.yarn.util.ProcfsBasedProcessTree: The process 28978 may have finished in the interim.
2013-03-13 12:41:51,958 INFO [communication thread] org.apache.hadoop.yarn.util.ProcfsBasedProcessTree: The process 28979 may have finished in the interim.
{noformat}
As described in MAPREDUCE-4570, this is something that naturally occurs in the process of monitoring processes via procfs. It's uninteresting at best and can confuse users who think it's a reason their job isn't running as expected when it appears in their logs.
We should either make this DEBUG or remove it entirely.

YARN-475.
Major sub-task reported by Hitesh Shah and fixed by Hitesh Shah Remove ApplicationConstants.AM_APP_ATTEMPT_ID_ENV as it is no longer set in an AM's environment

AMs are expected to use ApplicationConstants.AM_CONTAINER_ID_ENV and derive the application attempt id from the container id.

YARN-474.
Major bug reported by Hitesh Shah and fixed by Zhijie Shen (capacityscheduler)CapacityScheduler does not activate applications when maximum-am-resource-percent configuration is refreshed

Submit 3 applications to a cluster where capacity scheduler limits allow only 1 running application. Modify capacity scheduler config to increase value of yarn.scheduler.capacity.maximum-am-resource-percent and invoke refresh queues.
The 2 applications not yet in running state do not get launched even though limits are increased.

Currently, scheduling mode in FS is limited to Fair and FIFO. The code typically has an if condition at multiple places to determine the correct course of action.
Making the scheduling mode pluggable helps in simplifying this process, particularly as we add new modes (DRF in this case).

YARN-468.
Major sub-task reported by Aleksey Gorshkov and fixed by Aleksey Gorshkov coverage fix for org.apache.hadoop.yarn.server.webproxy.amfilter

If we have multiple jobs which uses distributed cache with small size of files, the directory limit reaches before reaching the cache size and fails to create any directories in file cache (PUBLIC). The jobs start failing with the below exception.
java.io.IOException: mkdir of /tmp/nm-local-dir/filecache/3901886847734194975 failed
at org.apache.hadoop.fs.FileSystem.primitiveMkdir(FileSystem.java:909)
at org.apache.hadoop.fs.DelegateToFileSystem.mkdir(DelegateToFileSystem.java:143)
at org.apache.hadoop.fs.FilterFs.mkdir(FilterFs.java:189)
at org.apache.hadoop.fs.FileContext$4.next(FileContext.java:706)
at org.apache.hadoop.fs.FileContext$4.next(FileContext.java:703)
at org.apache.hadoop.fs.FileContext$FSLinkResolver.resolve(FileContext.java:2325)
at org.apache.hadoop.fs.FileContext.mkdir(FileContext.java:703)
at org.apache.hadoop.yarn.util.FSDownload.call(FSDownload.java:147)
at org.apache.hadoop.yarn.util.FSDownload.call(FSDownload.java:49)
at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:303)
at java.util.concurrent.FutureTask.run(FutureTask.java:138)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:441)
at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:303)
at java.util.concurrent.FutureTask.run(FutureTask.java:138)
at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
at java.lang.Thread.run(Thread.java:662)
we need to have a mechanism where in we can create directory hierarchy and limit number of files per directory.

YARN-460.
Blocker bug reported by Thomas Graves and fixed by Thomas Graves (capacityscheduler)CS user left in list of active users for the queue even when application finished

We have seen a user get left in the queues list of active users even though the application was removed. This can cause everyone else in the queue to get less resources if using the minimum user limit percent config.

YARN-458.
Major bug reported by Sandy Ryza and fixed by Sandy Ryza (nodemanager , resourcemanager)YARN daemon addresses must be placed in many different configs

The YARN resourcemanager's address is included in four different configs: yarn.resourcemanager.scheduler.address, yarn.resourcemanager.resource-tracker.address, yarn.resourcemanager.address, and yarn.resourcemanager.admin.address
A new user trying to configure a cluster needs to know the names of all these four configs.
The same issue exists for nodemanagers.
It would be much easier if they could simply specify yarn.resourcemanager.hostname and yarn.nodemanager.hostname and default ports for the other ones would kick in.

YARN-450.
Major sub-task reported by Bikas Saha and fixed by Zhijie Shen Define value for * in the scheduling protocol

The ResourceRequest has a string field to specify node/rack locations. For the cross-rack/cluster-wide location (ie when there is no locality constraint) the "*" string is used everywhere. However, its not defined anywhere and each piece of code either defines a local constant or uses the string literal. Defining "*" in the protocol and removing other local references from the code base will be good.

Now the compare code is :
return a1.getApplicationId().getId() - a2.getApplicationId().getId();
Will be replaced with :
return a1.getApplicationId().compareTo(a2.getApplicationId());
This will bring some benefits:
1,leave applicationId compare logic to ApplicationId class;
2,In future's HA mode,cluster time stamp may change,ApplicationId class already takes care of this condition.

YARN-444.
Major sub-task reported by Sandy Ryza and fixed by Sandy Ryza (api , applications/distributed-shell)Move special container exit codes from YarnConfiguration to API

YarnConfiguration currently contains the special container exit codes INVALID_CONTAINER_EXIT_STATUS = -1000, ABORTED_CONTAINER_EXIT_STATUS = -100, and DISKS_FAILED = -101.
These are not really not really related to configuration, and YarnConfiguration should not become a place to put miscellaneous constants.
Per discussion on YARN-417, appmaster writers need to be able to provide special handling for them, so it might make sense to move these to their own user-facing class.

YARN-441.
Major sub-task reported by Siddharth Seth and fixed by Xuan Gong Clean up unused collection methods in various APIs

There's a bunch of unused methods like getAskCount() and getAsk(index) in AllocateRequest, and other interfaces. These should be removed.
In YARN, found them in. MR will have it's own set.
AllocateRequest
StartContaienrResponse

NodeheartbeatResponse has another wrapper HeartbeatResponse under it, which can be removed.

YARN-426.
Critical bug reported by Jason Lowe and fixed by Jason Lowe (nodemanager)Failure to download a public resource on a node prevents further downloads of the resource from that node

If the NM encounters an error while downloading a public resource, it fails to empty the list of request events corresponding to the resource request in {{attempts}}. If the same public resource is subsequently requested on that node, {{PublicLocalizer.addResource}} will skip the download since it will mistakenly believe a download of that resource is already in progress. At that point any container that requests the public resource will just hang in the {{LOCALIZING}} state.

In the FifoScheduler, the assignNodeLocalContainers method is checking if the data is local to a node by searching for the nodeAddress of the node in the set of outstanding requests for the app. This seems to be incorrect as it should be checking hostname instead. The offending line of code is 455:
application.getResourceRequest(priority, node.getRMNode().getNodeAddress());
Requests are formated by hostname (e.g. host1.foo.com) whereas node addresses are a concatenation of hostname and command port (e.g. host1.foo.com:1234)
In the CapacityScheduler, it's done using hostname. See LeafQueue.assignNodeLocalContainers, line 1129
application.getResourceRequest(priority, node.getHostName());
Note that this bug does not affect the actual scheduling decisions made by the FifoScheduler because even though it incorrect determines that a request is not local to the node, it will still schedule the request immediately because it's rack-local. However, this bug may be adversely affecting the reporting of job status by underreporting the number of tasks that were node local.

YARN-410.
Major bug reported by Vinod Kumar Vavilapalli and fixed by Omkar Vinit Joshi New lines in diagnostics for a failed app on the per-application page make it hard to read

We need to fix the following issues on YARN web-UI:
- Remove the "Note" column from the application list. When a failure happens, this "Note" spoils the table layout.
- When the Application is still not running, the Tracking UI should be title "UNASSIGNED", for some reason it is titled "ApplicationMaster" but (correctly) links to "#".
- The per-application page has all the RM related information like version, start-time etc. Must be some accidental change by one of the patches.
- The diagnostics for a failed app on the per-application page don't retain new lines and wrap'em around - looks hard to read.

YARN-406.
Minor improvement reported by Hitesh Shah and fixed by Hitesh Shah TestRackResolver fails when local network resolves "host1" to a valid host

RMAppImpl.createAndGetApplicationReport can return a report with a null resource usage report if full access to the app is allowed but the application has no current attempt. This leads to NPEs in client code that assumes an app report will always have at least an empty resource usage report.

YARN-398.
Major sub-task reported by Arun C Murthy and fixed by Arun C Murthy Enhance CS to allow for white-list of resources

AllocateResponse contains an AMResponse and cluster node count. AMResponse that more data. Unless there is a good reason for this object structure, there should be either AMResponse or AllocateResponse.

YARN-392.
Major sub-task reported by Bikas Saha and fixed by Sandy Ryza (resourcemanager)Make it possible to specify hard locality constraints in resource requests

Currently its not possible to specify scheduling requests for specific nodes and nowhere else. The RM automatically relaxes locality to rack and * and assigns non-specified machines to the app.

We now have different and inconsistent naming schemes for various protocols. It was hard to explain to users, mainly in direct interactions at talks/presentations and user group meetings, with such naming.
We should fix these before we go beta.

YARN-385.
Major improvement reported by Sandy Ryza and fixed by Sandy Ryza (api)ResourceRequestPBImpl's toString() is missing location and # containers

ResourceRequestPBImpl's toString method includes priority and resource capability, but omits location and number of containers.

2013-02-06 09:31:33,813 INFO [Thread-2] service.CompositeService (CompositeService.java:stop(101)) - Error stopping org.apache.hadoop.yarn.client.AMRMClientImpl
org.apache.hadoop.HadoopIllegalArgumentException: Cannot close proxy since it is null
at org.apache.hadoop.ipc.RPC.stopProxy(RPC.java:605)
at org.apache.hadoop.yarn.client.AMRMClientImpl.stop(AMRMClientImpl.java:150)
at org.apache.hadoop.yarn.service.CompositeService.stop(CompositeService.java:99)
at org.apache.hadoop.yarn.service.CompositeService.stop(CompositeService.java:89)

YARN-382.
Major improvement reported by Thomas Graves and fixed by Zhijie Shen (scheduler)SchedulerUtils improve way normalizeRequest sets the resource capabilities

In YARN-370, we changed it from setting the capability to directly setting memory and cores:
- ask.setCapability(normalized);
+ ask.getCapability().setMemory(normalized.getMemory());
+ ask.getCapability().setVirtualCores(normalized.getVirtualCores());
We did this because it is directly setting the values in the original resource object passed in when the AM gets allocated and without it the AM doesn't get the resource normalized correctly in the submission context. See YARN-370 for more details.
I think we should find a better way of doing this long term, one so we don't have to keep adding things there when new resources are added, two because its a bit confusing as to what its doing and prone to someone accidentally breaking it in the future again. Something closer to what Arun suggested in YARN-370 would be better but we need to make sure all the places work and get some more testing on it before putting it in.

The MR2 FS docs could use some improvements.
Configuration:
- sizebasedweight - what is the "size" here? Total memory usage?
Pool properties:
- minResources - what does min amount of aggregate memory mean given that this is not a reservation?
- maxResources - is this a hard limit?
- weight: How is this ratio configured? Eg base is 1 and all weights are relative to that?
- schedulingMode - what is the default? Is fifo pure fifo, eg waits until all tasks for the job are finished before launching the next job?
There's no mention of ACLs, even though they're supported. See the CS docs for comparison.
Also there are a couple typos worth fixing while we're at it, eg "finish. apps to run"
Worth keeping in mind that some of these will need to be updated to reflect that resource calculators are now pluggable.

HADOOP-9252 slightly changed the format of some StringUtils outputs. It caused TestContainersMonitor to fail.
Also, some methods were deprecated by HADOOP-9252. The use of them should be replaced with the new methods.

YARN-376.
Blocker bug reported by Jason Lowe and fixed by Jason Lowe (resourcemanager)Apps that have completed can appear as RUNNING on the NM UI

On a busy cluster we've noticed a growing number of applications appear as RUNNING on a nodemanager web pages but the applications have long since finished. Looking at the NM logs, it appears the RM never told the nodemanager that the application had finished. This is also reflected in a jstack of the NM process, since many more log aggregation threads are running then one would expect from the number of actively running applications.

YARN-369.
Major sub-task reported by Hitesh Shah and fixed by Mayank Bansal (resourcemanager)Handle ( or throw a proper error when receiving) status updates from application masters that have not registered

Currently, an allocate call from an unregistered application is allowed and the status update for it throws a statemachine error that is silently dropped.
org.apache.hadoop.yarn.state.InvalidStateTransitonException: Invalid event: STATUS_UPDATE at LAUNCHED
at org.apache.hadoop.yarn.state.StateMachineFactory.doTransition(StateMachineFactory.java:302)
at org.apache.hadoop.yarn.state.StateMachineFactory.access$300(StateMachineFactory.java:43)
at org.apache.hadoop.yarn.state.StateMachineFactory$InternalStateMachine.doTransition(StateMachineFactory.java:445)
at org.apache.hadoop.yarn.server.resourcemanager.rmapp.attempt.RMAppAttemptImpl.handle(RMAppAttemptImpl.java:588)
at org.apache.hadoop.yarn.server.resourcemanager.rmapp.attempt.RMAppAttemptImpl.handle(RMAppAttemptImpl.java:99)
at org.apache.hadoop.yarn.server.resourcemanager.ResourceManager$ApplicationAttemptEventDispatcher.handle(ResourceManager.java:471)
at org.apache.hadoop.yarn.server.resourcemanager.ResourceManager$ApplicationAttemptEventDispatcher.handle(ResourceManager.java:452)
at org.apache.hadoop.yarn.event.AsyncDispatcher.dispatch(AsyncDispatcher.java:130)
at org.apache.hadoop.yarn.event.AsyncDispatcher$1.run(AsyncDispatcher.java:77)
at java.lang.Thread.run(Thread.java:680)
ApplicationMasterService should likely throw an appropriate error for applications' requests that should not be handled in such cases.

Noticed the following in an error log output while doing some experiements
./1066018/nodes/hyperion987/log/yarn-achu-nodemanager-hyperion987.out:java.lang.RuntimeException: No class defiend for uda.shuffle
"defiend" should be "defined"

YARN-365.
Major sub-task reported by Siddharth Seth and fixed by Xuan Gong (resourcemanager , scheduler)Each NM heartbeat should not generate an event for the Scheduler

Follow up from YARN-275
https://issues.apache.org/jira/secure/attachment/12567075/Prototype.txt

Starting up the proxy server fails with this error:
{noformat}
2013-01-29 17:37:41,357 FATAL webproxy.WebAppProxy (WebAppProxy.java:start(99)) - Could not start proxy web server
java.io.FileNotFoundException: webapps/proxy not found in CLASSPATH
at org.apache.hadoop.http.HttpServer.getWebAppsPath(HttpServer.java:533)
at org.apache.hadoop.http.HttpServer.<init>(HttpServer.java:225)
at org.apache.hadoop.http.HttpServer.<init>(HttpServer.java:164)
at org.apache.hadoop.yarn.server.webproxy.WebAppProxy.start(WebAppProxy.java:90)
at org.apache.hadoop.yarn.service.CompositeService.start(CompositeService.java:68)
at org.apache.hadoop.yarn.server.webproxy.WebAppProxyServer.main(WebAppProxyServer.java:94)
{noformat}

When using the search box on the web UI to search for a specific task number (e.g.: "0831"), sometimes unexpected extra results are shown. Using the web browser's built-in search-within-page does not show any hits, so these look like completely spurious results.
It looks like the raw timestamp value for time columns, which is not shown in the table, is also being searched with the search box.

With YARN-2 checked in, CPU info are taken into consideration in resource scheduling. yarn node -status <NodeID> should show CPU used and capacity info as memory info.

YARN-345.
Critical bug reported by Devaraj K and fixed by Robert Parker (nodemanager)Many InvalidStateTransitonException errors for ApplicationImpl in Node Manager

{code:xml}
org.apache.hadoop.yarn.state.InvalidStateTransitonException: Invalid event: FINISH_APPLICATION at FINISHED
at org.apache.hadoop.yarn.state.StateMachineFactory.doTransition(StateMachineFactory.java:301)
at org.apache.hadoop.yarn.state.StateMachineFactory.access$300(StateMachineFactory.java:43)
at org.apache.hadoop.yarn.state.StateMachineFactory$InternalStateMachine.doTransition(StateMachineFactory.java:443)
at org.apache.hadoop.yarn.server.nodemanager.containermanager.application.ApplicationImpl.handle(ApplicationImpl.java:398)
at org.apache.hadoop.yarn.server.nodemanager.containermanager.application.ApplicationImpl.handle(ApplicationImpl.java:58)
at org.apache.hadoop.yarn.server.nodemanager.containermanager.ContainerManagerImpl$ApplicationEventDispatcher.handle(ContainerManagerImpl.java:520)
at org.apache.hadoop.yarn.server.nodemanager.containermanager.ContainerManagerImpl$ApplicationEventDispatcher.handle(ContainerManagerImpl.java:512)
at org.apache.hadoop.yarn.event.AsyncDispatcher.dispatch(AsyncDispatcher.java:126)
at org.apache.hadoop.yarn.event.AsyncDispatcher$1.run(AsyncDispatcher.java:75)
at java.lang.Thread.run(Thread.java:662)
{code}
{code:xml}
2013-01-17 04:03:46,726 WARN org.apache.hadoop.yarn.server.nodemanager.containermanager.application.Application: Can't handle this event at current state
org.apache.hadoop.yarn.state.InvalidStateTransitonException: Invalid event: FINISH_APPLICATION at APPLICATION_RESOURCES_CLEANINGUP
at org.apache.hadoop.yarn.state.StateMachineFactory.doTransition(StateMachineFactory.java:301)
at org.apache.hadoop.yarn.state.StateMachineFactory.access$300(StateMachineFactory.java:43)
at org.apache.hadoop.yarn.state.StateMachineFactory$InternalStateMachine.doTransition(StateMachineFactory.java:443)
at org.apache.hadoop.yarn.server.nodemanager.containermanager.application.ApplicationImpl.handle(ApplicationImpl.java:398)
at org.apache.hadoop.yarn.server.nodemanager.containermanager.application.ApplicationImpl.handle(ApplicationImpl.java:58)
at org.apache.hadoop.yarn.server.nodemanager.containermanager.ContainerManagerImpl$ApplicationEventDispatcher.handle(ContainerManagerImpl.java:520)
at org.apache.hadoop.yarn.server.nodemanager.containermanager.ContainerManagerImpl$ApplicationEventDispatcher.handle(ContainerManagerImpl.java:512)
at org.apache.hadoop.yarn.event.AsyncDispatcher.dispatch(AsyncDispatcher.java:126)
at org.apache.hadoop.yarn.event.AsyncDispatcher$1.run(AsyncDispatcher.java:75)
at java.lang.Thread.run(Thread.java:662)
{code}
{code:xml}
2013-01-17 00:01:11,006 WARN org.apache.hadoop.yarn.server.nodemanager.containermanager.application.Application: Can't handle this event at current state
org.apache.hadoop.yarn.state.InvalidStateTransitonException: Invalid event: FINISH_APPLICATION at FINISHING_CONTAINERS_WAIT
at org.apache.hadoop.yarn.state.StateMachineFactory.doTransition(StateMachineFactory.java:301)
at org.apache.hadoop.yarn.state.StateMachineFactory.access$300(StateMachineFactory.java:43)
at org.apache.hadoop.yarn.state.StateMachineFactory$InternalStateMachine.doTransition(StateMachineFactory.java:443)
at org.apache.hadoop.yarn.server.nodemanager.containermanager.application.ApplicationImpl.handle(ApplicationImpl.java:398)
at org.apache.hadoop.yarn.server.nodemanager.containermanager.application.ApplicationImpl.handle(ApplicationImpl.java:58)
at org.apache.hadoop.yarn.server.nodemanager.containermanager.ContainerManagerImpl$ApplicationEventDispatcher.handle(ContainerManagerImpl.java:520)
at org.apache.hadoop.yarn.server.nodemanager.containermanager.ContainerManagerImpl$ApplicationEventDispatcher.handle(ContainerManagerImpl.java:512)
at org.apache.hadoop.yarn.event.AsyncDispatcher.dispatch(AsyncDispatcher.java:126)
at org.apache.hadoop.yarn.event.AsyncDispatcher$1.run(AsyncDispatcher.java:75)
at java.lang.Thread.run(Thread.java:662)
{code}
{code:xml}
2013-01-17 10:56:36,975 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.container.Container: Container container_1358385982671_1304_01_000001 transitioned from NEW to DONE
2013-01-17 10:56:36,975 WARN org.apache.hadoop.yarn.server.nodemanager.containermanager.application.Application: Can't handle this event at current state
org.apache.hadoop.yarn.state.InvalidStateTransitonException: Invalid event: APPLICATION_CONTAINER_FINISHED at FINISHED
at org.apache.hadoop.yarn.state.StateMachineFactory.doTransition(StateMachineFactory.java:301)
at org.apache.hadoop.yarn.state.StateMachineFactory.access$300(StateMachineFactory.java:43)
at org.apache.hadoop.yarn.state.StateMachineFactory$InternalStateMachine.doTransition(StateMachineFactory.java:443)
at org.apache.hadoop.yarn.server.nodemanager.containermanager.application.ApplicationImpl.handle(ApplicationImpl.java:398)
at org.apache.hadoop.yarn.server.nodemanager.containermanager.application.ApplicationImpl.handle(ApplicationImpl.java:58)
at org.apache.hadoop.yarn.server.nodemanager.containermanager.ContainerManagerImpl$ApplicationEventDispatcher.handle(ContainerManagerImpl.java:520)
at org.apache.hadoop.yarn.server.nodemanager.containermanager.ContainerManagerImpl$ApplicationEventDispatcher.handle(ContainerManagerImpl.java:512)
at org.apache.hadoop.yarn.event.AsyncDispatcher.dispatch(AsyncDispatcher.java:126)
at org.apache.hadoop.yarn.event.AsyncDispatcher$1.run(AsyncDispatcher.java:75)
at java.lang.Thread.run(Thread.java:662)
2013-01-17 10:56:36,975 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.application.Application: Application application_1358385982671_1304 transitioned from FINISHED to null
{code}
{code:xml}
2013-01-17 10:56:36,026 WARN org.apache.hadoop.yarn.server.nodemanager.containermanager.application.Application: Can't handle this event at current state
org.apache.hadoop.yarn.state.InvalidStateTransitonException: Invalid event: INIT_CONTAINER at FINISHED
at org.apache.hadoop.yarn.state.StateMachineFactory.doTransition(StateMachineFactory.java:301)
at org.apache.hadoop.yarn.state.StateMachineFactory.access$300(StateMachineFactory.java:43)
at org.apache.hadoop.yarn.state.StateMachineFactory$InternalStateMachine.doTransition(StateMachineFactory.java:443)
at org.apache.hadoop.yarn.server.nodemanager.containermanager.application.ApplicationImpl.handle(ApplicationImpl.java:398)
at org.apache.hadoop.yarn.server.nodemanager.containermanager.application.ApplicationImpl.handle(ApplicationImpl.java:58)
at org.apache.hadoop.yarn.server.nodemanager.containermanager.ContainerManagerImpl$ApplicationEventDispatcher.handle(ContainerManagerImpl.java:520)
at org.apache.hadoop.yarn.server.nodemanager.containermanager.ContainerManagerImpl$ApplicationEventDispatcher.handle(ContainerManagerImpl.java:512)
at org.apache.hadoop.yarn.event.AsyncDispatcher.dispatch(AsyncDispatcher.java:126)
at org.apache.hadoop.yarn.event.AsyncDispatcher$1.run(AsyncDispatcher.java:75)
at java.lang.Thread.run(Thread.java:662)
2013-01-17 10:56:36,026 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.application.Application: Application application_1358385982671_1304 transitioned from FINISHED to null
{code}

YARN-333.
Major bug reported by Sandy Ryza and fixed by Sandy Ryza Schedulers cannot control the queue-name of an application

Currently, if an app is submitted without a queue, RMAppManager sets the RMApp's queue to "default".
A scheduler may wish to make its own decision on which queue to place an app in if none is specified. For example, when the fair scheduler user-as-default-queue config option is set to true, and an app is submitted with no queue specified, the fair scheduler should assign the app to a queue with the user's name.

YARN-326.
Major new feature reported by Sandy Ryza and fixed by Sandy Ryza (scheduler)Add multi-resource scheduling to the fair scheduler

With YARN-2 in, the capacity scheduler has the ability to schedule based on multiple resources, using dominant resource fairness. The fair scheduler should be able to do multiple resource scheduling as well, also using dominant resource fairness.
More details to come on how the corner cases with fair scheduler configs such as min and max resources will be handled.

YARN-319.
Major bug reported by shenhong and fixed by shenhong (resourcemanager , scheduler)Submit a job to a queue that not allowed in fairScheduler, client will hold forever.

RM use fairScheduler, when client submit a job to a queue, but the queue do not allow the user to submit job it, in this case, client will hold forever.

{code:xml}
2012-12-28 14:03:56,956 ERROR org.apache.hadoop.yarn.server.resourcemanager.rmapp.attempt.RMAppAttemptImpl: Can't handle this event at current state
org.apache.hadoop.yarn.state.InvalidStateTransitonException: Invalid event: CONTAINER_FINISHED at ALLOCATED
at org.apache.hadoop.yarn.state.StateMachineFactory.doTransition(StateMachineFactory.java:301)
at org.apache.hadoop.yarn.state.StateMachineFactory.access$300(StateMachineFactory.java:43)
at org.apache.hadoop.yarn.state.StateMachineFactory$InternalStateMachine.doTransition(StateMachineFactory.java:443)
at org.apache.hadoop.yarn.server.resourcemanager.rmapp.attempt.RMAppAttemptImpl.handle(RMAppAttemptImpl.java:490)
at org.apache.hadoop.yarn.server.resourcemanager.rmapp.attempt.RMAppAttemptImpl.handle(RMAppAttemptImpl.java:80)
at org.apache.hadoop.yarn.server.resourcemanager.ResourceManager$ApplicationAttemptEventDispatcher.handle(ResourceManager.java:433)
at org.apache.hadoop.yarn.server.resourcemanager.ResourceManager$ApplicationAttemptEventDispatcher.handle(ResourceManager.java:414)
at org.apache.hadoop.yarn.event.AsyncDispatcher.dispatch(AsyncDispatcher.java:126)
at org.apache.hadoop.yarn.event.AsyncDispatcher$1.run(AsyncDispatcher.java:75)
at java.lang.Thread.run(Thread.java:662)
{code}

An application requests a container with 1024 MB. It then requests a container with 2048 MB. A node shows up with 1024 MB available. Even if the application is the only one running, neither request will be scheduled on it.

YARN-269.
Major bug reported by Thomas Graves and fixed by Jason Lowe (resourcemanager)Resource Manager not logging the health_check_script result when taking it out

The Resource Manager not logging the health_check_script result when taking it out. This was added to jobtracker in 1.x with MAPREDUCE-2451, we should do the same thing for RM.

YARN-249.
Major improvement reported by Ravi Prakash and fixed by Ravi Prakash (capacityscheduler)Capacity Scheduler web page should show list of active users per queue like it used to (in 1.x)

On the jobtracker, the web ui showed the active users for each queue and how much resources each of those users were using. That currently isn't being displayed on the RM capacity scheduler web ui.

YARN-237.
Major improvement reported by Ravi Prakash and fixed by Jian He (resourcemanager)Refreshing the RM page forgets how many rows I had in my Datatables

If I choose a 100 rows, and then refresh the page, DataTables goes back to showing me 20 rows.
This user preference should be stored in a cookie.

YARN-236.
Major bug reported by Jason Lowe and fixed by Jason Lowe (resourcemanager)RM should point tracking URL to RM web page when app fails to start

Similar to YARN-165, the RM should redirect the tracking URL to the specific app page on the RM web UI when the application fails to start. For example, if the AM completely fails to start due to bad AM config or bad job config like invalid queuename, then the user gets the unhelpful "The requested application exited before setting a tracking URL".
Usually the diagnostic string on the RM app page has something useful, so we might as well point there.

YARN-227.
Major bug reported by Jason Lowe and fixed by Jason Lowe (resourcemanager)Application expiration difficult to debug for end-users

When an AM attempt expires the AMLivelinessMonitor in the RM will kill the job and mark it as failed. However there are no diagnostic messages set for the application indicating that the application failed because of expiration. Even if the AM logs are examined, it's often not obvious that the application was externally killed. The only evidence of what happened to the application is currently in the RM logs, and those are often not accessible by users.

Say application A is submitted but at that time it does not meet the bar for activation because of resource limit settings for applications. After that if more hardware is added to the system and the application becomes valid it still remains in pending state, likely forever.
This might be rare to hit in real life because enough NM's heartbeat to the RM before applications can get submitted. But a change in settings or heartbeat interval might make it easier to repro. In RM restart scenarios, this will likely hit more if its implemented by re-playing events and re-submitting applications to the scheduler before the RPC to NM's is activated.

YARN-200.
Major sub-task reported by Robert Joseph Evans and fixed by Ravi Prakash yarn log does not output all needed information, and is in a binary format

yarn logs does not output attemptid, nodename, or container-id. Missing these makes it very difficult to look through the logs for failed containers and tie them back to actual tasks and task attempts.
Also the output currently includes several binary characters. This is OK for being machine readable, but difficult for being human readable, or even for using standard tool like grep.
The help message can also be more useful to users

YARN-198.
Minor improvement reported by Ramgopal N and fixed by Jian He (nodemanager)If we are navigating to Nodemanager UI from Resourcemanager,then there is not link to navigate back to Resource manager

If we are navigating to Nodemanager by clicking on the node link in RM,there is no link provided on the NM to navigate back to RM.
If there is a link to navigate back to RM it would be good

YARN-196.
Major bug reported by Ramgopal N and fixed by Xuan Gong (nodemanager)Nodemanager should be more robust in handling connection failure to ResourceManager when a cluster is started

Ref: MAPREDUCE-4067
All YARN APIs currently throw YarnRemoteException.
1) This cannot be extended in it's current form.
2) The RPC layer can throw IOExceptions. These end up showing up as UndeclaredThrowableExceptions.

Make the yarn client services more robust against being shut down while not started, or shutdown more than once, by null-checking fields before closing them, setting to null afterwards to prevent double-invocation. This is a subset of MAPREDUCE-3502

Add the nodemanager bits of MAPREDUCE-3502 to shut down the Nodemanager services. This is done by checking for fields being non-null before shutting down/closing etc, and setting the fields to null afterwards -to be resilient against re-entrancy.
No tests other than manual review.

Split MAPREDUCE-3502 patches to make the RM code more resilient to being stopped more than once, or before started.
This depends on MAPREDUCE-4014.

YARN-117.
Major improvement reported by Steve Loughran and fixed by Steve Loughran Enhance YARN service model

Having played the YARN service model, there are some issues
that I've identified based on past work and initial use.
This JIRA issue is an overall one to cover the issues, with solutions pushed out to separate JIRAs.
h2. state model prevents stopped state being entered if you could not successfully start the service.
In the current lifecycle you cannot stop a service unless it was successfully started, but
* {{init()}} may acquire resources that need to be explicitly released
* if the {{start()}} operation fails partway through, the {{stop()}} operation may be needed to release resources.
*Fix:* make {{stop()}} a valid state transition from all states and require the implementations to be able to stop safely without requiring all fields to be non null.
Before anyone points out that the {{stop()}} operations assume that all fields are valid; and if called before a {{start()}} they will NPE; MAPREDUCE-3431 shows that this problem arises today, MAPREDUCE-3502 is a fix for this. It is independent of the rest of the issues in this doc but it will aid making {{stop()}} execute from all states other than "stopped".
MAPREDUCE-3502 is too big a patch and needs to be broken down for easier review and take up; this can be done with issues linked to this one.
h2. AbstractService doesn't prevent duplicate state change requests.
The {{ensureState()}} checks to verify whether or not a state transition is allowed from the current state are performed in the base {{AbstractService}} class -yet subclasses tend to call this *after* their own {{init()}}, {{start()}} & {{stop()}} operations. This means that these operations can be performed out of order, and even if the outcome of the call is an exception, all actions performed by the subclasses will have taken place. MAPREDUCE-3877 demonstrates this.
This is a tricky one to address. In HADOOP-3128 I used a base class instead of an interface and made the {{init()}}, {{start()}} & {{stop()}} methods {{final}}. These methods would do the checks, and then invoke protected inner methods, {{innerStart()}}, {{innerStop()}}, etc. It should be possible to retrofit the same behaviour to everything that extends {{AbstractService}} -something that must be done before the class is considered stable (because once the lifecycle methods are declared final, all subclasses that are out of the source tree will need fixing by the respective developers.
h2. AbstractService state change doesn't defend against race conditions.
There's no concurrency locks on the state transitions. Whatever fix for wrong state calls is added should correct this to prevent re-entrancy, such as {{stop()}} being called from two threads.
h2. Static methods to choreograph of lifecycle operations
Helper methods to move things through lifecycles. init->start is common, stop-if-service!=null another. Some static methods can execute these, and even call {{stop()}} if {{init()}} raises an exception. These could go into a class {{ServiceOps}} in the same package. These can be used by those services that wrap other services, and help manage more robust shutdowns.
h2. state transition failures are something that registered service listeners may wish to be informed of.
When a state transition fails a {{RuntimeException}} can be thrown -and the service listeners are not informed as the notification point isn't reached. They may wish to know this, especially for management and diagnostics.
*Fix:* extend {{ServiceStateChangeListener}} with a callback such as {{stateChangeFailed(Service service,Service.State targeted-state, RuntimeException e)}} that is invoked from the (final) state change methods in the {{AbstractService}} class (once they delegate to their inner {{innerStart()}}, {{innerStop()}} methods; make a no-op on the existing implementations of the interface.
h2. Service listener failures not handled
Is this an error an error or not? Log and ignore may not be what is desired.
*Proposed:* during {{stop()}} any exception by a listener is caught and discarded, to increase the likelihood of a better shutdown, but do not add try-catch clauses to the other state changes.
h2. Support static listeners for all AbstractServices
Add support to {{AbstractService}} that allow callers to register listeners for all instances. The existing listener interface could be used. This allows management tools to hook into the events.
The static listeners would be invoked for all state changes except creation (base class shouldn't be handing out references to itself at this point).
These static events could all be async, pushed through a shared {{ConcurrentLinkedQueue}}; failures logged at warn and the rest of the listeners invoked.
h2. Add some example listeners for management/diagnostics
* event to commons log for humans.
* events for machines hooked up to the JSON logger.
* for testing: something that be told to fail.
h2. Services should support signal interruptibility
The services would benefit from a way of shutting them down on a kill signal; this can be done via a runtime hook. It should not be automatic though, as composite services will get into a very complex state during shutdown. Better to provide a hook that lets you register/unregister services to terminate, and have the relevant {{main()}} entry points tell their root services to register themselves.

YARN-112.
Major sub-task reported by Jason Lowe and fixed by Omkar Vinit Joshi (nodemanager)Race in localization can cause containers to fail

On one of our 0.23 clusters, I saw a case of two containers, corresponding to two map tasks of a MR job, that were launched almost simultaneously on the same node. It appears they both tried to localize job.jar and job.xml at the same time. One of the containers failed when it couldn't rename the temporary job.jar directory to its final name because the target directory wasn't empty. Shortly afterwards the second container failed because job.xml could not be found, presumably because the first container removed it when it cleaned up.

YARN-109.
Major bug reported by Jason Lowe and fixed by Mayank Bansal (nodemanager).tmp file is not deleted for localized archives

When archives are localized they are initially created as a .tmp file and unpacked from that file. However the .tmp file is not deleted afterwards.