AM info in job -list does not reflect the actual AM hostname

Details

Description

The AM info field on "bin/mapred job -list" currently has a value <resourcemanager hostname>:8088/proxy/appID. This info is irrelevant unless it shows the real information of where the AM was launched. This needs to be fixed to show the AM host details.

Also the whole point of the proxy is to mitigate the opportunities for an AM to steal cookies from an end user. It does not do a very good job of it, and I really would prefer to see a different architecture for how it does this, but this is what we have. If you want to put in the change as is I am not going to block it, but I suspect that the security people here will force me to change it before we can deploy into production, so I would prefer to just have a separate field that is not a URL.

Robert Joseph Evans
added a comment - 01/Mar/12 18:11 Also the whole point of the proxy is to mitigate the opportunities for an AM to steal cookies from an end user. It does not do a very good job of it, and I really would prefer to see a different architecture for how it does this, but this is what we have. If you want to put in the change as is I am not going to block it, but I suspect that the security people here will force me to change it before we can deploy into production, so I would prefer to just have a separate field that is not a URL.

The issue is that the AM is arbitrary code. It does not have to be something that we wrote. The only thing that we can do to restrict it is what the OS allows us to do. If there is a good way for us to make the OS not allow any incoming connections to the HTTP port of the AM then I am all for that. Sadly the HTTP port is an ephemeral port and so is the RPC port that needs to be open to allow connections from other locations.

I would prefer a different column that can show the host that the AM is running on. It should come from the RM indicating where it was scheduler, because it is the AM that reports back both the web app URL and the RPC url.

Robert Joseph Evans
added a comment - 01/Mar/12 18:06 The issue is that the AM is arbitrary code. It does not have to be something that we wrote. The only thing that we can do to restrict it is what the OS allows us to do. If there is a good way for us to make the OS not allow any incoming connections to the HTTP port of the AM then I am all for that. Sadly the HTTP port is an ephemeral port and so is the RPC port that needs to be open to allow connections from other locations.
I would prefer a different column that can show the host that the AM is running on. It should come from the RM indicating where it was scheduler, because it is the AM that reports back both the web app URL and the RPC url.

What we really want to show is the AM hostname, either through the scheduling-Info field or another.

If an end user sees this output they are going to copy and past this into the browser and go to that link. If the AM is well behaved it will redirect the user back through the proxy, but if the AM is malicious, it will do exactly what the proxy is intended to help mitigate.

This clearly seems like an issue with proxy or the AMs. We cannot rely on the fact that users will behave properly and will always click through the proxy for the sake of security. I agree that exposing this directly reduces security that exists today but this is as good as anyone doing an RPC call to figure out where an AM is running.

The real issue (if there is one) is that AMs accept requests that do not originate from the proxy. We should fix that if at all, instead of trying to avoid users from not seeing information like this. What do you think?

Vinod Kumar Vavilapalli
added a comment - 29/Feb/12 04:28 What we really want to show is the AM hostname, either through the scheduling-Info field or another.
If an end user sees this output they are going to copy and past this into the browser and go to that link. If the AM is well behaved it will redirect the user back through the proxy, but if the AM is malicious, it will do exactly what the proxy is intended to help mitigate.
This clearly seems like an issue with proxy or the AMs. We cannot rely on the fact that users will behave properly and will always click through the proxy for the sake of security. I agree that exposing this directly reduces security that exists today but this is as good as anyone doing an RPC call to figure out where an AM is running.
The real issue (if there is one) is that AMs accept requests that do not originate from the proxy. We should fix that if at all, instead of trying to avoid users from not seeing information like this. What do you think?

I am a bit confused by why we want to expose this to the end user. I can see that it is very useful to QE trying to test, or a developer trying to debug an issue, but this can potentially bypass the web application proxy, which is a security issue. If an end user sees this output they are going to copy and past this into the browser and go to that link. If the AM is well behaved it will redirect the user back through the proxy, but if the AM is malicious, it will do exactly what the proxy is intended to help mitigate. I don't think that this is critical, there are enough holes in the proxy when it is working properly anyways, but this patch is potentially reducing the security of the system.

Robert Joseph Evans
added a comment - 27/Feb/12 18:22 I am a bit confused by why we want to expose this to the end user. I can see that it is very useful to QE trying to test, or a developer trying to debug an issue, but this can potentially bypass the web application proxy, which is a security issue. If an end user sees this output they are going to copy and past this into the browser and go to that link. If the AM is well behaved it will redirect the user back through the proxy, but if the AM is malicious, it will do exactly what the proxy is intended to help mitigate. I don't think that this is critical, there are enough holes in the proxy when it is working properly anyways, but this patch is potentially reducing the security of the system.

Hadoop QA
added a comment - 15/Feb/12 10:06 +1 overall. Here are the results of testing the latest attachment
http://issues.apache.org/jira/secure/attachment/12514547/MAPREDUCE-3761-20120214.1.txt
against trunk revision .
+1 @author. The patch does not contain any @author tags.
+1 tests included. The patch appears to include 3 new or modified tests.
+1 javadoc. The javadoc tool did not generate any warning messages.
+1 javac. The applied patch does not increase the total number of javac compiler warnings.
+1 eclipse:eclipse. The patch built with eclipse:eclipse.
+1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9) warnings.
+1 release audit. The applied patch does not increase the total number of release audit warnings.
+1 core tests. The patch passed unit tests in .
+1 contrib tests. The patch passed contrib unit tests.
Test results: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/1856//testReport/
Console output: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/1856//console
This message is automatically generated.

Hadoop QA
added a comment - 04/Feb/12 20:06 -1 overall. Here are the results of testing the latest attachment
http://issues.apache.org/jira/secure/attachment/12513071/MAPREDUCE-3761-20120202.txt
against trunk revision .
+1 @author. The patch does not contain any @author tags.
+1 tests included. The patch appears to include 3 new or modified tests.
+1 javadoc. The javadoc tool did not generate any warning messages.
+1 javac. The applied patch does not increase the total number of javac compiler warnings.
+1 eclipse:eclipse. The patch built with eclipse:eclipse.
+1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9) warnings.
+1 release audit. The applied patch does not increase the total number of release audit warnings.
-1 core tests. The patch failed these unit tests:
org.apache.hadoop.mapreduce.TestTypeConverter
+1 contrib tests. The patch passed contrib unit tests.
Test results: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/1777//testReport/
Console output: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/1777//console
This message is automatically generated.

ContainerMemoryManager while managing memory of container_0_0000_01_000000
java.lang.NumberFormatException: For input string: "18446743988089421650"
at java.lang.NumberFormatException.forInputString(NumberFormatException.java:48)
at java.lang.Long.parseLong(Long.java:422)
at java.lang.Long.parseLong(Long.java:468)
at org.apache.hadoop.yarn.util.ProcfsBasedProcessTree.constructProcessInfo(ProcfsBasedProcessTree.java:427)
at org.apache.hadoop.yarn.util.ProcfsBasedProcessTree.getProcessTree(ProcfsBasedProcessTree.java:168)

Mahadev konar
added a comment - 04/Feb/12 19:46 Looks like the test failed because of:
ContainerMemoryManager while managing memory of container_0_0000_01_000000
java.lang.NumberFormatException: For input string: "18446743988089421650"
at java.lang.NumberFormatException.forInputString(NumberFormatException.java:48)
at java.lang. Long .parseLong( Long .java:422)
at java.lang. Long .parseLong( Long .java:468)
at org.apache.hadoop.yarn.util.ProcfsBasedProcessTree.constructProcessInfo(ProcfsBasedProcessTree.java:427)
at org.apache.hadoop.yarn.util.ProcfsBasedProcessTree.getProcessTree(ProcfsBasedProcessTree.java:168)
which is MAPREDUCE-3583 . Running hudson again.

Hadoop QA
added a comment - 03/Feb/12 00:35 -1 overall. Here are the results of testing the latest attachment
http://issues.apache.org/jira/secure/attachment/12513071/MAPREDUCE-3761-20120202.txt
against trunk revision .
+1 @author. The patch does not contain any @author tags.
+1 tests included. The patch appears to include 3 new or modified tests.
-1 javadoc. The javadoc tool appears to have generated 1 warning messages.
+1 javac. The applied patch does not increase the total number of javac compiler warnings.
+1 eclipse:eclipse. The patch built with eclipse:eclipse.
+1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9) warnings.
+1 release audit. The applied patch does not increase the total number of release audit warnings.
-1 core tests. The patch failed these unit tests:
org.apache.hadoop.yarn.server.nodemanager.containermanager.monitor.TestContainersMonitor
+1 contrib tests. The patch passed contrib unit tests.
Test results: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/1758//testReport/
Console output: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/1758//console
This message is automatically generated.

Vinod Kumar Vavilapalli
added a comment - 02/Feb/12 23:52 This should fix it. Modified test to verify the same.
Also added a BuilderUtil for ApplicationResourceUsageReport which I am then using it everywhere.