capacity scheduler web ui bar graphs for used capacity wrong

Details

Description

The capacity scheduler web ui has bar graphs showing the capacity/used capacity/max capacity for each queue. The used capacity it is showing is actually the % of its parents queue it is using, which doesn't make sense on the bar graphs when compared to the capacity and max capacity of that particular queue. The bar graphs should be using utilization so that the user can see that its using x% or the y% allocated to that queue.

Activity

In this screen shots, this is single node cluster 8 GB total memory. The default queue capacity is 20% which would be 1638MB allocated to it. You will see that its currently using 2G so the usage is 125% - which is the utilization.

Thomas Graves
added a comment - 06/Feb/12 15:20 In this screen shots, this is single node cluster 8 GB total memory. The default queue capacity is 20% which would be 1638MB allocated to it. You will see that its currently using 2G so the usage is 125% - which is the utilization.

after thinking about this more. I changed the bar graphs to use the absolute capacities rather then the relative capacity/max capacity. The reason is that max capacity is not relative to capacity so having the graph show up properly when used capacity > capacity when max capacity configured to be > capacity is not possible in a fashion that shows up both capacity and max capacity properly.

The above should be a valid config but the graph would totally be messed up if displaying capacity 80% and max capacity 20%.

I also changed used capacity to be what utilization was defined as. I think this will make more sense to the user and matches what was in 0.20.X/1.0. I removed utilization and added an absolute used capacity that can be used by the bar graphs.

While I was in there I also changed it so that instead of opening all the divs and then closing the leaf queue ones, now everything starts closed and we only open the parent queues.

Thomas Graves
added a comment - 08/Feb/12 00:07 after thinking about this more. I changed the bar graphs to use the absolute capacities rather then the relative capacity/max capacity. The reason is that max capacity is not relative to capacity so having the graph show up properly when used capacity > capacity when max capacity configured to be > capacity is not possible in a fashion that shows up both capacity and max capacity properly.
Another issue with using the relative ones is if max capacity is < capacity:
root queue = 100%
root.test: cap = 20%, max capacity = 100%, absolute cap = 20%, absolute max = 100%
root.test.a1: cap = 80%, max capacity = 20%, absolute cap = 16%, absolute max = 20%
The above should be a valid config but the graph would totally be messed up if displaying capacity 80% and max capacity 20%.
I also changed used capacity to be what utilization was defined as. I think this will make more sense to the user and matches what was in 0.20.X/1.0. I removed utilization and added an absolute used capacity that can be used by the bar graphs.
While I was in there I also changed it so that instead of opening all the divs and then closing the leaf queue ones, now everything starts closed and we only open the parent queues.

Hadoop QA
added a comment - 08/Feb/12 02:05 +1 overall. Here are the results of testing the latest attachment
http://issues.apache.org/jira/secure/attachment/12513736/MAPREDUCE-3816.patch
against trunk revision .
+1 @author. The patch does not contain any @author tags.
+1 tests included. The patch appears to include 6 new or modified tests.
+1 javadoc. The javadoc tool did not generate any warning messages.
+1 javac. The applied patch does not increase the total number of javac compiler warnings.
+1 eclipse:eclipse. The patch built with eclipse:eclipse.
+1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9) warnings.
+1 release audit. The applied patch does not increase the total number of release audit warnings.
+1 core tests. The patch passed unit tests in .
+1 contrib tests. The patch passed contrib unit tests.
Test results: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/1821//testReport/
Console output: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/1821//console
This message is automatically generated.

Hadoop QA
added a comment - 08/Feb/12 15:25 +1 overall. Here are the results of testing the latest attachment
http://issues.apache.org/jira/secure/attachment/12513818/MAPREDUCE-3816.patch
against trunk revision .
+1 @author. The patch does not contain any @author tags.
+1 tests included. The patch appears to include 6 new or modified tests.
+1 javadoc. The javadoc tool did not generate any warning messages.
+1 javac. The applied patch does not increase the total number of javac compiler warnings.
+1 eclipse:eclipse. The patch built with eclipse:eclipse.
+1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9) warnings.
+1 release audit. The applied patch does not increase the total number of release audit warnings.
+1 core tests. The patch passed unit tests in .
+1 contrib tests. The patch passed contrib unit tests.
Test results: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/1824//testReport/
Console output: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/1824//console
This message is automatically generated.

CSQueueUtils.java seems to have been reformatted, which adds a lot of diffs to the patch that aren't necessary

In CSQueueUtils.checkAbsoluteCapacities(), do we need to worry about floating point precision when comparing absMaxCapacity with absCapacity? Wondering if we need a tolerance there to handle floating point computation errors.

Super-Nit: In ParentQueue.java and LeafQueue.java, we should add the @Override decorator to setUsedCapacity() and setAbsoluteUsedCapacity()

Jason Lowe
added a comment - 08/Feb/12 17:43 Patch looks pretty good. Some comments:
CSQueueUtils.java seems to have been reformatted, which adds a lot of diffs to the patch that aren't necessary
In CSQueueUtils.checkAbsoluteCapacities(), do we need to worry about floating point precision when comparing absMaxCapacity with absCapacity? Wondering if we need a tolerance there to handle floating point computation errors.
Super-Nit: In ParentQueue.java and LeafQueue.java, we should add the @Override decorator to setUsedCapacity() and setAbsoluteUsedCapacity()

Hadoop QA
added a comment - 08/Feb/12 22:55 +1 overall. Here are the results of testing the latest attachment
http://issues.apache.org/jira/secure/attachment/12513871/MAPREDUCE-3816.patch
against trunk revision .
+1 @author. The patch does not contain any @author tags.
+1 tests included. The patch appears to include 6 new or modified tests.
+1 javadoc. The javadoc tool did not generate any warning messages.
+1 javac. The applied patch does not increase the total number of javac compiler warnings.
+1 eclipse:eclipse. The patch built with eclipse:eclipse.
+1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9) warnings.
+1 release audit. The applied patch does not increase the total number of release audit warnings.
+1 core tests. The patch passed unit tests in .
+1 contrib tests. The patch passed contrib unit tests.
Test results: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/1827//testReport/
Console output: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/1827//console
This message is automatically generated.

Hadoop QA
added a comment - 28/Feb/12 20:29 +1 overall. Here are the results of testing the latest attachment
http://issues.apache.org/jira/secure/attachment/12516378/MAPREDUCE-3816.patch
against trunk revision .
+1 @author. The patch does not contain any @author tags.
+1 tests included. The patch appears to include 6 new or modified tests.
+1 javadoc. The javadoc tool did not generate any warning messages.
+1 javac. The applied patch does not increase the total number of javac compiler warnings.
+1 eclipse:eclipse. The patch built with eclipse:eclipse.
+1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9) warnings.
+1 release audit. The applied patch does not increase the total number of release audit warnings.
+1 core tests. The patch passed unit tests in .
+1 contrib tests. The patch passed contrib unit tests.
Test results: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/1953//testReport/
Console output: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/1953//console
This message is automatically generated.