Sunday, November 6, 2011

When we are talking with customers about Oracle ADF performance, very often we can hear such question - "Hey, ADF works well when there are few concurrent users, how it behaves when there is more serious load?". In order to answer this question I will publish series of posts, where we will study different parameters for AM pool tuning and test AM pool configuration under stress loads. Today we will see how stable is AM pool passivation/activation mechanism.

Stress tests are implemented with JMeter. Check Chris Muir blog about JMeter configuration for ADF 11g. I'm using JMeter configuration file from Chris blog, however original file is updated with internal loops to repeat session requests. JMeter allows to run stress test with multiple parallel users, each users starts its own HTTP session. It is not enough for stress test, additionally I would like to repeat user action within the same HTTP session - its why internal loops are needed, these loops allow to iterate through the same requests multiple times.

Download sample application for stress test - stresstest_v1.zip. Sample contains two applications - main and application library. Both applications implement two AM modules, pool settings for both AM's are configured to the same values. There are two pages inside main application, first page brings data from local AM, second from imported region and related AM:

Second page integrates ADF region from ADF library:

Stress test is designed to run such scenarion, where user clicks on Submit button from ADF UI, this button calls custom method from AM interface. Custom AM method is accessing VO row, by randomized value in range based on VO rowset size. Numeric attribute value is changed (incremented by 1) for every accessed row, transaction is committed or reverted back, depending on concurrent modification. Additionally we have one transient attribute, it is set to be passivated:

All 3 stress test scenarios (Optimistic, Average and Pessimistic) are executed with the same AM pool configuration (for both AM's) - based on rough estimate of approximately 20 concurrent users in the system:

- Initial Pool Size = 22 (number of AM instances created on first ever access, recommended to be 10% more than estimated number of concurrent users)

- Referenced Pool Size = 20 (number of active AM instances, recommended to be the same as estimated number of concurrent users - to avoid frequent passivation/activation)

- Minimum Available Size = 0 (when set to 0, allows to release all AM instances after idle timeout, this helps to release reserved database connections as well. Is set to 0 for tests in stress environment, in your system you may set it to higher value, but less than referenced pool size)

- Idle Instance Timeout = 300 (AM instance is considered inactive after 5 minutes of inactivity, this if for stress test. In your system you would set it something close to Web session timeout - to prevent frequent passivation/activation)

- Pool Pooling Interval = 120 (AM pool is cleaned every 2 minutes)

1. Optimistic (15 concurrent users, 300 transactions in 10 minutes)

This test is performed on first ever access. As you can see, 22 AM instances are created, as per initial pool size setting. However, soon (idle time 5 mins + pool cleaning interval 2 mins) 7 AM instances are removed - because we have only 15 online users. Passivation/activation never happens (because referenced pool is set to be for 20, and we have only 15). MainModule statistics:

Same for HrModule module:

Very important, because Minimum Available Size = 0, after idle time entire pool is cleaned:

Same with DB connections, there is no need to wait for AM time to live, AM instances and DB connections can be released earlier - however this will trigger passivation. DB connection can be released before AM time to live, by tuning AM pool Minimum Available Size and AM Instance Timeout:

With 80 users, AM pool starts to passivate/activate much more (as expected). There are 20 active AM's, as per Referenced Pool Size and those 20 AM's need to serve all 80 users. There are lots of switch activity happening - around 1500 passivation/activation circles in 1 minute:

Runtime statistics for the second AM:

During strong stress test, we can see that we were using at some moments more than 40 (20 per each AM) database connections (as previous test). Means AM pool during heavy stress test may maintain more active AM instances than specified by Referenced Pool Size - but still, AM size will not exceed Maximum Pool Size setting (30 in our case):

Conclusion: AM Pool is strong enough to handle stress peaks with online users greatly exceeding number of AM Pool Size and and AM Referenced Pool Size. However, in your system you should try to maintain AM Pool sizing as close as possible to expected number of online users - this will help to avoid performance costly passivation/activation operations.

18 comments:

Can I ask a question on the "AM Maximum Pool Size" setting. Up to now I thought that was a hard limit that when hit, the framework will throw errors. Is that your understanding?

An alternative to that last question, can you see any reason not to leave the Maximum Pool Size as the default 4096 value?

Also in the last Pessimistic scenario, against diagram 3c.png you comment "we can see that we were using at some moments more than 40 (20 per each AM) database connections". If that's the case, in diagrams relating to the Pessimistic scenario, shouldn't we see the Active Instances grow above 20 to reflect the growth in the pool size?

"AM Maximum Pool Size" - based on stress test, this setting is not a hard limit. If "AM Maximum Pool Size" is set to smaller value, framework just executes slightly more passivation/activation, but still serves all users (however number of passivation/activation mostly depends on "Referenced Pool Size"). As per doc, "AM Maximum Pool Size" should be 30% more than expected concurrent number of users, it looks like there is no need to keep 4096 - at least is not giving any visible performance effect to keep "AM Maximum Pool Size" large. However I will continue tests on this.

At the same time, I don't see so far any reason not to leave "AM Maximum Pool Size" = 4096, unless AM Pool is wasting memory when this value is set to be large - will need to test memory consumption.

For Pessimistic scenario - yes, its what I was thinking as well. Why we don't see more than 20 Active Instances. I guess, during stress test conditions, AM Pool maintains some AM instances in transition state (moving from Active to Passive). May be its not able to switch AM instance fast enough, its why it consumes slightly more DB connections.

- "AM Maximum Pool Size" is not hard limit. However, on very high load and when WLS is running out of free resources - AM pool expands and creates more active AM instances, then specified by Referenced AM Pool size. It keeps growing until "AM Maximum Pool Size" and then it stops serving new users. However, this happens only with really heavy stress test - usually AM pool tries to passivate/activate fast and stay in limits of Referenced AM Pool size - rarely growing higher and reaching Max size.

- We saw more DB connections on high load, because of internal AM connection for passivation/activation. I declared jbo.server.internal_connection for both AM's and pointed to separate dedicated Data Source - now number of consumed connections stays stable.

I'd like to ask you 2 things:1. We have nested AMs under one RootAM. Is it enough to set these settings just to the RootAM? Or do we need to set it to all AMs. 2. We've encountering problems about max-user-connection. And would like to use doconnectionpooling=true (with disconnect_level=1). But when we set it like that, we get an java.lang.reflect.InvocationTargetException error. It throws when our App try to execute VOImpl method which locates under one of the child AM.

I set concurrent users and number of loops in JMeter. Then I calculate based on concurrent users and number of loops how many transactions it will execute (you can see this from JMeter log file as well). Time is approximate - it takes around 10 minutes for JMeter to execute each of the tests.

Hi Andrejus,I have a question, does every page in adf application takes a connection from pool? if the system has 10 pages and the concurrent users is 10 then i must have 100 connection ? i'm asking also what to do if the same user closes his session and open new session, is he considered a new user?/

I already enabled application module pooling. when i saw the number of taken connections when monitoring the server, each page i entered take a connection form the pool. could you please help me if i had to do another configurtions ?

Hi Andrejus,the error was from me,In my test when i made a request to the page i removed the adf-ctrl-state parameter so it is considered as new session .The whole Application takes on connection only. sorry Again for misunderstanding