OK, your changes in that revision to
opennms-webapp/src/main/webapp/WEB-INF/dispatcher-servlet.xml are
identical to mine that I committed to 1.10 except that your edits have
some double- and triple-duplicated lines for the
AnnotationMethodHandlerAdapter and DefaultAnnotationHandlerMapping bean
definitions for some reason. My changes were pushed to sf.net yesterday,
not sure if Ben has had a chance to merge to 1.11 (master) yet.
I'll hold off on any further changes until we can coordinate but I don't
think we've stepped on each others' toes too badly so far. :) Looks like
good work! I'm looking forward to see some of these controllers get
simplified with annotations.
-- Seth
On 07/28/2011 07:21 PM, DJ Gregor wrote:
> Hey Seth,
>
> Did you push your changes to sf.net? Into which branch? I pushed Melania's support for controller annotations to master on sf.net on 7/22 along with the conversion of element/service.jsp (commit 3b378424bcb15de58359669ce09fde51ce91152f). I want to make sure we aren't stepping on each other's toes and that we try to keep things synced between different branches except when it doesn't make sense. Also, since Melania is converting JSPs and servlets to use Hibernate, I want to make sure we aren't stepping on each others toes there, either.
>
> On Jul 28, 2011, at 5:03 PM, Seth Leger <seth@...> wrote:
>
>
>> Hi Melania,
>>
>> I happened to be looking at the Spring annotation-based servlets today. I committed a change that should allow us to start using @Controller and @RequestMapping attributes to map Spring servlets into the webapp. The only examples in our current codebase of this are very simple cases:
>>
>> opennms-webapp/src/main/java/org/opennms/web/controller/FrontPageController.java
>> opennms-webapp/src/main/java/org/opennms/web/controller/CategoryStatusController.java
>>
>> Now that the dispatcher-servlet.xml context has been updated to support the @RequestMapping annotations, it should be easy to start moving legacy Spring MVC servlets over to the new annotation-based conventions. The first priority for me is to move all of the servlets that inherit from org.springframework.web.servlet.mvc.SimpleFormController since it was deprecated in Spring 3.0 specifically in favor of using annotations for your controllers.
>>
>> Please let us know on this opennms-devel list if you have any questions about this work! It will be great to move the code over to annotations: it will make the controller code simpler and much easier to read and maintain. Thanks!
>>
>> Seth Leger
>> The OpenNMS Group
>>
>>
>> On 07/28/2011 01:42 PM, Melania Galea wrote:
>>
>>> I am Melania. My google summer of code target is to finish converting the
>>> OpenNMS GUI pages to Hibernate. For the UI layer refactoring process, the
>>> SpringMVC web framework was chosen, which due to its underlying MVC design
>>> pattern offers the possibility to build flexible and loosely coupled
>>> applications. My mentor and I have managed to create a design for the
>>> controllers which are meant to process the requests and display a response.
>>> The controller classes are using annotations and autowired dependency
>>> injection which leaves out the burden of the xml configuration. They are
>>> also easily to be unit tested directly. All the logic regarding the requests
>>> is contained by the controllers since now a Hibernate session is
>>> automatically opened for all web requests. There will be no need for a
>>> separate service layer that handles the database queries. I have built a
>>> wiki page which will be more explanatory regarding the above topics.
>>>
>>> http://opennms.org/wiki/Annotation_Driven_Controllers
>>>
>>>
>>>
>>> ------------------------------------------------------------------------------
>>> Got Input? Slashdot Needs You.
>>> Take our quick survey online. Come on, we don't ask for help often.
>>> Plus, you'll get a chance to win $100 to spend on ThinkGeek.
>>> http://p.sf.net/sfu/slashdot-survey
>>>
>>> _______________________________________________
>>> Please read the OpenNMS Mailing List FAQ:
>>> http://www.opennms.org/index.php/Mailing_List_FAQ
>>>
>>> opennms-devel mailing list
>>>
>>> To *unsubscribe* or change your subscription options, see the bottom of this page:
>>> https://lists.sourceforge.net/lists/listinfo/opennms-devel
>>>
>> ------------------------------------------------------------------------------
>> Got Input? Slashdot Needs You.
>> Take our quick survey online. Come on, we don't ask for help often.
>> Plus, you'll get a chance to win $100 to spend on ThinkGeek.
>> http://p.sf.net/sfu/slashdot-survey
>> _______________________________________________
>> Please read the OpenNMS Mailing List FAQ:
>> http://www.opennms.org/index.php/Mailing_List_FAQ
>>
>> opennms-devel mailing list
>>
>> To *unsubscribe* or change your subscription options, see the bottom of this page:
>> https://lists.sourceforge.net/lists/listinfo/opennms-devel
>>
>
>
>
> ------------------------------------------------------------------------------
> Got Input? Slashdot Needs You.
> Take our quick survey online. Come on, we don't ask for help often.
> Plus, you'll get a chance to win $100 to spend on ThinkGeek.
> http://p.sf.net/sfu/slashdot-survey
>
>
> _______________________________________________
> Please read the OpenNMS Mailing List FAQ:
> http://www.opennms.org/index.php/Mailing_List_FAQ
>
> opennms-devel mailing list
>
> To *unsubscribe* or change your subscription options, see the bottom of this page:
> https://lists.sourceforge.net/lists/listinfo/opennms-devel

Hey Seth,
Did you push your changes to sf.net? Into which branch? I pushed Melania's support for controller annotations to master on sf.net on 7/22 along with the conversion of element/service.jsp (commit 3b378424bcb15de58359669ce09fde51ce91152f). I want to make sure we aren't stepping on each other's toes and that we try to keep things synced between different branches except when it doesn't make sense. Also, since Melania is converting JSPs and servlets to use Hibernate, I want to make sure we aren't stepping on each others toes there, either.
On Jul 28, 2011, at 5:03 PM, Seth Leger <seth@...> wrote:
> Hi Melania,
>
> I happened to be looking at the Spring annotation-based servlets today. I committed a change that should allow us to start using @Controller and @RequestMapping attributes to map Spring servlets into the webapp. The only examples in our current codebase of this are very simple cases:
>
> opennms-webapp/src/main/java/org/opennms/web/controller/FrontPageController.java
> opennms-webapp/src/main/java/org/opennms/web/controller/CategoryStatusController.java
>
> Now that the dispatcher-servlet.xml context has been updated to support the @RequestMapping annotations, it should be easy to start moving legacy Spring MVC servlets over to the new annotation-based conventions. The first priority for me is to move all of the servlets that inherit from org.springframework.web.servlet.mvc.SimpleFormController since it was deprecated in Spring 3.0 specifically in favor of using annotations for your controllers.
>
> Please let us know on this opennms-devel list if you have any questions about this work! It will be great to move the code over to annotations: it will make the controller code simpler and much easier to read and maintain. Thanks!
>
> Seth Leger
> The OpenNMS Group
>
>
> On 07/28/2011 01:42 PM, Melania Galea wrote:
>>
>> I am Melania. My google summer of code target is to finish converting the
>> OpenNMS GUI pages to Hibernate. For the UI layer refactoring process, the
>> SpringMVC web framework was chosen, which due to its underlying MVC design
>> pattern offers the possibility to build flexible and loosely coupled
>> applications. My mentor and I have managed to create a design for the
>> controllers which are meant to process the requests and display a response.
>> The controller classes are using annotations and autowired dependency
>> injection which leaves out the burden of the xml configuration. They are
>> also easily to be unit tested directly. All the logic regarding the requests
>> is contained by the controllers since now a Hibernate session is
>> automatically opened for all web requests. There will be no need for a
>> separate service layer that handles the database queries. I have built a
>> wiki page which will be more explanatory regarding the above topics.
>>
>> http://opennms.org/wiki/Annotation_Driven_Controllers
>>
>>
>>
>> ------------------------------------------------------------------------------
>> Got Input? Slashdot Needs You.
>> Take our quick survey online. Come on, we don't ask for help often.
>> Plus, you'll get a chance to win $100 to spend on ThinkGeek.
>> http://p.sf.net/sfu/slashdot-survey
>>
>> _______________________________________________
>> Please read the OpenNMS Mailing List FAQ:
>> http://www.opennms.org/index.php/Mailing_List_FAQ
>>
>> opennms-devel mailing list
>>
>> To *unsubscribe* or change your subscription options, see the bottom of this page:
>> https://lists.sourceforge.net/lists/listinfo/opennms-devel
>
> ------------------------------------------------------------------------------
> Got Input? Slashdot Needs You.
> Take our quick survey online. Come on, we don't ask for help often.
> Plus, you'll get a chance to win $100 to spend on ThinkGeek.
> http://p.sf.net/sfu/slashdot-survey
> _______________________________________________
> Please read the OpenNMS Mailing List FAQ:
> http://www.opennms.org/index.php/Mailing_List_FAQ
>
> opennms-devel mailing list
>
> To *unsubscribe* or change your subscription options, see the bottom of this page:
> https://lists.sourceforge.net/lists/listinfo/opennms-devel

Hi Melania,
I happened to be looking at the Spring annotation-based servlets today.
I committed a change that should allow us to start using @Controller and
@RequestMapping attributes to map Spring servlets into the webapp. The
only examples in our current codebase of this are very simple cases:
opennms-webapp/src/main/java/org/opennms/web/controller/FrontPageController.java
opennms-webapp/src/main/java/org/opennms/web/controller/CategoryStatusController.java
Now that the dispatcher-servlet.xml context has been updated to support
the @RequestMapping annotations, it should be easy to start moving
legacy Spring MVC servlets over to the new annotation-based conventions.
The first priority for me is to move all of the servlets that inherit
from org.springframework.web.servlet.mvc.SimpleFormController since it
was deprecated in Spring 3.0 specifically in favor of using annotations
for your controllers.
Please let us know on this opennms-devel list if you have any questions
about this work! It will be great to move the code over to annotations:
it will make the controller code simpler and much easier to read and
maintain. Thanks!
Seth Leger
The OpenNMS Group
On 07/28/2011 01:42 PM, Melania Galea wrote:
> I am Melania. My google summer of code target is to finish converting the
> OpenNMS GUI pages to Hibernate. For the UI layer refactoring process, the
> SpringMVC web framework was chosen, which due to its underlying MVC design
> pattern offers the possibility to build flexible and loosely coupled
> applications. My mentor and I have managed to create a design for the
> controllers which are meant to process the requests and display a response.
> The controller classes are using annotations and autowired dependency
> injection which leaves out the burden of the xml configuration. They are
> also easily to be unit tested directly. All the logic regarding the requests
> is contained by the controllers since now a Hibernate session is
> automatically opened for all web requests. There will be no need for a
> separate service layer that handles the database queries. I have built a
> wiki page which will be more explanatory regarding the above topics.
>
> http://opennms.org/wiki/Annotation_Driven_Controllers
>
>
>
>
> ------------------------------------------------------------------------------
> Got Input? Slashdot Needs You.
> Take our quick survey online. Come on, we don't ask for help often.
> Plus, you'll get a chance to win $100 to spend on ThinkGeek.
> http://p.sf.net/sfu/slashdot-survey
>
>
> _______________________________________________
> Please read the OpenNMS Mailing List FAQ:
> http://www.opennms.org/index.php/Mailing_List_FAQ
>
> opennms-devel mailing list
>
> To *unsubscribe* or change your subscription options, see the bottom of this page:
> https://lists.sourceforge.net/lists/listinfo/opennms-devel

I am Melania. My google summer of code target is to finish converting the
OpenNMS GUI pages to Hibernate. For the UI layer refactoring process, the
SpringMVC web framework was chosen, which due to its underlying MVC design
pattern offers the possibility to build flexible and loosely coupled
applications. My mentor and I have managed to create a design for the
controllers which are meant to process the requests and display a response.
The controller classes are using annotations and autowired dependency
injection which leaves out the burden of the xml configuration. They are
also easily to be unit tested directly. All the logic regarding the requests
is contained by the controllers since now a Hibernate session is
automatically opened for all web requests. There will be no need for a
separate service layer that handles the database queries. I have built a
wiki page which will be more explanatory regarding the above topics.
http://opennms.org/wiki/Annotation_Driven_Controllers

I happened to be checking a thread dump from my opennms and noticed that 9 out of 10 of the QueueingRrdStrategy-X threads were parking on the same lock. This got me to thinking, does it make sense if your using org.jrobin.core.RrdBackendFactory=MNIO to increase the number of threads since that particular jrobin backend storage method has its own locking?
Here's what the parked threads looked like with the exception being that the RrdDb.store() lock was unique for each thread.
"QueuingRrdStrategy-2" prio=3 tid=0x00000001069f6800 nid=0xed waiting on condition [0xfffffffe476fe000]
java.lang.Thread.State: WAITING (parking)
at sun.misc.Unsafe.park(Native Method)
- parking to wait for <0xfffffffebd497ce8> (a java.util.concurrent.locks.ReentrantReadWriteLock$NonfairSync)
at java.util.concurrent.locks.LockSupport.park(LockSupport.java:158)
at java.util.concurrent.locks.AbstractQueuedSynchronizer.parkAndCheckInterrupt(AbstractQueuedSynchronizer.java:811)
at java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireQueued(AbstractQueuedSynchronizer.java:842)
at java.util.concurrent.locks.AbstractQueuedSynchronizer.acquire(AbstractQueuedSynchronizer.java:1178)
at java.util.concurrent.locks.ReentrantReadWriteLock$WriteLock.lock(ReentrantReadWriteLock.java:807)
at org.jrobin.core.RrdNioByteBufferBackend.write(RrdNioByteBufferBackend.java:93)
at org.jrobin.core.RrdBackend.writeDouble(RrdBackend.java:180)
at org.jrobin.core.RrdPrimitive.writeDouble(RrdPrimitive.java:87)
at org.jrobin.core.RrdDouble.set(RrdDouble.java:43)
at org.jrobin.core.Datasource.accumulate(Datasource.java:268)
at org.jrobin.core.Datasource.process(Datasource.java:204)
at org.jrobin.core.RrdDb.store(RrdDb.java:600)
- locked <0xfffffffe8a93c5b0> (a org.jrobin.core.RrdDb)
at org.jrobin.core.Sample.update(Sample.java:221)
at org.jrobin.core.Sample.setAndUpdate(Sample.java:243)
at org.opennms.netmgt.rrd.jrobin.JRobinRrdStrategy.updateFile(JRobinRrdStrategy.java:219)
at org.opennms.netmgt.rrd.jrobin.JRobinRrdStrategy.updateFile(JRobinRrdStrategy.java:80)
at org.opennms.netmgt.rrd.QueuingRrdStrategy$UpdateOperation.process(QueuingRrdStrategy.java:520)
at org.opennms.netmgt.rrd.QueuingRrdStrategy.processPendingOperations(QueuingRrdStrategy.java:1139)
at org.opennms.netmgt.rrd.QueuingRrdStrategy.run(QueuingRrdStrategy.java:1094)
at java.lang.Thread.run(Thread.java:662)
Ron
This e-mail message is being sent solely for use by the intended recipient(s) and may contain confidential information. Any unauthorized review, use, disclosure or distribution is prohibited. If you are not the intended recipient, please contact the sender by phone or reply by e-mail, delete the original message and destroy all copies. Thank you.

I've had a patch for this soak testing for some time on our problematic test server. It's still leaking a few handles, but it's not crashed in a week or so now which seems to be a substantial improvement so I'm going to take this as a confirmation that I've at least fixed *something*.
I couldn't quite get it down to a single NioSocketConnector as connection-timeout is a property of the connector, not the connection, so it now pools Connectors per timeout value and disposes them with a reference count to active connections.
I filed bug NMS-4846 with details and attached the patch there.
Thanks,
Duncan Mackintosh (dijm)
________________________________________
From: Duncan Mackintosh [DMackintosh@...]
Sent: 06 July 2011 13:20
To: opennms-devel@...
Subject: [opennms-devel] Provisiond, "Too many open files" and Mina
I've been doing a lot of digging around various 'Too many open files' crashes we've been seeing locally, and I think I've pinned down a big leak of file descriptors in provisiond's use of org.apache.mina connectors.
What it's currently doing in AsyncBasicDetector#isServiceDetected:
- For each service, create a new NioSocketConnector
- Configure that connector with a handler, filters etc
- Make a connection out, check for results etc
There seem to be two problems with this approach:
1) Constructing an NioSocketConnector creates a lot of 'anon_inode' and 'pipe' file descriptors - on one machine it was 8 & 12 respectiovely and on another 4/8, so I'm not sure quite what the difference is there (under linux, at least; I assume some equivalent under Windows). The actual connect() call only uses one more handle. This causes it to run out of descriptors a lot faster than expected.
2) If new NioSocketConnector() crashes due to a "Too many open files" exception, Mina sometimes just sort of falls over dead with "NoClassDefFoundError: Could not initialize class sun.nio.ch.FileDispatcher". This class does exist in my JVM (openjdk 6) and if I reflectively inspect it first, it sometimes stops the crashes happening. I'm pretty baffled there, to be honest. If it does get itself into this state, you can't close existing sockets, you can't open new ones; all the anon_inode and pipe FDs just sit there. This seems to tally with behaviour we've witnessed in opennms instances where we've had a Too many open files crash - lsof shows a few thousand pipe/anon_inode handles just sitting around long after the crash.
For reference, I've attached a simple test class that just opens ~60 connections using the current methodology. If you lsof the process while it pauses, you can see how many new file descriptors are being created each time; if you drop the 60 down to 50 it cleans up gracefully but at 60 it doesn't seem possible to free the descriptors again (you'll need mina-core and slf4j-log4j12 in a project to run it). I'd be quite interested to see if others get the same behaviour I do.
What I think Mina wants you to be doing is creating a single NioSocketConnector to reuse everywhere and using the optional IoSessionInitializer in .connect() to configure filters and attach state objects to the IoSession. This would take a moderate overhaul of AsyncBasicDetector, as the handler would need to be rewritten to be a singleton that takes some state using IoSession.get/setAttribute rather than having one handler per service detect attempt and probably a fair chunk of refactoring at the same time.
Before I embark on making those changes I wanted to throw this out there for comment, and to see if there's already a refactor of this code planned (I couldn't see any changes on the provisiond-refactor branch yet).
Thanks,
Duncan Mackintosh (dijm)
Cambridge Broadband Networks Limited Registered in England and Wales under company number: 03879840 Registered office: Selwyn House, Cambridge Business Park, Cowley Road, Cambridge CB4 0WZ, UK. VAT number: GB 741 0186 64
Cambridge Broadband Networks Limited Registered in England and Wales under company number: 03879840 Registered office: Selwyn House, Cambridge Business Park, Cowley Road, Cambridge CB4 0WZ, UK. VAT number: GB 741 0186 64

Just take a look at this issue...
http://issues.opennms.org/browse/NMS-4631
Antonio
Il giorno 06/lug/2011, alle ore 20.03, Alex Bennee ha scritto:
> On 6 July 2011 16:19, Matt Brozowski <brozow@...> wrote:
>> There is a property in opennms.properties that allows you to set the
>> max number of async connections it can do..
>>
>> Have you tried setting that?
>>
>> org.opennms.netmgt.provision.maxConcurrentConnector
>
> Yes, we doing that to mitigate the effects of this. However I think
> the point is provisiond's usage of the library is
> broken (or at least fundamentally inefficient). It means OpenNMS is
> using up far more FDs that needed for each provision thread
> and with the default of 2000 provision instances it's quite easy to kill.
>
>>
>> Matt
>>
>>
>>
>> On Wed, Jul 6, 2011 at 8:20 AM, Duncan Mackintosh <DMackintosh@...> wrote:
>>> I've been doing a lot of digging around various 'Too many open files' crashes we've been seeing locally, and I think I've pinned down a big leak of file descriptors in provisiond's use of org.apache.mina connectors.
>>>
>>> What it's currently doing in AsyncBasicDetector#isServiceDetected:
>>> - For each service, create a new NioSocketConnector
>>> - Configure that connector with a handler, filters etc
>>> - Make a connection out, check for results etc
>>>
>>> There seem to be two problems with this approach:
>>>
>>> 1) Constructing an NioSocketConnector creates a lot of 'anon_inode' and 'pipe' file descriptors - on one machine it was 8 & 12 respectiovely and on another 4/8, so I'm not sure quite what the difference is there (under linux, at least; I assume some equivalent under Windows). The actual connect() call only uses one more handle. This causes it to run out of descriptors a lot faster than expected.
>>>
>>> 2) If new NioSocketConnector() crashes due to a "Too many open files" exception, Mina sometimes just sort of falls over dead with "NoClassDefFoundError: Could not initialize class sun.nio.ch.FileDispatcher". This class does exist in my JVM (openjdk 6) and if I reflectively inspect it first, it sometimes stops the crashes happening. I'm pretty baffled there, to be honest. If it does get itself into this state, you can't close existing sockets, you can't open new ones; all the anon_inode and pipe FDs just sit there. This seems to tally with behaviour we've witnessed in opennms instances where we've had a Too many open files crash - lsof shows a few thousand pipe/anon_inode handles just sitting around long after the crash.
>>>
>>> For reference, I've attached a simple test class that just opens ~60 connections using the current methodology. If you lsof the process while it pauses, you can see how many new file descriptors are being created each time; if you drop the 60 down to 50 it cleans up gracefully but at 60 it doesn't seem possible to free the descriptors again (you'll need mina-core and slf4j-log4j12 in a project to run it). I'd be quite interested to see if others get the same behaviour I do.
>>>
>>> What I think Mina wants you to be doing is creating a single NioSocketConnector to reuse everywhere and using the optional IoSessionInitializer in .connect() to configure filters and attach state objects to the IoSession. This would take a moderate overhaul of AsyncBasicDetector, as the handler would need to be rewritten to be a singleton that takes some state using IoSession.get/setAttribute rather than having one handler per service detect attempt and probably a fair chunk of refactoring at the same time.
>>>
>>> Before I embark on making those changes I wanted to throw this out there for comment, and to see if there's already a refactor of this code planned (I couldn't see any changes on the provisiond-refactor branch yet).
>>>
>>> Thanks,
>>> Duncan Mackintosh (dijm)
>>> Cambridge Broadband Networks Limited Registered in England and Wales under company number: 03879840 Registered office: Selwyn House, Cambridge Business Park, Cowley Road, Cambridge CB4 0WZ, UK. VAT number: GB 741 0186 64
>>>
>>> ------------------------------------------------------------------------------
>>> All of the data generated in your IT infrastructure is seriously valuable.
>>> Why? It contains a definitive record of application performance, security
>>> threats, fraudulent activity, and more. Splunk takes this data and makes
>>> sense of it. IT sense. And common sense.
>>> http://p.sf.net/sfu/splunk-d2d-c2
>>> _______________________________________________
>>> Please read the OpenNMS Mailing List FAQ:
>>> http://www.opennms.org/index.php/Mailing_List_FAQ
>>>
>>> opennms-devel mailing list
>>>
>>> To *unsubscribe* or change your subscription options, see the bottom of this page:
>>> https://lists.sourceforge.net/lists/listinfo/opennms-devel
>>>
>>
>> ------------------------------------------------------------------------------
>> All of the data generated in your IT infrastructure is seriously valuable.
>> Why? It contains a definitive record of application performance, security
>> threats, fraudulent activity, and more. Splunk takes this data and makes
>> sense of it. IT sense. And common sense.
>> http://p.sf.net/sfu/splunk-d2d-c2
>> _______________________________________________
>> Please read the OpenNMS Mailing List FAQ:
>> http://www.opennms.org/index.php/Mailing_List_FAQ
>>
>> opennms-devel mailing list
>>
>> To *unsubscribe* or change your subscription options, see the bottom of this page:
>> https://lists.sourceforge.net/lists/listinfo/opennms-devel
>>
>
>
>
> --
> Alex, homepage: http://www.bennee.com/~alex/
> http://www.half-llama.co.uk
>
> ------------------------------------------------------------------------------
> All of the data generated in your IT infrastructure is seriously valuable.
> Why? It contains a definitive record of application performance, security
> threats, fraudulent activity, and more. Splunk takes this data and makes
> sense of it. IT sense. And common sense.
> http://p.sf.net/sfu/splunk-d2d-c2
> _______________________________________________
> Please read the OpenNMS Mailing List FAQ:
> http://www.opennms.org/index.php/Mailing_List_FAQ
>
> opennms-devel mailing list
>
> To *unsubscribe* or change your subscription options, see the bottom of this page:
> https://lists.sourceforge.net/lists/listinfo/opennms-devel

Just committed this change to the 1.10 branch, it should get merged to
master soon. I diff'd the target directory of opennms-base-assembly with
both versions of the plugin and found no differences, in addition to
grepping through the code for escaped tokens like \${something}. I
didn't see any occurrences in our configs or script code.
-- Seth
On 07/05/2011 11:45 AM, Roskens, Ronald wrote:
> I'm fairly certain you can also bump maven-resources-plugin from 2.3 to 2.5 even though the comments in opennms/pom.xml say there's major escaping issues beyond 2.3. As best I can tell all the filtered files now using the ${...} syntax instead of @...@ like they used to so the comment doesn't seem to apply any more. This will get rid of the builds errors about m2e needing a newer version of maven-resources-plugin.
>

On 6 July 2011 16:19, Matt Brozowski <brozow@...> wrote:
> There is a property in opennms.properties that allows you to set the
> max number of async connections it can do..
>
> Have you tried setting that?
>
> org.opennms.netmgt.provision.maxConcurrentConnector
Yes, we doing that to mitigate the effects of this. However I think
the point is provisiond's usage of the library is
broken (or at least fundamentally inefficient). It means OpenNMS is
using up far more FDs that needed for each provision thread
and with the default of 2000 provision instances it's quite easy to kill.
>
> Matt
>
>
>
> On Wed, Jul 6, 2011 at 8:20 AM, Duncan Mackintosh <DMackintosh@...> wrote:
>> I've been doing a lot of digging around various 'Too many open files' crashes we've been seeing locally, and I think I've pinned down a big leak of file descriptors in provisiond's use of org.apache.mina connectors.
>>
>> What it's currently doing in AsyncBasicDetector#isServiceDetected:
>> - For each service, create a new NioSocketConnector
>> - Configure that connector with a handler, filters etc
>> - Make a connection out, check for results etc
>>
>> There seem to be two problems with this approach:
>>
>> 1) Constructing an NioSocketConnector creates a lot of 'anon_inode' and 'pipe' file descriptors - on one machine it was 8 & 12 respectiovely and on another 4/8, so I'm not sure quite what the difference is there (under linux, at least; I assume some equivalent under Windows). The actual connect() call only uses one more handle. This causes it to run out of descriptors a lot faster than expected.
>>
>> 2) If new NioSocketConnector() crashes due to a "Too many open files" exception, Mina sometimes just sort of falls over dead with "NoClassDefFoundError: Could not initialize class sun.nio.ch.FileDispatcher". This class does exist in my JVM (openjdk 6) and if I reflectively inspect it first, it sometimes stops the crashes happening. I'm pretty baffled there, to be honest. If it does get itself into this state, you can't close existing sockets, you can't open new ones; all the anon_inode and pipe FDs just sit there. This seems to tally with behaviour we've witnessed in opennms instances where we've had a Too many open files crash - lsof shows a few thousand pipe/anon_inode handles just sitting around long after the crash.
>>
>> For reference, I've attached a simple test class that just opens ~60 connections using the current methodology. If you lsof the process while it pauses, you can see how many new file descriptors are being created each time; if you drop the 60 down to 50 it cleans up gracefully but at 60 it doesn't seem possible to free the descriptors again (you'll need mina-core and slf4j-log4j12 in a project to run it). I'd be quite interested to see if others get the same behaviour I do.
>>
>> What I think Mina wants you to be doing is creating a single NioSocketConnector to reuse everywhere and using the optional IoSessionInitializer in .connect() to configure filters and attach state objects to the IoSession. This would take a moderate overhaul of AsyncBasicDetector, as the handler would need to be rewritten to be a singleton that takes some state using IoSession.get/setAttribute rather than having one handler per service detect attempt and probably a fair chunk of refactoring at the same time.
>>
>> Before I embark on making those changes I wanted to throw this out there for comment, and to see if there's already a refactor of this code planned (I couldn't see any changes on the provisiond-refactor branch yet).
>>
>> Thanks,
>> Duncan Mackintosh (dijm)
>> Cambridge Broadband Networks Limited Registered in England and Wales under company number: 03879840 Registered office: Selwyn House, Cambridge Business Park, Cowley Road, Cambridge CB4 0WZ, UK. VAT number: GB 741 0186 64
>>
>> ------------------------------------------------------------------------------
>> All of the data generated in your IT infrastructure is seriously valuable.
>> Why? It contains a definitive record of application performance, security
>> threats, fraudulent activity, and more. Splunk takes this data and makes
>> sense of it. IT sense. And common sense.
>> http://p.sf.net/sfu/splunk-d2d-c2
>> _______________________________________________
>> Please read the OpenNMS Mailing List FAQ:
>> http://www.opennms.org/index.php/Mailing_List_FAQ
>>
>> opennms-devel mailing list
>>
>> To *unsubscribe* or change your subscription options, see the bottom of this page:
>> https://lists.sourceforge.net/lists/listinfo/opennms-devel
>>
>
> ------------------------------------------------------------------------------
> All of the data generated in your IT infrastructure is seriously valuable.
> Why? It contains a definitive record of application performance, security
> threats, fraudulent activity, and more. Splunk takes this data and makes
> sense of it. IT sense. And common sense.
> http://p.sf.net/sfu/splunk-d2d-c2
> _______________________________________________
> Please read the OpenNMS Mailing List FAQ:
> http://www.opennms.org/index.php/Mailing_List_FAQ
>
> opennms-devel mailing list
>
> To *unsubscribe* or change your subscription options, see the bottom of this page:
> https://lists.sourceforge.net/lists/listinfo/opennms-devel
>
--
Alex, homepage: http://www.bennee.com/~alex/http://www.half-llama.co.uk

There is a property in opennms.properties that allows you to set the
max number of async connections it can do..
Have you tried setting that?
org.opennms.netmgt.provision.maxConcurrentConnector
Matt
On Wed, Jul 6, 2011 at 8:20 AM, Duncan Mackintosh <DMackintosh@...> wrote:
> I've been doing a lot of digging around various 'Too many open files' crashes we've been seeing locally, and I think I've pinned down a big leak of file descriptors in provisiond's use of org.apache.mina connectors.
>
> What it's currently doing in AsyncBasicDetector#isServiceDetected:
> - For each service, create a new NioSocketConnector
> - Configure that connector with a handler, filters etc
> - Make a connection out, check for results etc
>
> There seem to be two problems with this approach:
>
> 1) Constructing an NioSocketConnector creates a lot of 'anon_inode' and 'pipe' file descriptors - on one machine it was 8 & 12 respectiovely and on another 4/8, so I'm not sure quite what the difference is there (under linux, at least; I assume some equivalent under Windows). The actual connect() call only uses one more handle. This causes it to run out of descriptors a lot faster than expected.
>
> 2) If new NioSocketConnector() crashes due to a "Too many open files" exception, Mina sometimes just sort of falls over dead with "NoClassDefFoundError: Could not initialize class sun.nio.ch.FileDispatcher". This class does exist in my JVM (openjdk 6) and if I reflectively inspect it first, it sometimes stops the crashes happening. I'm pretty baffled there, to be honest. If it does get itself into this state, you can't close existing sockets, you can't open new ones; all the anon_inode and pipe FDs just sit there. This seems to tally with behaviour we've witnessed in opennms instances where we've had a Too many open files crash - lsof shows a few thousand pipe/anon_inode handles just sitting around long after the crash.
>
> For reference, I've attached a simple test class that just opens ~60 connections using the current methodology. If you lsof the process while it pauses, you can see how many new file descriptors are being created each time; if you drop the 60 down to 50 it cleans up gracefully but at 60 it doesn't seem possible to free the descriptors again (you'll need mina-core and slf4j-log4j12 in a project to run it). I'd be quite interested to see if others get the same behaviour I do.
>
> What I think Mina wants you to be doing is creating a single NioSocketConnector to reuse everywhere and using the optional IoSessionInitializer in .connect() to configure filters and attach state objects to the IoSession. This would take a moderate overhaul of AsyncBasicDetector, as the handler would need to be rewritten to be a singleton that takes some state using IoSession.get/setAttribute rather than having one handler per service detect attempt and probably a fair chunk of refactoring at the same time.
>
> Before I embark on making those changes I wanted to throw this out there for comment, and to see if there's already a refactor of this code planned (I couldn't see any changes on the provisiond-refactor branch yet).
>
> Thanks,
> Duncan Mackintosh (dijm)
> Cambridge Broadband Networks Limited Registered in England and Wales under company number: 03879840 Registered office: Selwyn House, Cambridge Business Park, Cowley Road, Cambridge CB4 0WZ, UK. VAT number: GB 741 0186 64
>
> ------------------------------------------------------------------------------
> All of the data generated in your IT infrastructure is seriously valuable.
> Why? It contains a definitive record of application performance, security
> threats, fraudulent activity, and more. Splunk takes this data and makes
> sense of it. IT sense. And common sense.
> http://p.sf.net/sfu/splunk-d2d-c2
> _______________________________________________
> Please read the OpenNMS Mailing List FAQ:
> http://www.opennms.org/index.php/Mailing_List_FAQ
>
> opennms-devel mailing list
>
> To *unsubscribe* or change your subscription options, see the bottom of this page:
> https://lists.sourceforge.net/lists/listinfo/opennms-devel
>

I've been doing a lot of digging around various 'Too many open files' crashes we've been seeing locally, and I think I've pinned down a big leak of file descriptors in provisiond's use of org.apache.mina connectors.
What it's currently doing in AsyncBasicDetector#isServiceDetected:
- For each service, create a new NioSocketConnector
- Configure that connector with a handler, filters etc
- Make a connection out, check for results etc
There seem to be two problems with this approach:
1) Constructing an NioSocketConnector creates a lot of 'anon_inode' and 'pipe' file descriptors - on one machine it was 8 & 12 respectiovely and on another 4/8, so I'm not sure quite what the difference is there (under linux, at least; I assume some equivalent under Windows). The actual connect() call only uses one more handle. This causes it to run out of descriptors a lot faster than expected.
2) If new NioSocketConnector() crashes due to a "Too many open files" exception, Mina sometimes just sort of falls over dead with "NoClassDefFoundError: Could not initialize class sun.nio.ch.FileDispatcher". This class does exist in my JVM (openjdk 6) and if I reflectively inspect it first, it sometimes stops the crashes happening. I'm pretty baffled there, to be honest. If it does get itself into this state, you can't close existing sockets, you can't open new ones; all the anon_inode and pipe FDs just sit there. This seems to tally with behaviour we've witnessed in opennms instances where we've had a Too many open files crash - lsof shows a few thousand pipe/anon_inode handles just sitting around long after the crash.
For reference, I've attached a simple test class that just opens ~60 connections using the current methodology. If you lsof the process while it pauses, you can see how many new file descriptors are being created each time; if you drop the 60 down to 50 it cleans up gracefully but at 60 it doesn't seem possible to free the descriptors again (you'll need mina-core and slf4j-log4j12 in a project to run it). I'd be quite interested to see if others get the same behaviour I do.
What I think Mina wants you to be doing is creating a single NioSocketConnector to reuse everywhere and using the optional IoSessionInitializer in .connect() to configure filters and attach state objects to the IoSession. This would take a moderate overhaul of AsyncBasicDetector, as the handler would need to be rewritten to be a singleton that takes some state using IoSession.get/setAttribute rather than having one handler per service detect attempt and probably a fair chunk of refactoring at the same time.
Before I embark on making those changes I wanted to throw this out there for comment, and to see if there's already a refactor of this code planned (I couldn't see any changes on the provisiond-refactor branch yet).
Thanks,
Duncan Mackintosh (dijm)
Cambridge Broadband Networks Limited Registered in England and Wales under company number: 03879840 Registered office: Selwyn House, Cambridge Business Park, Cowley Road, Cambridge CB4 0WZ, UK. VAT number: GB 741 0186 64

I've spent some over the past week or two also working on building opennms inside eclipse. I think you'll have better luck building the completed project using compile.pl and assemble.pl outside of eclipse than inside it. Just install ActiveState perl on your system, and you'll have no problems running the build process.
Here's my notes on issues I've found building the project inside eclipse:
There is a plugin you can add to the project to add the generated-sources directories to the build path. See http://issues.opennms.org/browse/NMS-4814. This will solve most of the can't compile due to missing sources.
I'm fairly certain you can also bump maven-resources-plugin from 2.3 to 2.5 even though the comments in opennms/pom.xml say there's major escaping issues beyond 2.3. As best I can tell all the filtered files now using the ${...} syntax instead of @...@ like they used to so the comment doesn't seem to apply any more. This will get rid of the builds errors about m2e needing a newer version of maven-resources-plugin.
Next, there are a couple of plugins that opennms uses that eclipse & m2e don't know what to do with. You'll need to add a lifecycle-mapping plugin so eclipse knows how to handle those specific plugins. It is pretty simple since eclipse suggests a quick-fix of adding an "ignore" action which you can then switch to execute so it will still execute those plugins. Adding these will cause the command line build to throw warnings about these unknown plugins though.
There are two projects that will still fail to build, opennms-tools/jira-troubleticketer and opennms-tools/quickbase-troubleticketer. Both have dependencies on external jars that are not available via maven. It is probably easier to not import the two projects in when you do the initial import existing maven projects.
At this point Eclipse is telling me I have 158 errors and 1457 warnings in the problems window. If I close the jira-troubleticketer and quickbase-troubleticketer projects, I'm down to 5 errors and 1457 warnings.
1. Eclipse is correct about though, features/isoc-ipv6-gui/src/main/webapp/WEB-INF/web.xml is invalid xml. The welcome-file-list section should be after the servlet-mapping section.
2. opennms-web-api/src/test/java/org/opennms/web/org/opennms/web/api/CommandBeanMockup.java - java package name wrong?
3. opennms-web-api/src/test/java/org/opennms/web/org/opennms/web/api/WebSecurityUtilsTest.java - java package name wrong?
4. features/remote-poller-jnlp/pom.xml (x2) - can add the ignore lifecycle-mapping for webstart-maven-plugin, but execute mapping has build issues inside eclipse (due to customized opennms version of the plugin?)
Ron
-----Original Message-----
From: thotasridevi [mailto:thotasri_17@...]
Sent: Tuesday, July 05, 2011 4:54 AM
To: opennms-devel@...
Subject: Re: [opennms-devel] Problem With Building in Eclipse-Windows
Hi,
Are you able to build opennms with eclipse on windows .I am expecting by this time you might completed .If so can you please guide me I am also getting build errors missing buildpath eventhough after following all the instructions from wikki page.Can you reply to thotasri_17@...
thanks
sridevi
--
View this message in context: http://opennms.530661.n2.nabble.com/Re-Problem-With-Building-in-Eclipse-Windows-tp5351086p6549078.html
Sent from the OpenNMS - devel mailing list archive at Nabble.com.
------------------------------------------------------------------------------
All of the data generated in your IT infrastructure is seriously valuable.
Why? It contains a definitive record of application performance, security threats, fraudulent activity, and more. Splunk takes this data and makes sense of it. IT sense. And common sense.
http://p.sf.net/sfu/splunk-d2d-c2
_______________________________________________
Please read the OpenNMS Mailing List FAQ:
http://www.opennms.org/index.php/Mailing_List_FAQ
opennms-devel mailing list
To *unsubscribe* or change your subscription options, see the bottom of this page:
https://lists.sourceforge.net/lists/listinfo/opennms-devel
This e-mail message is being sent solely for use by the intended recipient(s) and may contain confidential information. Any unauthorized review, use, disclosure or distribution is prohibited. If you are not the intended recipient, please contact the sender by phone or reply by e-mail, delete the original message and destroy all copies. Thank you.