From dev-return-13737-apmail-activemq-dev-archive=activemq.apache.org@activemq.apache.org Mon Dec 15 10:29:33 2008
Return-Path:
Delivered-To: apmail-activemq-dev-archive@www.apache.org
Received: (qmail 54345 invoked from network); 15 Dec 2008 10:29:33 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.2)
by minotaur.apache.org with SMTP; 15 Dec 2008 10:29:33 -0000
Received: (qmail 77066 invoked by uid 500); 15 Dec 2008 10:29:46 -0000
Delivered-To: apmail-activemq-dev-archive@activemq.apache.org
Received: (qmail 77033 invoked by uid 500); 15 Dec 2008 10:29:45 -0000
Mailing-List: contact dev-help@activemq.apache.org; run by ezmlm
Precedence: bulk
List-Help:
List-Unsubscribe:
List-Post:
List-Id:
Reply-To: dev@activemq.apache.org
Delivered-To: mailing list dev@activemq.apache.org
Received: (qmail 77022 invoked by uid 99); 15 Dec 2008 10:29:45 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 15 Dec 2008 02:29:45 -0800
X-ASF-Spam-Status: No, hits=-1999.8 required=10.0
tests=ALL_TRUSTED,WHOIS_MYPRIVREG
X-Spam-Check-By: apache.org
Received: from [140.211.11.140] (HELO brutus.apache.org) (140.211.11.140)
by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 15 Dec 2008 10:29:25 +0000
Received: from brutus (localhost [127.0.0.1])
by brutus.apache.org (Postfix) with ESMTP id 8FB9B234C3DA
for ; Mon, 15 Dec 2008 02:29:05 -0800 (PST)
Message-ID: <1126054910.1229336945587.JavaMail.jira@brutus>
Date: Mon, 15 Dec 2008 02:29:05 -0800 (PST)
From: "Maurits Lucas (JIRA)"
To: dev@activemq.apache.org
Subject: [jira] Commented: (AMQ-2009) Problem with message dispatch after a
while
In-Reply-To: <1503857571.1227225545425.JavaMail.jira@brutus>
MIME-Version: 1.0
Content-Type: text/plain; charset=UTF-8
Content-Transfer-Encoding: 7bit
X-Virus-Checked: Checked by ClamAV on apache.org
[ https://issues.apache.org/activemq/browse/AMQ-2009?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=48177#action_48177 ]
Maurits Lucas commented on AMQ-2009:
------------------------------------
We have been able to reproduce this issue in our test runs every time.
The only problem, and the reason I haven't got a unit test for you, is that it takes a very long time to rear its ugly head.
The minimum setup which exposes the issue is have a producer (P1) send a lot (6 million in our case) messages to a topic (t1) and have one consumer (C1) consume the messages from the topic and publish them to a queue (q1).
P1 --> [t1] --> C1 --> [q1]
Both P1 and C1 produce and consume / produce messages as fast as they can. Because P1 produces faster than C1 can consume, we get a fast producer / slow consumer for that topic.
At the end of the run, there are 6 million messages on the queue, but if you add a consumer to the queue, it won't consume any messages. If you browse the queue using the webconsole, it appears empty. Only restarting ActiveMQ gets q1 working again.
In our previous tests, we discovered that this "freezing" of dispatching (it affects other queues than q1 as well) starts somewhere during the publishing of the 6 million messages to t1.
We are using a standalone broker, Kaha for persistence and Spring 2.5.4 for JMS support.
OS: Linux (2.6.18-92.1.18.el5 #1 SMP Wed Nov 12 09:19:49 EST 2008 x86_64 x86_64 x86_64 GNU/Linux)
Java:
java version "1.6.0_10"
Java(TM) SE Runtime Environment (build 1.6.0_10-b33)
Java HotSpot(TM) 64-Bit Server VM (build 11.0-b15, mixed mode)
We have tested with 5.1.0, 5.2.0 and 5.3-SNAPSHOT. Using 5.1.0 and the snapshot we ran into the issue, with 5.2.0 we ran into AMQ-2020 and so didn't test any further.
We saw some peculiar values for the inflight counter on t1, this leveled out at 32766 for hours on end, sometimes going down to 32765 for a couple of seconds. Not sure if that is significant, but I am including a screenshot of JConsole just in case.
> Problem with message dispatch after a while
> -------------------------------------------
>
> Key: AMQ-2009
> URL: https://issues.apache.org/activemq/browse/AMQ-2009
> Project: ActiveMQ
> Issue Type: Bug
> Components: Broker
> Affects Versions: 5.1.0, 5.2.0
> Reporter: Rajani Chennamaneni
> Priority: Blocker
> Attachments: DispatchMultipleConsumersTest.java
>
>
> Messages are not getting dispatched after a while (although it accepts new incoming messages) until restart of the broker. This problem is described in several posts.
> http://www.nabble.com/Pending-Messages-are-shown-in-ActiveMQ-td20241332.html
> http://www.nabble.com/Consumer-Listener-stop-receving-message-until-ActiveMQ-restart-td20355247.html
> http://www.nabble.com/Stuck-messages---Dispatch-issues-td20467949.html
> There was also an issue opened in Spring project for this thinking it was Spring problem.
> http://jira.springframework.org/browse/SPR-5110
> I am not able to reproduce with Junit test case having BrokerService started with in the test case. I guess I am not hitting the right stress conditions this way. But when I run the test case against an externally running ActiveMQ instance backed with oracle database persistence, it is reproducible most of the times. This is not a every time failure situation, it takes more time once than the other.
> I was able to hit this situation of stuck messages on queue using following scenario most of the times:
> 1) Start 2 concurrent consumers for the queue using Spring's DefaultMessageListenerContainer using cacheLevelName as CACHE_CONSUMER
> 2) Send messages using JMETER 2.3.2 to the queue on ActiveMQ stand alone broker instance with 50 threads looping 20 times.
> 3) After a while, you will notice that Spring logs that no messages are being received but the messages are shown jconsole of ActiveMQ and the database backing it for persistence.
> But in 5.2 RC3, the problem is that it dispatches duplicate messages and does not remove them from broker's database after acknowledge properly.
> Attached test case might help to reproduce when run against externally running stand alone ActiveMQ broker. Another way to see the problem is that try to load test using JMETER by sending messages to a queue with a camel route that moves messages from this queue to another and you will notice that it stops moving after while or copied duplicates in case of 5.2 RC3.
> Sorry about such a huge description but it is a real problem! A different team at our company are having this issue in production with 5.1. They are using it as an embedded broker with derby for persistence.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.