James,
Yes, it sounds like the JEDI thing and the partitioning approach is what I
need. And yes, I'm talking queues for the most part. For the partitioning,
is that something that AMQ, or are you talking about me having a layer in
front of AMQ that would do this. In this case, instead of having one big
network of brokers, I would have several smaller networks of brokers?
Marc
James.Strachan wrote:
>
> On 05/12/2007, Marc Zampetti wrote:
>> James,
>>
>> You are right about the HA comment I originally made. I was referring to
>> fact that I'm not looking for persistent messages. But I am concerned
>> about
>> what happens when a broker fails and being able to recover from that
>> quickly, even if that means losing messages.
>
> Master/Slave is all about replicating messages to avoid message loss -
> it comes at a heavy performance cost. If you can't afford that cost,
> you don't need master/slave. i.e. the normal failover of the JMS
> client will do fine.
>
>
>> I understand the point about the need for more information. Here is what
>> I
>> say so far.
>>
>> There will be around 30 - 50 processes producing messages, each one
>> producing the same type of messages. Think it of a farm of processes
>> doing
>> the same basic work, spread out to handle processing load, scalability,
>> fault tolerance, etc. These processes know nothing about what happens to
>> these messages once they send them.
>>
>> On the consuming side, there are multiple farms of processes, each farm
>> doing a particular type of processing on some subset of the messages.
>> There
>> is nothing in the message, other than a unique id, that can be used to
>> determine which messages should be sent to each farm. Each of these farms
>> would have anywhere from 2 - 30 processes, depending upon the
>> functionality
>> and load requirements.
>
> So we're talking queues right? And by the sounds of it you could
> partition messages across multiple brokers for scalability. (Sounds
> like a good use case for the JEDI transport that still needs
> completing BTW to stripe message flows across different brokers to
> avoid the downsides of the store and forward network of brokers).
>
>
>> For the AMQ topologies, I was thinking of a Virtual Topic that all of the
>> publishers send messages to, with queues behind that go to some sort of
>> routing layer that in turns sends the message to the queue that the
>> service
>> the relevant consumer farm.
>
> Yeah. I guess it depends on how complex the routing is. It might be
> easier to just have an input queue, then some consumer process (which
> could be hosted inside the broker if need be) to do the routing;
> whether its Camel or some custom JMS code or whatever.
>
> --
> James
> -------
> http://macstrac.blogspot.com/
>
> Open Source Integration
> http://open.iona.com
>
>
--
View this message in context: http://www.nabble.com/Questions-on-Network-of-Brokers-and-high-message-rates-tf4941283s2354.html#a14174653
Sent from the ActiveMQ - User mailing list archive at Nabble.com.