incubator-couchdb-user mailing list archives

Joan,
Thanks for the advice. In some initial tests across only 3 design docs
I'm seeing a noticeable speed increase. Granted with the workload I'm
talking about we're dealing with times measured in fractions of
milliseconds, but it's enough of an improvement to know it's a proper
way to take advantage of the concurrency / parallelism on the server
side as well.
Thank you,
Cameron
On 08/06/2013 04:29 PM, Joan Touzet wrote:
> Hi Cameron,
>
> When you split up your views, did you split them into separate view
> groups (different design docs) or keep them in the same view group
> (ddoc)? If not, try to do so. This should lead to additional couchjs
> processes being spawned for each view group, which (if your server can
> support it) will allow increased parallelism.
>
> That said, given your data set, I would be inclined to implicate your
> libevent code. If you can attempt to reproduce the situation using
> something "lower tech" such as multiple shell scripts + curl, this might
> be more useful information.
>
> On Tue, Aug 06, 2013 at 11:50:24AM -0400, Cameron Jacobson wrote:
>> Hi,
>>
>> I apologize if this has already reached a list. I tried sending
>> previously, but based on the online archives it doesn't appear that it
>> was received by the list.
>>
>>
>> Original Message:
>> -----------------------------------
>>
>> I've got a question regarding how CouchDB handles concurrency in views.
>>
>> Background: I'm writing a CouchDB client where I'm trying to leverage
>> libevent so the client can handle requests concurrently.
>>
>> When I started to benchmark the "async" functionality I noticed that no
>> matter what I did the time-spent in the closure where all the buffers
>> were being retrieved was growing linearly. My expectation, given that
>> this is concurrent accesses, would be that time would at worst grow
>> logarithmically (at least while the server had enough resources to
>> handle the requests concurrently). No noticeable issues in other
>> critical areas, such as I/O traffic thresholds, CPU or memory bottlenecks.
>>
>> I made several adjustments in the default.ini to ensure there were
>> enough resources allocated to the server. I tried the "stale"
>> parameter, and "warmed" the views with pre-test queries. Thinking it
>> was possible the view handling was the bottleneck I broke out my queries
>> against views1-10, all doing exactly the same thing. Nothing seemed to
>> help. This got me thinking that I may be implementing the libevent code
>> incorrectly. I realized this wasn't the issue once I ran the code
>> against a remote environment (non-couchdb) and was getting no noticeable
>> increase in time over as many as 100 concurrent requests (tests on
>> couchdb until now have not exceeded 50 concurrent requests).
>>
>> I don't have the background in CouchDB to know that it's the bottleneck,
>> or how I would go about proving this, but based on all the tests so far
>> I think I've narrowed in on CouchDB as the cause. I'm running version
>> 1.3.0.
>>
>> I came across the following link, which may explain part of the problem,
>> but I don't think it explains why multiple concurrent queries across
>> multiple views showed the same performance characteristics as multiple
>> concurrent queries against a single view. Also the comments in the
>> following thread are talking about large documents / views. My tests
>> were against 10 "cloned" views which all only contained 2 documents,
>> each of which was less than 1kb in size:
>> http://comments.gmane.org/gmane.comp.db.couchdb.user/22200
>>
>> Any insights into how I can tweak my configuration or what measures I
>> can take to improve this would be greatly appreciated.
>>
>> Thanks