From dev-return-40847-apmail-directory-dev-archive=directory.apache.org@directory.apache.org Tue May 1 13:06:12 2012
Return-Path:
X-Original-To: apmail-directory-dev-archive@www.apache.org
Delivered-To: apmail-directory-dev-archive@www.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
by minotaur.apache.org (Postfix) with SMTP id C43BB9B55
for ; Tue, 1 May 2012 13:06:12 +0000 (UTC)
Received: (qmail 62874 invoked by uid 500); 1 May 2012 13:06:12 -0000
Delivered-To: apmail-directory-dev-archive@directory.apache.org
Received: (qmail 62814 invoked by uid 500); 1 May 2012 13:06:12 -0000
Mailing-List: contact dev-help@directory.apache.org; run by ezmlm
Precedence: bulk
List-Help:
List-Unsubscribe:
List-Post:
List-Id:
Reply-To: "Apache Directory Developers List"
Delivered-To: mailing list dev@directory.apache.org
Received: (qmail 62800 invoked by uid 99); 1 May 2012 13:06:12 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 01 May 2012 13:06:12 +0000
X-ASF-Spam-Status: No, hits=1.5 required=5.0
tests=HTML_MESSAGE,RCVD_IN_DNSWL_LOW,SPF_PASS
X-Spam-Check-By: apache.org
Received-SPF: pass (nike.apache.org: domain of akarasulu@gmail.com designates 209.85.212.172 as permitted sender)
Received: from [209.85.212.172] (HELO mail-wi0-f172.google.com) (209.85.212.172)
by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 01 May 2012 13:06:04 +0000
Received: by wibhj6 with SMTP id hj6so3711172wib.1
for ; Tue, 01 May 2012 06:05:44 -0700 (PDT)
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed;
d=gmail.com; s=20120113;
h=mime-version:sender:in-reply-to:references:date
:x-google-sender-auth:message-id:subject:from:to:content-type;
bh=iNrqqhUfll4JoEGCvUqUfntLrZVYM+IK1JWATGm9Usg=;
b=hZWsgHvunb5xVxNtZ4duWSthSx1QAR1nFM9e3Yr5EBb3fR3xM55XInPi0iJUkDasw3
PCDsG4J0C0uBL2EBjFLPiY0kED0g7NIelevns+/mafxE03p9miwbohxzEpM9AnlfpHwF
0l5k+dA7HscLs8q4YyrZOpEUB4pgMOFvFGV0g1nogyP4RcgTK4jwBQZfVzX6Jl99gDR6
OJS3LSEC/Ug/9fzFkLL2nQQIdrFyo+JbjXR0yIfxmkxSvvCVuv8ku6xoBh4kgeJ39OdS
8yjJ9WI5Q8UkjNb2elK0XCwU4XWR1A42GQ4rjdrCrasDctABN4o9t1UVueIotUmZEpPh
cQDQ==
MIME-Version: 1.0
Received: by 10.180.100.230 with SMTP id fb6mr5910645wib.3.1335877543871; Tue,
01 May 2012 06:05:43 -0700 (PDT)
Sender: akarasulu@gmail.com
Received: by 10.180.126.97 with HTTP; Tue, 1 May 2012 06:05:43 -0700 (PDT)
In-Reply-To: <4F9F3795.2040907@gmail.com>
References: <4F9F3795.2040907@gmail.com>
Date: Tue, 1 May 2012 16:05:43 +0300
X-Google-Sender-Auth: Xq8svhHUuHI1M9sI6cHpZptEAIs
Message-ID:
Subject: Re: Inex branch merged into trunk...
From: Alex Karasulu
To: Apache Directory Developers List , elecharny@apache.org
Content-Type: multipart/alternative; boundary=f46d041824ee7745aa04bef93a63
--f46d041824ee7745aa04bef93a63
Content-Type: text/plain; charset=ISO-8859-1
Content-Transfer-Encoding: quoted-printable
On Tue, May 1, 2012 at 4:08 AM, Emmanuel L=E9charny wr=
ote:
> Hi,
>
> just to inform you that the index branch has been merged with no harm
> today. I just had to fix 3 conflicts, and two bugs I introduced in the
> branch before the commit.
>
> The server performance is way better for searches, with a few improvement=
s
> I did those last 4 days. It was impressive how easy it was to improve the
> speed with little modifications. The global result is that the server is
> now :
> o Object scope search (lookup) : 49 880 req/s compared to 23 081 on the
> previous trunk
> o One Level scope search (5 entries returned) : 68 715 entries returned
> per second, compared to 33 120/s
> o Sub Level scope search (10 entries returned ) : 70 830 entries returned
> per second, compared to 18 910/s
>
>
This is great work Emmanuel. Nicely done!
> There is room for more improvement, but it will be more complex. The area
> that can be improved are :
> o get rid of the extra getSearchControls() call in intercepotrs. This is
> the easiest fix
> o review the way we handle entries modification before we return them.
> Currently, we clone the entry, and remove the attributes the user has not
> required. See DIRSERVER-1719 for more explaination on this subject. Note
> that the filtering of attributes represent around 9% of the global CPU ti=
me.
> o getting back the ID from a Dn is a very costly operation (19% of the
> global CPU time), and the longer the DN, the longer the operation. For ea=
ch
> RDN, we have to do a lookup in the RdnIndex. The only solution would be t=
o
> have a Dn -> ID cache somewhere. This would boost the server performance,
> that's for sure.
> o fetching an entry from the backend cost 38% of the global time, out of
> which 29% represent the cost to clone the entry. If we could avoid doing
> this clone (see upper), we may have some major performances increase.
> o when evaluating an entry to see if it fits the filter, we use the
> reverseIndex, which is also a costly operation. We shoudl re-evaluate if =
it
> wouldn't be better to use the MatchingRules comparator to do that instead
> (reverse lookups account for 4% of the used CPU time)
>
>
I guess we have these in JIRA?
> One interesting result is that the LRUCache.get() operation represent 13%
> of the used time. This is definitively not small. There is probably some
> room for some improvement here, but this is way more complex...
>
> All those numbers have been collected using YourKit on a Lookup test (150
> 000 lookups on one single element have been done)
>
>
I wonder what the over the network stats are with a client machine separate
from the server machine. Oh and with multiple clients. It's too bad we
never got a chance to setup such an environment :( .
>
> There are also some improvements to expect on the Add/Delete/Move
> operation, as we have to delete/add the keys on the RdnIndex. This is
> something Im going to work on tomorrow.
>
>
Cool.
>
> One more thing : the number I get when running the server-integ search
> perf are way below (from 2900 to 5400 per second). This is plain normal.
> When going through th network, we pay some extra price :
> o the client code eats 57% of all the time it takes to run the test
> o On the server, normalizing the incoming Dn costs 7% of the processing
> time
> o the entries encoding is very expensive
>
> All in all, on the server, unless we test it on a different machine than
> the injectors, all the measures are pretty impossible to do. There is too
> much noise...
>
> I'd be interested to conduct largest tests on a multi-core server, with
> lots of memory, and a lot of entries, with external injectors, to see wha=
t
> kind of performances we can get...
>
>
Ditto.
>
> In the next few days, I will probably fix some pending bugs. I think we
> can cut a M7 release by the end of this week, and make it available by ne=
xt
> week.
>
>
That sounds great. Thanks!
--=20
Best Regards,
-- Alex
--f46d041824ee7745aa04bef93a63
Content-Type: text/html; charset=ISO-8859-1
Content-Transfer-Encoding: quoted-printable

just to inform you that the index branch has been merged with no harm today=
. I just had to fix 3 conflicts, and two bugs I introduced in the branch be=
fore the commit.

The server performance is way better for searches, with a few improvements =
I did those last 4 days. It was impressive how easy it was to improve the s=
peed with little modifications. The global result is that the server is now=
:
o Object scope search (lookup) : 49 880 req/s compared to 23 081 on the pre=
vious trunk
o One Level scope search (5 entries returned) : 68 715 entries returned per=
second, compared to 33 120/s
o Sub Level scope search (10 entries returned ) : 70 830 entries returned p=
er second, compared to 18 910/s

This is great work Emmanuel. Nicely do=
ne!

=A0

There is room for more improvement, but it will be more complex. The area t=
hat can be improved are :
o get rid of the extra getSearchControls() call in intercepotrs. This is th=
e easiest fix
o review the way we handle entries modification before we return them. Curr=
ently, we clone the entry, and remove the attributes the user has not requi=
red. See DIRSERVER-1719 for more explaination on this subject. Note that th=
e filtering of attributes represent around 9% of the global CPU time.
o getting back the ID from a Dn is a very costly operation (19% of the glob=
al CPU time), and the longer the DN, the longer the operation. For each RDN=
, we have to do a lookup in the RdnIndex. The only solution would be to hav=
e a Dn -> ID cache somewhere. This would boost the server performance, t=
hat's for sure.
o fetching an entry from the backend cost 38% of the global time, out of wh=
ich 29% represent the cost to clone the entry. If we could avoid doing this=
clone (see upper), we may have some major performances increase.
o when evaluating an entry to see if it fits the filter, we use the reverse=
Index, which is also a costly operation. We shoudl re-evaluate if it wouldn=
't be better to use the MatchingRules comparator to do that instead (re=
verse lookups account for 4% of the used CPU time)

I guess we have these in JIRA?

=A0

One interesting result is that the LRUCache.get() operation represent 13% o=
f the used time. This is definitively not small. There is probably some roo=
m for some improvement here, but this is way more complex...

All those numbers have been collected using YourKit on a Lookup test (150 0=
00 lookups on one single element have been done)

I wonder what the over the network sta=
ts are with a client machine separate from the server machine. Oh and with =
multiple clients. It's too bad we never got a chance to setup such an e=
nvironment :( .

=A0

There are also some improvements to expect on the Add/Delete/Move operation=
, as we have to delete/add the keys on the RdnIndex. This is something Im g=
oing to work on tomorrow.

Cool.

=A0

One more thing : the number I get when running the server-integ search perf=
are way below (from 2900 to 5400 per second). This is plain normal. When g=
oing through th network, we pay some extra price :
o the client code eats 57% of all the time it takes to run the test
o On the server, normalizing the incoming Dn costs 7% of the processing tim=
e
o the entries encoding is very expensive

All in all, on the server, unless we test it on a different machine than th=
e injectors, all the measures are pretty impossible to do. There is too muc=
h noise...

I'd be interested to conduct largest tests on a multi-core server, with=
lots of memory, and a lot of entries, with external injectors, to see what=
kind of performances we can get...

Ditto.

=A0

In the next few days, I will probably fix some pending bugs. I think we can=
cut a M7 release by the end of this week, and make it available by next we=
ek.