Sharon Barr (Inactive)
added a comment - 23/Apr/13 12:40 PM I assume this relates to the tcmalloc bug fix we submitted for large objects.
I am raising this to a blocker for 2.0.2, to make sure it is on the radar and prioritized accordingly.
We have some major customers that might encounter this issue.

Maria McDuff (Inactive)
added a comment - 23/Apr/13 2:48 PM mike, can you check if the tcmalloc fix is already checked-in? sharon thinks it is... is it just a matter of merging the fix into the 2.0.2 branch?
if so, can we go ahead and merge the fix?
thanks.

<para>
If you continuously perform numerous appends to a document,
it may lead to memory fragmentation and overuse. This is due to an underlying
issue of inefficient memory allocation and deallocation. If you know you will
perform numerous small 1K appends totaling 5MB,
you can mitigate this problem by performing one larger append of about 1 MB and then
perform your numerous, smaller 1K appends. You can use a similar approach for other appends, depending on your total, aggregate size of appends..
</para>

kzeller
added a comment - 21/May/13 2:41 PM Added to 2.0.2 RN as Known issue. Workaround proposed by Chiyoung. Commenting out until release:
<rnentry type="knownissue">
<version ver="2.0.0m"/>
<class id="db"/>
<issue type="cb" ref=" MB-7887 "/>
<rntext>
<para>
If you continuously perform numerous appends to a document,
it may lead to memory fragmentation and overuse. This is due to an underlying
issue of inefficient memory allocation and deallocation. If you know you will
perform numerous small 1K appends totaling 5MB,
you can mitigate this problem by performing one larger append of about 1 MB and then
perform your numerous, smaller 1K appends. You can use a similar approach for other appends, depending on your total, aggregate size of appends..
</para>
</rntext>
</rnentry>

<para>
If you continuously perform numerous appends to a document,
it may lead to memory fragmentation and overuse. This is due to an underlying
issue of inefficient memory allocation and deallocation with third party
software <literal>tcmalloc</literal>. This has been fixed.
</para>

kzeller
added a comment - 30/Jul/13 4:47 PM Cross-ref to separate doc tag: http://www.couchbase.com/issues/browse/MB-8691
Added to 2.2 RN:
<rnentry type="fix">
<version ver="2.2.0a"/>
<class id="db"/>
<issue type="cb" ref=" MB-7887 "/>
<rntext>
<para>
If you continuously perform numerous appends to a document,
it may lead to memory fragmentation and overuse. This is due to an underlying
issue of inefficient memory allocation and deallocation with third party
software <literal>tcmalloc</literal>. This has been fixed.
</para>
</rntext>
</rnentry>

I added a version of tcmalloc into grommit that contains all of the latest changes that are in the Couchbase tcmalloc fork. The source tarball has been uploaded under the filename gperftools-2.0.1.tar.gz.

Mike Wiederhold
added a comment - 01/Aug/13 3:12 PM I added a version of tcmalloc into grommit that contains all of the latest changes that are in the Couchbase tcmalloc fork. The source tarball has been uploaded under the filename gperftools-2.0.1.tar.gz.

my results:
description:
6 node cluster, default bucket, 2M items, 200K of them with append operation (1024 Bytes). Finally those 200K items will be ~10000Bytes. Test duration is 3.7 hours
2.1.1.-764:
In 10.3.121.112 bucket default, total_fragmentation_bytes + the total_allocated_bytes = 1364344768
In 10.3.121.112 bucket default, mem_used = 1202357848
In 10.3.121.112 bucket default, the difference between acutal memory used by memcached and mem_used is 1.13472438365 times
In 10.3.121.113 bucket default, total_fragmentation_bytes + the total_allocated_bytes = 1362452448
In 10.3.121.113 bucket default, mem_used = 1199415544
In 10.3.121.113 bucket default, the difference between acutal memory used by memcached and mem_used is 1.13593029106 times
In 10.3.121.114 bucket default, total_fragmentation_bytes + the total_allocated_bytes = 1360363424
In 10.3.121.114 bucket default, mem_used = 1196221528
In 10.3.121.114 bucket default, the difference between acutal memory used by memcached and mem_used is 1.13721697207 times
In 10.3.121.115 bucket default, total_fragmentation_bytes + the total_allocated_bytes = 1364107232
In 10.3.121.115 bucket default, mem_used = 1200871256
In 10.3.121.115 bucket default, the difference between acutal memory used by memcached and mem_used is 1.13593128754 times
In 10.3.121.116 bucket default, total_fragmentation_bytes + the total_allocated_bytes = 1354063840
In 10.3.121.116 bucket default, mem_used = 1191618776
In 10.3.121.116 bucket default, the difference between acutal memory used by memcached and mem_used is 1.13632301477 times
In 10.3.121.117 bucket default, total_fragmentation_bytes + the total_allocated_bytes = 1356414912
In 10.3.121.117 bucket default, mem_used = 1194777720
In 10.3.121.117 bucket default, the difference between acutal memory used by memcached and mem_used is 1.13528641294 times

2.2.0-746:
In 10.3.121.112 bucket default, total_fragmentation_bytes + the total_allocated_bytes = 1363624000
In 10.3.121.112 bucket default, mem_used = 1203659784
In 10.3.121.112 bucket default, the difference between acutal memory used by memcached and mem_used is 1.13289819775 times
In 10.3.121.113 bucket default, total_fragmentation_bytes + the total_allocated_bytes = 1360993728
In 10.3.121.113 bucket default, mem_used = 1198378296
In 10.3.121.113 bucket default, the difference between acutal memory used by memcached and mem_used is 1.13569624262 times
In 10.3.121.114 bucket default, total_fragmentation_bytes + the total_allocated_bytes = 1356701600
In 10.3.121.114 bucket default, mem_used = 1194089384
In 10.3.121.114 bucket default, the difference between acutal memory used by memcached and mem_used is 1.13618094104 times
In 10.3.121.115 bucket default, total_fragmentation_bytes + the total_allocated_bytes = 1363165152
In 10.3.121.115 bucket default, mem_used = 1199148440
In 10.3.121.115 bucket default, the difference between acutal memory used by memcached and mem_used is 1.13677765532 times
In 10.3.121.116 bucket default, total_fragmentation_bytes + the total_allocated_bytes = 1352851424
In 10.3.121.116 bucket default, mem_used = 1189074216
In 10.3.121.116 bucket default, the difference between acutal memory used by memcached and mem_used is 1.13773505959 times
In 10.3.121.117 bucket default, total_fragmentation_bytes + the total_allocated_bytes = 1356660704
In 10.3.121.117 bucket default, mem_used = 1192302088
In 10.3.121.117 bucket default, the difference between acutal memory used by memcached and mem_used is 1.13784980975 times

so, I don't see such improvements in my scenario for now and I'm planning to execute all cases that we have to compare results
will try to get the results over the weekend

Andrei Baranouski
added a comment - 16/Aug/13 8:49 AM Test Case/Version 2.1.1.-764 2.2.0-746
results from Chisheng: https://docs.google.com/spreadsheet/ccc?key=0Al5dV9XAr9l3dGFGNDAtQnNTOHQ3LVRLc1lyZXB2Rnc#gid=0
description:
2 node cluster, default bucket, 100000 items, 100 of them with append operation (128 Bytes). Finally those 100 items will be 5MB. Test duration is 4 hours
2.1.1.-764:
total_fragmentation_bytes + the total_allocated_bytes = 1160512064, mem_used = 575781688, the difference between acutal memory used by memcached and mem_used is 2.015
2.2.0-746:
total_fragmentation_bytes + the total_allocated_bytes = 910137600, mem_used = 537464960, the difference between acutal memory used by memcached and mem_used is 1.693
my results:
description:
6 node cluster, default bucket, 2M items, 200K of them with append operation (1024 Bytes). Finally those 200K items will be ~10000Bytes. Test duration is 3.7 hours
2.1.1.-764:
In 10.3.121.112 bucket default, total_fragmentation_bytes + the total_allocated_bytes = 1364344768
In 10.3.121.112 bucket default, mem_used = 1202357848
In 10.3.121.112 bucket default, the difference between acutal memory used by memcached and mem_used is 1.13472438365 times
In 10.3.121.113 bucket default, total_fragmentation_bytes + the total_allocated_bytes = 1362452448
In 10.3.121.113 bucket default, mem_used = 1199415544
In 10.3.121.113 bucket default, the difference between acutal memory used by memcached and mem_used is 1.13593029106 times
In 10.3.121.114 bucket default, total_fragmentation_bytes + the total_allocated_bytes = 1360363424
In 10.3.121.114 bucket default, mem_used = 1196221528
In 10.3.121.114 bucket default, the difference between acutal memory used by memcached and mem_used is 1.13721697207 times
In 10.3.121.115 bucket default, total_fragmentation_bytes + the total_allocated_bytes = 1364107232
In 10.3.121.115 bucket default, mem_used = 1200871256
In 10.3.121.115 bucket default, the difference between acutal memory used by memcached and mem_used is 1.13593128754 times
In 10.3.121.116 bucket default, total_fragmentation_bytes + the total_allocated_bytes = 1354063840
In 10.3.121.116 bucket default, mem_used = 1191618776
In 10.3.121.116 bucket default, the difference between acutal memory used by memcached and mem_used is 1.13632301477 times
In 10.3.121.117 bucket default, total_fragmentation_bytes + the total_allocated_bytes = 1356414912
In 10.3.121.117 bucket default, mem_used = 1194777720
In 10.3.121.117 bucket default, the difference between acutal memory used by memcached and mem_used is 1.13528641294 times
2.2.0-746:
In 10.3.121.112 bucket default, total_fragmentation_bytes + the total_allocated_bytes = 1363624000
In 10.3.121.112 bucket default, mem_used = 1203659784
In 10.3.121.112 bucket default, the difference between acutal memory used by memcached and mem_used is 1.13289819775 times
In 10.3.121.113 bucket default, total_fragmentation_bytes + the total_allocated_bytes = 1360993728
In 10.3.121.113 bucket default, mem_used = 1198378296
In 10.3.121.113 bucket default, the difference between acutal memory used by memcached and mem_used is 1.13569624262 times
In 10.3.121.114 bucket default, total_fragmentation_bytes + the total_allocated_bytes = 1356701600
In 10.3.121.114 bucket default, mem_used = 1194089384
In 10.3.121.114 bucket default, the difference between acutal memory used by memcached and mem_used is 1.13618094104 times
In 10.3.121.115 bucket default, total_fragmentation_bytes + the total_allocated_bytes = 1363165152
In 10.3.121.115 bucket default, mem_used = 1199148440
In 10.3.121.115 bucket default, the difference between acutal memory used by memcached and mem_used is 1.13677765532 times
In 10.3.121.116 bucket default, total_fragmentation_bytes + the total_allocated_bytes = 1352851424
In 10.3.121.116 bucket default, mem_used = 1189074216
In 10.3.121.116 bucket default, the difference between acutal memory used by memcached and mem_used is 1.13773505959 times
In 10.3.121.117 bucket default, total_fragmentation_bytes + the total_allocated_bytes = 1356660704
In 10.3.121.117 bucket default, mem_used = 1192302088
In 10.3.121.117 bucket default, the difference between acutal memory used by memcached and mem_used is 1.13784980975 times
so, I don't see such improvements in my scenario for now and I'm planning to execute all cases that we have to compare results
will try to get the results over the weekend

Maria McDuff (Inactive)
added a comment - 23/Dec/13 1:51 AM
From: Maria McDuff
Sent: Friday, December 20, 2013 4:50 PM
To: Mike Wiederhold
Cc: Wayne Siu; Anil Kumar; Ravi Mayuram
Subject: follow up with tcmalloc
Mike,
Can you pls let us know if the memory fragmentation bytes from Andrei's test result: https://www.couchbase.com/issues/browse/mb-7887 is acceptable?
There's no change between 2.2 and 2.5. This is concerning. There should be at least minimal improvement.
Can you re-confirm that the tcmalloc is in 2.5 build 1011 or when it got in?
Thanks,
Maria

select 15% of items(150K) and for them perform append key while their size doesn't reach 10000 bytes, like
value+2^3+2^4+2^5.....

at the end of the test we get stats mem_used, total_allocated_bytes and total_fragmentation_bytes on every node
and
the difference between actual memory used by memcached and mem_used through calculation:
(total_fragmentation_bytes_stats[server]) + int(total_allocated_bytes_stats[server]) / float(mem_used_stats[server])

test was run on two different cb versions with identical parameters and the same vms

Andrei Baranouski
added a comment - 08/Jan/14 11:28 AM the test_items_append was developed by Chisheng and performs the following steps
example for cases with parameters:
./testrunner -i cluster3nodes.ini -t memorysanitytests.MemorySanity.test_items_append,items=1000000,desired_item_size=10000,append_ratio=0.15,sasl_buckets=1,kv_verify=False
3 nodes in cluster
1 default and 1 sasl bucket with 1M items per bucket
select 15% of items(150K) and for them perform append key while their size doesn't reach 10000 bytes, like
value+2^3+2^4+2^5.....
at the end of the test we get stats mem_used, total_allocated_bytes and total_fragmentation_bytes on every node
and
the difference between actual memory used by memcached and mem_used through calculation:
(total_fragmentation_bytes_stats [server] ) + int(total_allocated_bytes_stats [server] ) / float(mem_used_stats [server] )
test was run on two different cb versions with identical parameters and the same vms

if compare the results against 2.5.0-994 & 2.2.0-821 we can see that the stats are identical and no improvement observed.

2.5.0-994
In 10.3.4.144 bucket default, the difference between actual memory used by memcached and mem_used is 2.03309838435 times
In 10.3.4.146 bucket default, the difference between actual memory used by memcached and mem_used is 2.04164548088 times
In 10.3.4.145 bucket default, the difference between actual memory used by memcached and mem_used is 2.10698300229 times

2.2.0-821

In 10.3.4.148 bucket default, the difference between actual memory used by memcached and mem_used is 2.01762128225 times
In 10.3.4.147 bucket default, the difference between actual memory used by memcached and mem_used is 2.05577910781 times
In 10.3.4.149 bucket default, the difference between actual memory used by memcached and mem_used is 2.10078181556 times

Andrei Baranouski
added a comment - 08/Jan/14 1:00 PM if compare the results against 2.5.0-994 & 2.2.0-821 we can see that the stats are identical and no improvement observed.
2.5.0-994
In 10.3.4.144 bucket default, the difference between actual memory used by memcached and mem_used is 2.03309838435 times
In 10.3.4.146 bucket default, the difference between actual memory used by memcached and mem_used is 2.04164548088 times
In 10.3.4.145 bucket default, the difference between actual memory used by memcached and mem_used is 2.10698300229 times
2.2.0-821
In 10.3.4.148 bucket default, the difference between actual memory used by memcached and mem_used is 2.01762128225 times
In 10.3.4.147 bucket default, the difference between actual memory used by memcached and mem_used is 2.05577910781 times
In 10.3.4.149 bucket default, the difference between actual memory used by memcached and mem_used is 2.10078181556 times

In 10.3.4.144 bucket default, total_fragmentation_bytes + the total_allocated_bytes = 1540784160
In 10.3.4.144 bucket default, mem_used = 604608360
In 10.3.4.144 bucket default, the difference between actual memory used by memcached and mem_used is 2.54840035622 times
In 10.3.4.146 bucket default, total_fragmentation_bytes + the total_allocated_bytes = 1437491168
In 10.3.4.146 bucket default, mem_used = 554534408
In 10.3.4.146 bucket default, the difference between actual memory used by memcached and mem_used is 2.59224882579 times
In 10.3.4.145 bucket default, total_fragmentation_bytes + the total_allocated_bytes = 1524727840
In 10.3.4.145 bucket default, mem_used = 608097400
In 10.3.4.145 bucket default, the difference between actual memory used by memcached and mem_used is 2.50737437785 times

In 10.3.4.144 bucket bucket0, total_fragmentation_bytes + the total_allocated_bytes = 1546715392
In 10.3.4.144 bucket bucket0, mem_used = 573434776
In 10.3.4.144 bucket bucket0, the difference between actual memory used by memcached and mem_used is 2.69728216135 times
In 10.3.4.146 bucket bucket0, total_fragmentation_bytes + the total_allocated_bytes = 1481072992
In 10.3.4.146 bucket bucket0, mem_used = 565021864
In 10.3.4.146 bucket bucket0, the difference between actual memory used by memcached and mem_used is 2.62126669137 times
In 10.3.4.145 bucket bucket0, total_fragmentation_bytes + the total_allocated_bytes = 1532223104
In 10.3.4.145 bucket bucket0, mem_used = 560881912
In 10.3.4.145 bucket bucket0, the difference between actual memory used by memcached and mem_used is 2.73181051344 times

build 2.5.0-1036:

In 10.3.4.148 bucket default, total_fragmentation_bytes + the total_allocated_bytes = 1593826880
In 10.3.4.148 bucket default, mem_used = 539971928
In 10.3.4.148 bucket default, the difference between actual memory used by memcached and mem_used is 2.95168470314 times
In 10.3.4.147 bucket default, total_fragmentation_bytes + the total_allocated_bytes = 1677467072
In 10.3.4.147 bucket default, mem_used = 584436664
In 10.3.4.147 bucket default, the difference between actual memory used by memcached and mem_used is 2.87022901766 times
In 10.3.4.149 bucket default, total_fragmentation_bytes + the total_allocated_bytes = 1532477440
In 10.3.4.149 bucket default, mem_used = 575876168
In 10.3.4.149 bucket default, the difference between actual memory used by memcached and mem_used is 2.66112321564 times

In 10.3.4.148 bucket bucket0, total_fragmentation_bytes + the total_allocated_bytes = 1604477248
In 10.3.4.148 bucket bucket0, mem_used = 558971400
In 10.3.4.148 bucket bucket0, the difference between actual memory used by memcached and mem_used is 2.87041027144 times
In 10.3.4.147 bucket bucket0, total_fragmentation_bytes + the total_allocated_bytes = 1685250592
In 10.3.4.147 bucket bucket0, mem_used = 568804984
In 10.3.4.147 bucket bucket0, the difference between actual memory used by memcached and mem_used is 2.9627915356 times
In 10.3.4.149 bucket bucket0, total_fragmentation_bytes + the total_allocated_bytes = 1571995840
In 10.3.4.149 bucket bucket0, mem_used = 560109160
In 10.3.4.149 bucket bucket0, the difference between actual memory used by memcached and mem_used is 2.80658834431 times

here I see the results on 2.5.0-1036 became worse by about 10 percent than on 2.2:

In 10.3.4.144 bucket default, total_fragmentation_bytes + the total_allocated_bytes = 1335845776
In 10.3.4.144 bucket default, mem_used = 608118296
In 10.3.4.144 bucket default, the difference between actual memory used by memcached and mem_used is 2.19668736295 times
In 10.3.4.146 bucket default, total_fragmentation_bytes + the total_allocated_bytes = 1366215208
In 10.3.4.146 bucket default, mem_used = 614061672
In 10.3.4.146 bucket default, the difference between actual memory used by memcached and mem_used is 2.22488272807 times
In 10.3.4.145 bucket default, total_fragmentation_bytes + the total_allocated_bytes = 1336260640
In 10.3.4.145 bucket default, mem_used = 592927640
In 10.3.4.145 bucket default, the difference between actual memory used by memcached and mem_used is 2.25366562436 times

In 10.3.4.144 bucket bucket0, total_fragmentation_bytes + the total_allocated_bytes = 1336082080
In 10.3.4.144 bucket bucket0, mem_used = 544514488
In 10.3.4.144 bucket bucket0, the difference between actual memory used by memcached and mem_used is 2.45371263657 times
In 10.3.4.146 bucket bucket0, total_fragmentation_bytes + the total_allocated_bytes = 1361140336
In 10.3.4.146 bucket bucket0, mem_used = 576695448
In 10.3.4.146 bucket bucket0, the difference between actual memory used by memcached and mem_used is 2.36024116493 times
In 10.3.4.145 bucket bucket0, total_fragmentation_bytes + the total_allocated_bytes = 1336205024
In 10.3.4.145 bucket bucket0, mem_used = 570649352
In 10.3.4.145 bucket bucket0, the difference between actual memory used by memcached and mem_used is 2.34155181166 times

build 2.5.0-1046:

In 10.3.4.148 bucket default, total_fragmentation_bytes + the total_allocated_bytes = 1700445864
In 10.3.4.148 bucket default, mem_used = 620101656
In 10.3.4.148 bucket default, the difference between actual memory used by memcached and mem_used is 2.74220500388 times
In 10.3.4.147 bucket default, total_fragmentation_bytes + the total_allocated_bytes = 1694731184
In 10.3.4.147 bucket default, mem_used = 608031784
In 10.3.4.147 bucket default, the difference between actual memory used by memcached and mem_used is 2.78724110909 times
In 10.3.4.149 bucket default, total_fragmentation_bytes + the total_allocated_bytes = 1786141968
In 10.3.4.149 bucket default, mem_used = 620130840
In 10.3.4.149 bucket default, the difference between actual memory used by memcached and mem_used is 2.88026631283 times

In 10.3.4.148 bucket bucket0, total_fragmentation_bytes + the total_allocated_bytes = 1700446208
In 10.3.4.148 bucket bucket0, mem_used = 580733208
In 10.3.4.148 bucket bucket0, the difference between actual memory used by memcached and mem_used is 2.9281022414 times
In 10.3.4.147 bucket bucket0, total_fragmentation_bytes + the total_allocated_bytes = 1691246496
In 10.3.4.147 bucket bucket0, mem_used = 535454056
In 10.3.4.147 bucket bucket0, the difference between actual memory used by memcached and mem_used is 3.15852775238 times
In 10.3.4.149 bucket bucket0, total_fragmentation_bytes + the total_allocated_bytes = 1735417856
In 10.3.4.149 bucket bucket0, mem_used = 563604536
In 10.3.4.149 bucket bucket0, the difference between actual memory used by memcached and mem_used is 3.079141038 times

First test uploads 1000 of documents each document size is 1000000 bytes (a million). And then does 200 appends of 10000 bytes for all docs.

Second test upload same initial documents as first, but appends not all but only 10 docs. And it does 1000 apppends.

I'm also comparing 2.5.0 and 2.5.0 with tcmalloc taking from 2.2.0-821.

You can see how 2.5.0 is having less memory overall. And even though it's still asking OS for more memory than is required to hold all values, it returns unused memory back to OS. Badness of 2.2.0 can be observed by big "Bytes in page heap freelist" statistic.

It's also notable how tcmalloc from 2.2.0-821 reports "NOTE: SMALL MEMORY MODEL IS IN USE, PERFORMANCE MAY SUFFER." where other tcmalloc builds are not doing it.

This node is caused by C preprocessor define we used to have. And looks like 2.2.0-840 and 2.5.0 do not have that define anymore. Which might explain regression.

Andrei, I'd like to discuss all this with you again. But prior to that it would be very interesting if you could do same runs as you did before with 2.2.0-840 (vs. 2.2.0-821 you already have) and post numbers.

Aleksey Kondratenko (Inactive)
added a comment - 20/Jan/14 6:40 PM It's also notable how tcmalloc from 2.2.0-821 reports "NOTE: SMALL MEMORY MODEL IS IN USE, PERFORMANCE MAY SUFFER." where other tcmalloc builds are not doing it.
This node is caused by C preprocessor define we used to have. And looks like 2.2.0-840 and 2.5.0 do not have that define anymore . Which might explain regression.
Andrei, I'd like to discuss all this with you again. But prior to that it would be very interesting if you could do same runs as you did before with 2.2.0-840 (vs. 2.2.0-821 you already have) and post numbers.
Also for all runs please include output of allocator stats

I should notice that formula introduced in this thread is invalid for multiple bucket:

"the difference between actual memory used by memcached and mem_used through calculation:
(total_fragmentation_bytes_stats[server]) + int(total_allocated_bytes_stats[server]) / float(mem_used_stats[server])"

Pavel Paulau
added a comment - 25/Feb/14 1:08 PM I should notice that formula introduced in this thread is invalid for multiple bucket:
"the difference between actual memory used by memcached and mem_used through calculation:
(total_fragmentation_bytes_stats [server] ) + int(total_allocated_bytes_stats [server] ) / float(mem_used_stats [server] )"
https://github.com/couchbase/testrunner/blob/master/pytests/memorysanitytests.py#L179
"mem_used" is measured per bucket, "total_fragmentation_bytes_stats" and "total_allocated_bytes_stats" per process.
Thus all metrics reported for 2 buckets are most likely misleading.
Tests with single bucket or proper calculation based on allocator stats will make more sense.

Also I've decided not to bother with any special tcmalloc builds and extra test passes and filed https://www.couchbase.com/issues/browse/MB-10371 directly. It looks quite plausible that regression on small appends was caused by build regression.

Aleksey Kondratenko (Inactive)
added a comment - 05/Mar/14 3:34 PM Agree with Pavel above. Need single bucket.
Also I've decided not to bother with any special tcmalloc builds and extra test passes and filed https://www.couchbase.com/issues/browse/MB-10371 directly. It looks quite plausible that regression on small appends was caused by build regression.

I read through the notes. looks like we already did some fragmentation fixes in the lifetime of the bug and at this point, we don't see a high rate of impact to customers or large repro rate in the wild. Downgrading to a Minor. Raise your voice if you feel otherwise pls.

Cihan Biyikoglu
added a comment - 19/Mar/14 11:19 AM - edited I read through the notes. looks like we already did some fragmentation fixes in the lifetime of the bug and at this point, we don't see a high rate of impact to customers or large repro rate in the wild. Downgrading to a Minor. Raise your voice if you feel otherwise pls.