Since I have been a home recently, I put some time correcting
bugs in WaffleGrid and adding new features. Thanks to gdb, I have
been able to understand a silly bug that was affecting WaffleGrid
with sysbench but, weird enough, not with dbt2. Everything is in
the way connections are established. I will blog more about that
soon.

Regarding the new features, it is now possible to choose which
tables you want to push to memcached. For that purpose, two new
parameters have been introduce:

“Moving beyond just providing Oracle compatibility, EnterpriseDB
is adding new scalability features to their database. One of them
is a feature called ‘”Infinite Cache,” which is based on the open
source memcached application.

Mlodgenski commented that distributed memory caches are common,
though management from a system management and a database
perspective is often difficult.

“What we did is we took the distributed memory cache that was
sitting in front of the database and put it behind the database,”
Mlodgenski said. “This allows a simple SQL (define) …

So Ronald
Bradford has been getting some EC2 AMI’s setup for Waffle the
last couple of days. Really he is the first person to work with
us on testing out the 0.5 cream release. It’s funny but when you
are heads down in the code you tend to overlook some of the easy
and obvious bugs. Just by having him walk through the install we
have opened up several new bugs and come up with some new Idea’s
on how to make things easier in the future. I hope by early next
week we can have some of annoying bugs fixed and code pushed to
launchpad. When that happens I would encourage others to give it
a whirl and help us make Waffle Grid better …

Starting off simple lets look at the difference between the
wafflegrid modes. As mentioned before the LRU mode is the
“classic” Waffle Grid setup. A page is put into memcached when
the page is removed from the buffer pool via the LRU process.
When a page is retrieved from memcached it is expired so its no
longer valid. In the New “Non-LRU” mode when a page is read from
disk, the page is placed in memcached. When a dirty page is
flushed to disk, this page is overwritten in memcached. So how do
the different modes perform?

I wanted to let everyone know that we are releasing Waffle Grid
0.5 code named Cream. This release fixes the nasty secondary
index bug that plagued the butter release. I have been running
tests on this code base for about a week straight with no errors.
While I think this release is much more stable I would remind
everyone this is still not a fully GA release. This release
includes the ability to choose the mode of Waffle grid. By
setting innodb_memcached_enable to 1, we will push pages to
memcached when a disk read is done or when a page write is done,
setting this to 2 will enable the classic LRU mode. If you decide
to set this to 1 ( non-lru) I would recommend using the standard
memcached, as …

So I spent several hours over the last few days on the Secondary
index bug. Out of frustration I decided to try and bypass the LRU
concept all together and try going to a true secondary page
cache. In standard Waffle a page is written to memcached only
when it is expunged ( or LRU’d ) from the main buffer pool. This
means anything in the BP should not be in memcached. Obviously
with this approach we missed something, as Heikii pointed out in
a comment to a previous post, it seems likely we are getting an
old version of a page. Logically this could happen if we do not
correctly expire a page on get or we bypass a push/lru leaving an
old page in memcached to be retrieved later on.

So I spent the weekend looking at places where we may have missed
something in the code for waffle. You can actually see some of
the stuff I tried in the bug on launchpad about this, but the weird thing
is the very last thing I tried. As I took a step back and looked
at the problem ( secondary index corruption ) and our assumption
that we “missed” something, I decided to find the place where
pages are written to disk and to push to memcached from here as
well as from the LRU. With the double write buffer enabled that
place should be buf_flush_buffered_writes. By pushing to
memcached here we should eliminate the …

If you read Yves blog post about waffle yesterday we are
seeing some weird gremlins in the system and could use some
scoobey doo detective work if you have some ideas. The strange
thing is it only exhibits under high load. So it really seems
like we may have missed some background cleanup process that
accesses or removes pages from disk or the buffer pool without
going through the functions we call waffle in
(buf_LRU_search_and_free_block & buf_read_page_low ).

One of the idea’s I had was trying to narrow the scope of what’s
being pushed and read form Memcached. Even though I am using file
per table, system …

You probably haven’t noticed but I have not blogged since the UC.
It is not because I am upset by the perspective of working for
Oracle, I have simply been busy tracking down an issue we have
with WaffleGrid. We discovered that under high load, with DBT2 on
a tmpfs, we end up with error in a secondary index. In the error
of MySQL, we have entries like this one:

Content reproduced on this site is the property of the respective copyright holders. It is not reviewed in advance by Oracle and does not necessarily represent the opinion of Oracle or any other party.