3. Query the view with stale=false to build the index
curl -X GET 'http://127.0.0.1:9500/default/_design/d1/_view/v1?stale=false'
{"total_rows":2,"rows":[

{"id":"ab","key":"ab","value":"dmFs"}

,

{"id":"ab1","key":"ab1","value":"dmFs"}

]
}

4. Query the view after couple of minutes and expired items are still returned
curl -X GET 'http://127.0.0.1:9500/default/_design/d1/_view/v1?stale=false'
{"total_rows":2,"rows":[

{"id":"ab","key":"ab","value":"dmFs"}

,

{"id":"ab1","key":"ab1","value":"dmFs"}

]
}

If memcached get is used with these items, then these are excluded from the views. Otherwise these are always returned in view results.

Diagnostic is attached.

Checked with Mike on this and ep-engine seems to be doing things correctly:

"This does look like an issue, but not in the ep-engine side. Since ep-engine might take an hour to actually remove an expired item, it should be up to the view engine to filter out any expired items too. The reaon why doing a get will cause the item to disappear from the view results is that ep-engine will actually do the deletion."

Filipe Manana (Inactive)
added a comment - 30/Oct/12 4:00 AM Won't work unless the docs are deleted from the vbucket databases.
See:
http://hub.internal.couchbase.com/confluence/display/QA/Debugging+view+engine+issues+and+reporting+them#Debuggingviewengineissuesandreportingthem-section5
Simply doesn't work for several architectural reasons.
See the discussion there.

This does look like an issue, but not in the ep-engine side. Since ep-engine might take an hour to actually remove an expired item, it should be up to the view engine to filter out any expired items too. The reaon why doing a get will cause the item to disappear from the view results is that ep-engine will actually do the deletion.

Deepkaran Salooja
added a comment - 31/Oct/12 8:39 AM Copying full email conversation with Mike on this.
On Oct 29, 2012, at 11:56 AM, Mike Wiederhold <mike@couchbase.com> wrote:
Yes, that's what I would expect.
Mike
On Oct 29, 2012, at 11:53 AM, Farshid Ghods wrote:
okay in that case view index should be updated and stale=false query should pick that up
On Oct 29, 2012, at 11:46 AM, Mike Wiederhold <mike@couchbase.com> wrote:
Yes expiration time is updated on disk by ep-engine.
Mike
On Oct 29, 2012, at 11:44 AM, Farshid Ghods wrote:
Mike,
does ep-engine update the expiration time on for this key on the disk ?
if so then view-engine can skip this item when generating the results i guess
-Farshid
On Oct 29, 2012, at 11:41 AM, Mike Wiederhold <mike@couchbase.com> wrote:
This does look like an issue, but not in the ep-engine side. Since ep-engine might take an hour to actually remove an expired item, it should be up to the view engine to filter out any expired items too. The reaon why doing a get will cause the item to disappear from the view results is that ep-engine will actually do the deletion.
Mike

Farshid Ghods (Inactive)
added a comment - 31/Oct/12 8:44 AM One thing i am not quite sure about ep-engine behavior is whether the disk is updated before expiry pager runs ?
Yaseen,
can we confirm this with ep-engine folks or ideally if there is a spec about how ep-engine handles expirations before/after expiry pager runs

this is a change that can be addressed post 2.0 instead of changing the view-engine or ep-engine at this stage , and i agree that it is confusing for the users so maybe we can simply change how often expiry pager runs if they want faster updates ?

Farshid Ghods (Inactive)
added a comment - 31/Oct/12 8:49 AM Dipti/Yaseen,
this is a change that can be addressed post 2.0 instead of changing the view-engine or ep-engine at this stage , and i agree that it is confusing for the users so maybe we can simply change how often expiry pager runs if they want faster updates ?

Farshid Ghods (Inactive)
added a comment - 31/Oct/12 8:55 AM Dipti/Yaseen
Filipe has also explained the options in more details here as why skipping results in the time of reduction or view results wont work. so this bug is basically a duplicate.
http://www.couchbase.com/issues/browse/MB-6219?focusedCommentId=36489&page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-36489

idea from Damien (might repeat something Filipe had on MB-6219?): track expiration time in secondary index.

run expiry pager more often - this scans entire hashtable, locking each hashtable partition at a time. This only reduces the window of the issue, but doesn't fundamentally solve the issue. Ask some customers set their expiry pager more often (e.g., once every 10 minutes for small/medium cluster).

Steve Yen
added a comment - 01/Nov/12 4:41 PM options...
idea from Damien (might repeat something Filipe had on MB-6219 ?): track expiration time in secondary index.
run expiry pager more often - this scans entire hashtable, locking each hashtable partition at a time. This only reduces the window of the issue, but doesn't fundamentally solve the issue. Ask some customers set their expiry pager more often (e.g., once every 10 minutes for small/medium cluster).

Recommend to impacted customers to run expiry pager more often – e.g., once every 10 minutes for small/medium clusters. This can help mitigate the inconsistency (but not fully solve) the issue. Running expiry pager more often also might have disk I/O impact / tradeoff, where it would actually write deletes to disk. User should also be aware of expiry pager in tradeoff with all the other dispatcher / tasks.

Steve Yen
added a comment - 02/Nov/12 1:31 PM For 2.0, options discussed in bug-scrub...
Recommend to impacted customers to run expiry pager more often – e.g., once every 10 minutes for small/medium clusters. This can help mitigate the inconsistency (but not fully solve) the issue. Running expiry pager more often also might have disk I/O impact / tradeoff, where it would actually write deletes to disk. User should also be aware of expiry pager in tradeoff with all the other dispatcher / tasks.
Frank, Dipti, Yaseen to discuss today (2012/11/02) outside of bug-scrub mtg and resolve.

Tracking the expiration times in the indexes (values at the leaf nodes) would solve the issue for map view queries. True.
However, we already have plenty of metadata tracked in the indexes (vbucket id for each value, 1024 bits/128 bytes bitmasks), that makes us lose some performance (deeper trees, smaller branching factor per tree node). At the moment, querying Apache CouchDB is faster than Couchbase Server (single node of course, to be fair).

Now, for reduce views... Excluding values that were contributed by now-expired documents, means going down to all the leaf nodes to find out which values come from expired documents and which ones come from non-expired documents - then grab all the values from non-expired documents, applying reduce function against those values, and going up to the tree applying re-reduces until reaching the root. In other words, we would be doing not better than a linear scan, and defeating the whole purpose of intermediary reductions/efficiency for which CouchDB trees are known for.
Basically doing this would, at the very best, give query response times of a few seconds (being very optimistic here) for any reasonably sized index (even less than than 1M items perhaps).

Filipe Manana (Inactive)
added a comment - 05/Nov/12 6:49 AM - edited Perhaps, I was not fully clear before.
Tracking the expiration times in the indexes (values at the leaf nodes) would solve the issue for map view queries. True.
However, we already have plenty of metadata tracked in the indexes (vbucket id for each value, 1024 bits/128 bytes bitmasks), that makes us lose some performance (deeper trees, smaller branching factor per tree node). At the moment, querying Apache CouchDB is faster than Couchbase Server (single node of course, to be fair).
Now, for reduce views... Excluding values that were contributed by now-expired documents, means going down to all the leaf nodes to find out which values come from expired documents and which ones come from non-expired documents - then grab all the values from non-expired documents, applying reduce function against those values, and going up to the tree applying re-reduces until reaching the root. In other words, we would be doing not better than a linear scan, and defeating the whole purpose of intermediary reductions/efficiency for which CouchDB trees are known for.
Basically doing this would, at the very best, give query response times of a few seconds (being very optimistic here) for any reasonably sized index (even less than than 1M items perhaps).

MC Brown (Inactive)
added a comment - 21/Nov/12 9:53 AM The documentation has been updated at multiple points to highlight the inclusion of documents that may have expired, but not been fully deleted yet.

Couchbase Server does lazy expiration, that is, expired items are flagged as
deleted rather than being immediately erased. Couchbase Server has
a maintenance process that will periodically look through all information and erase expired items.
This means expired items may still be indexed and appear in result sets of views. The workarounds
are available here <ulink url="http://www.couchbase.com/docs/couchbase-devguide-2.0/about-ttl-values.html">
About Document Expiration</ulink>.

kzeller
added a comment - 06/Dec/12 4:42 PM Added to Release Notes as:
Couchbase Server does lazy expiration, that is, expired items are flagged as
deleted rather than being immediately erased. Couchbase Server has
a maintenance process that will periodically look through all information and erase expired items.
This means expired items may still be indexed and appear in result sets of views. The workarounds
are available here <ulink url="http://www.couchbase.com/docs/couchbase-devguide-2.0/about-ttl-values.html">
About Document Expiration</ulink>.