I think https://bugs.edge.launchpad.net/launchpadlib/+bug/341950/comments/2 also explains this issue described here, but I'm not sure if both bugs are related and can be fixed the same way.
Maybe some insider of launchpad's code can mark them as duplicates and rise priority of this bug, for me it looks like a design issue which should be fixed in an early stage of launchpadlib and/or the API

bug = self.launchpad.bugs[id] bug.newMessage(content='This crash has the same stack trace characteristics as bug #%i. \
However, the latter was already fixed in an earlier package version than the \
one in this report. This might be a regression or because the problem is \
in a dependent package.' % master, subject='Possible regression detected')
bug.tags = bug.tags + ['regression-retracer'] # LP#254901 workaround

The values for If-None-Match and If-Match are totally random. They don't show up anywhere in the cache or anywhere else in the HTTP dump. In fact, even the initial requests to /beta/ have a random If-None-Match.

If I clear the cache, I don't send any If-None-Match headers, but eventually I do send a random If-Match and I get the same error.

OK, I found at least one problem that causes 412 errors. When you PATCH a representation you get back 209 ("Content Returned") with a new representation. The http_etag in this representation is the random ETag that's sent on a subsequent PATCH and then you get 412.

This is a problem specific to Launchpad bugs. The problem does not exist in the lazr.restful example web service, and it doesn't exist for other Launchpad resources (such as people). Everyone who's commented on this thread has had a problem with bugs.

The culprit is the date_last_updated field. Here are the only differences between the version of the bug received along with a 209, and the same bug retrieved an instant later with a fresh GET request.

date_last_updated is updated after the ETag is calculated, possibly in response to the ObjectModifiedEvent. Or maybe it's done by a database trigger. I'm not sure. Anyway, the bug gets changed twice, and the effect is the same as if someone else modified the bug after you did: your copy of the bug is no longer valid.

The culprit is a combination of the unmarshalled_field_cache and the ObjectModifiedEvent. The OME modified date_last_updated, but lazr.restful didn't hear about it, and only removed the fields the client modified from unmarshalled_field_cache. So the old date_last_updated stays in there.

I have a fix but since the lazr.restful example service doesn't use OMEs at all, a proper test will take a little while.

This is hitting me too in Hydrazine, and it's fairly easily reproducible there if you make changes quickly. It does seems timing-dependent which would be consistent with it relating to a last-modified field stored on the server: it seems like if you get two updates within a 1s window then it crashes.

On Tue, 02 Mar 2010 11:11:47 -0000, Martin Pitt <email address hidden> wrote:
> Apport already has workarounds for this, but apparently still not
> enough. This still keeps people from submitting debug data to Launchpad,
> see bug 528680.
>
> Can the priority of this please be bumped?

Martin, how sure are you that these are spurious?

Is there a chance that they are actually legitimate errors due to
something else modifying the bug between you fetching it and saving it?

On Tue, 02 Mar 2010 12:39:09 -0000, Martin Pitt <email address hidden> wrote:
> The window for that is pretty small, since it's an automated script with
> not much in between. Martin-Eric said that he tried this a number of
> times, and other people saw this as well.
>
> Also, it's perfectly reproducible in the Apport test suite if I remove
> all the workarounds for this bug.

Ok, thanks. I don't do either of these things in scripts myself, so
haven't seen it.

I did however do things with branches over the API that was getting 412
a lot, but this came down to automated tasks that LP does in the
background the modify the branch happening in about the same amount of
time after pushing the branch that my script took to get to the API
stage, leading to legitimate 412 errors that were easy to work around
once I understood what was going on.

If creating a new resource secretly modifies the parent resource, then the client-side fix is the correct thing to do. But I don't know if that would help the apport people. Can someone show me the apport code that has the problem, and the workarounds devised for it?

I know of two new bug fields that change behind-the-scenes automatically, bug_heat and task_age. task_age is going away, so let's ignore that. I'd like to fix bug_heat and then see if there's a separate problem to do with named operations.

Here's a paste of a message I posted to launchpad-dev about solving the bug_heat problem. I'm still working on this.

1. If a field is important enough to include in the representation, it's
important enough to include in the ETag. As Robert points out, omitting
a read-only field from the ETag will stop clients from knowing when the
value changes.

2. I like Bjorn's idea of a two-part ETag, where both parts are checked
for GET requests/caching, but only one part is checked for PATCH/PUT
requests. But, I don't think this would comply with the HTTP standard.

There are two types of ETags, "strong" and "weak". A strong ETag changes
whenever "the entity (the entity-body or any entity-headers) changes in
any way". A weak ETag changes "only on semantically significant changes,
and not when insignificant aspects of the entity change". These quotes
are from section 13.3.3.

Consider a two-part ETag, where one part describes the state of the
read-only fields and the other part describes the state of the
read-write fields. Taken together, the whole thing is a strong ETag.
Look at only the second part and you have a weak ETag.

But (13.3.3 again) "The only function that the HTTP/1.1 protocol defines
on [ETags] is comparison." There's no looking at the second half of an
ETag. The whole thing has to match.

OK, so let's define another function on ETags, another "weak comparison
function" which only looks at the second half of the ETag. That goes
beyond HTTP/1.1 but it doesn't contradict the standard. Now:

We would like to validate GET requests using the strong comparison
function (looking at the strong ETag), so that even minor changes will
invalidate the cache and the client will always have the most up-to-date
information. We would like to validate PATCH requests by using the weak
comparison function, so that irrelevant changes to the resource don't
affect the success of the PATCH.

But, the HTTP standard says we have to do the opposite. Section 14.26
says "The weak comparison function can only be used with GET or HEAD
requests."

The IETF draft that defines PATCH says, "Clients wishing to apply a
patch document to a known entity can first acquire the strong ETag of
the resource to be modified, and use that Etag in the If-Match header on
the PATCH request to verify that the resource is still unchanged."

As I read the standards, if bug_heat has changed and I try to PATCH an
unrelated field, the correct behavior is that my PATCH should fail. I
don't think this makes sense for us given the way we implement PATCH,
but it does make sense in general.

I don't think anything bad will happen if we implement the two-part
ETag. The whole ETag string is in fact a strong ETag, so anything that
treats it as a strong ETag will still work. We'll just have some
server-side magic that secretly implements a weak comparison function to
make PATCH requests work. And if anyone actually tries to PATCH bug_heat
they'll still get an error...

On Tue, 09 Mar 2010 18:55:48 -0000, Leonard Richardson <email address hidden> wrote:
> I know of two new bug fields that change behind-the-scenes
> automatically, bug_heat and task_age. task_age is going away, so let's
> ignore that. I'd like to fix bug_heat and then see if there's a separate
> problem to do with named operations.

There is a separate problem related to named operations, on bugs at
least, due to the dates of operations that are exported.

They are however read-only and so your proposal would allow them to
function assuming that those dates were not in the strong etag.

The lack of refresh means that the local representation will be out of
date after some named operations though.

I think the linked branch will resolve this problem. The only exception is if there is a named operation that modifies a field you can also modify directly, like IBug.status. Of course, this is what mutator methods like transitionToStatus do, but we got rid of those methods in version 1.0 of the web service, partly for this reason.

Are there any named operations that modify a field you can modify directly, but that aren't mutator methods whose _only_ purpose is to modify that field? Ie. operations that will still exist in version 1.0 of the web service.

On Tue, 16 Mar 2010 12:31:48 -0000, Leonard Richardson <email address hidden> wrote:
> I think the linked branch will resolve this problem. The only exception
> is if there is a named operation that modifies a field you can also
> modify directly, like IBug.status. Of course, this is what mutator
> methods like transitionToStatus do, but we got rid of those methods in
> version 1.0 of the web service, partly for this reason.
>
> Are there any named operations that modify a field you can modify
> directly, but that aren't mutator methods whose _only_ purpose is to
> modify that field? Ie. operations that will still exist in version 1.0
> of the web service.