[BUILDFIX PATCH] au0828: debug is a poor global identifier - Kernel

This is a discussion on [BUILDFIX PATCH] au0828: debug is a poor global identifier - Kernel ; On Sat, 26 Apr 2008, Sam Ravnborg wrote:
>
> It also depends on whare you are located in the dependency tree.
Absolutely.
> Being kbuidl maintainer I have very few people that actually pull me git
> tree (except ...

Re: If you want me to quit I will quit

On Sat, 26 Apr 2008, Sam Ravnborg wrote:
>
> It also depends on whare you are located in the dependency tree.

Absolutely.
> Being kbuidl maintainer I have very few people that actually pull me git
> tree (except from -mm and -next). So I rebase at will and have so far
> not got a single complaint from anyone pulling my tree.

I agree. Some trees are so specific (and/or simply don't have enough
patches in them) that it simply doesn't matter if two different people
pull the same tree. Even if it might end up causing some duplication of
commits (because the pulled tree might end up being then pulled further),
it's not a big deal if it's rare.

In fact, we have always had duplicated commits even when they are passed
around as email - just because perhaps two different trees simply needed
the same fix, and rather than wait for it, they both integrated it (and
then when they get merged, the same patch exists twice in the history,
just with different committer info etc).

So yeah, rebasing ends up being really convenient if you really don't
expect to have any other "real" end users than eventually being pulled
into my tree (or, even more commonly, and when rebasing is *really*
convenient: when it's just you keeping track of your own private patches
in your own private tree and don't know if they will *ever* go upstream at
all).
> But people like Davem and Ingo sits much higher in the dependency chain
> and thus they have a very different set of users and thus a different
> set of problems to take into account.

Yes. David has changed his workflow to accomodate others, while Ingo still
does the rebasing (and it works out because nobody else works on his trees
using git).

Re: If you want me to quit I will quit

Linus Torvalds wrote:
> But it's not a hard rule. Sometimes simple cleanliness means that you can
> decide to go "oops, that was *really* wrong, let's just throw that away
> and do a whole new set of patches". But it should be something rare - not
> normal coding practice.

Well, the need to amend single patches --- and folding the amendment in
before mainline submission to correct important problems of the first
shot --- is something which happens all the time.

(People will have different opinions about what is important enough to
redo a patch instead of keeping the amendment separate.)
> Because if it becomes normal coding practice, now people cannot work with
> you sanely any more (ie some random person pulls your tree for testing,
> and then I pull it at some other time, and the tester reports a problem,
> but now the commits he is talking about don't actually even exist in my
> tree any more, and it's all really messy!).

In my experience, the submission branch cannot practically be the same
as the development branch (I mean, a true ancestor of the of the
development branch). That's simply because the order of submission is
different.

The only way to keep commits from the development branch identical in
the submission branch would be to work with a huge number of topic
branches, with the number of branches approaching the number of commits.
But then testers and developers would still work with merge commits
which will not appear in the mainline submission. And there would be
weird histories because of the need to merge from other trees, notably
from torvalds/linux-2.6.git when necessary to avoid conflicts.

(I for one rebuild the development branch of linux1394-2.6.git from
scratch after 2.6.x-rc1 came out, then keep its history until 2.6.x.
During that time I occasionally 'pollute' it with reverts and with
merges from torvalds/linux-2.6.git. These will not be carried over into
the submission branch.)
--
Stefan Richter
-=====-==--- -=-- ==-=-http://arcgraph.de/sr/
--
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at http://vger.kernel.org/majordomo-info.html
Please read the FAQ at http://www.tux.org/lkml/

Re: If you want me to quit I will quit

On Sat, 26 Apr 2008, Stefan Richter wrote:
>
> Well, the need to amend single patches --- and folding the amendment in before
> mainline submission to correct important problems of the first shot --- is
> something which happens all the time.

... and you simply SHOULD NOT PUBLICIZE the tree before it has gotten to a
reasonable point.

Keep the rough-and-not-ready thing that is being discussed as patches on
lkml as your own working tree, and just don't expose it as a public git
branch. You can't do any sane discussion over git anyway - if things are
being actively worked-on among people, you'd be passing patches around as
emails etc.

Yes, people may be (and I would strongly suggest _should_ be) using
something like git or quilt etc to keep track of the patches that they
(and others) have been discussing over email, but that has absolutely
nothing to do with making a public git tree available to others.

So:
- making things public is *different* from developing them. Don't push
out just because you committed something!

- you shouldn't publicize a tree until it's in reasonable shape. EVER.
Even -mm or -next is *not* better off with a pile of sh*t just because
you're working in that area.

I cannot stress this enough. I think Andrew has been way too polite to
some people.

- and once it is, you generally shouldn't mess with old commits even when
you fix things. Full cleanliness or always being able to bisect
specific configurations is not an excuse for messing up all the other
things, and if this problem happens a lot, I would like to point you to
the two previous points.

Re: If you want me to quit I will quit

On Sat, 26 Apr 2008, Linus Torvalds wrote:
>
> So:
> - making things public is *different* from developing them. Don't push
> out just because you committed something!
>
> - you shouldn't publicize a tree until it's in reasonable shape. EVER.
> Even -mm or -next is *not* better off with a pile of sh*t just because
> you're working in that area.
>
> I cannot stress this enough. I think Andrew has been way too polite to
> some people.
>
> - and once it is, you generally shouldn't mess with old commits even when
> you fix things. Full cleanliness or always being able to bisect
> specific configurations is not an excuse for messing up all the other
> things, and if this problem happens a lot, I would like to point you to
> the two previous points.

And btw, a *big* part of the above is also:

- mistakes happen.

There will be bugs. There will be cases where things aren't bisectable
(although they should generally be bisectable for *your* configuration,
because if they aren't, that shows that you didn't even compile the
commits you made).

And there will be kernels that don't boot. Even expecting people to always
boot-test every single commit would be unrealistic - let's face it, most
things look really obvious, and the fact that even obvious fixes can have
bugs doesn't mean that there should be hard rules about "every single
commit has to be boot-tested on X machines".

So it's an important part of the process to try to do a good job, and not
publicizing crap - but it's *equally* important to realize that crap
happens, and that it's easily *more* distracting to try to clean it up
after-the-fact than it is to just admit that it happened.

I do generally try to check the mailing list to avoid this...but LKML
tends to be pretty busy. I also try to check the latest -git devel
tree for the affected area in case it's waiting to be pulled. In this
case it was so trivial I sent it directly to Linus.

In fact, I concentrate on looking at the sparse output because you
do such a good job of catching unnecessary exports/globals.
> And if you e.g. discover next that our x86 userspace headers in
> 2.6.25 (sic) are ****ed up guess who already sent a patch last
> weekend... [2]
>
> Why do other people get over 100 checkpatch fixes into the tree at once
> or Linus applies patches directly bypassing the maintainers (like the
> one you sent just before [3], which I've also already sent before [4])
> but my patches bitrot forever?

And look how many of mine have been sitting in -mm for weeks because I
thought they should go through a maintainer/-mm. This was just annoying
as it was making my sparse builds quit, and I thought others might be in
the same boat.

Re: If you want me to quit I will quit

On Sat, Apr 26, 2008 at 12:20:55PM -0700, Harvey Harrison wrote:
> On Sat, 2008-04-26 at 14:00 +0300, Adrian Bunk wrote:
> > > [BUILDFIX PATCH] au0828: debug is a poor global identifier
> >
> > Another issue for which I already sent a (slightly different) patch.
> >
> > Why not check the mailing list before duplicating work? [1]
>
> I do generally try to check the mailing list to avoid this...but LKML
> tends to be pretty busy. I also try to check the latest -git devel
> tree for the affected area in case it's waiting to be pulled. In this
> case it was so trivial I sent it directly to Linus.
>
> In fact, I concentrate on looking at the sparse output because you
> do such a good job of catching unnecessary exports/globals.
>...

Sorry that my mail was unfriendly, but unfriendly flames are
unfortunately sometimes the only choice for getting any response
on linux-kernel (and now I've gotten the responses my friendly
mail to Linus did not yield).
> Cheers,
>
> Harvey

cu
Adrian

--

"Is there not promise of rain?" Ling Tan asked suddenly out
of the darkness. There had been need of rain for many days.
"Only a promise," Lao Er said.
Pearl S. Buck - Dragon Seed

Re: If you want me to quit I will quit

Linus Torvalds wrote:
> On Sat, 26 Apr 2008, Stefan Richter wrote:
>> Well, the need to amend single patches --- and folding the amendment in before
>> mainline submission to correct important problems of the first shot --- is
>> something which happens all the time.
>
> .. and you simply SHOULD NOT PUBLICIZE the tree before it has gotten to a
> reasonable point.

I always aim for keeping the one subsystem git tree which I maintain in
reasonable shape. Including its -mm and -next feeding branches.
> Keep the rough-and-not-ready thing that is being discussed as patches on
> lkml as your own working tree, and just don't expose it as a public git
> branch. You can't do any sane discussion over git anyway - if things are
> being actively worked-on among people, you'd be passing patches around as
> emails etc.

Sometimes I want problem reporters to test more than a single patch. In
these cases, I refer them to patch kits and quilt queues which I
maintain independently of linux1394-2.6.git; I almost never ask testers
to try linux1394-2.6.git.

linux1394-2.6.git has been, and still is, first and foremost my
submission branch to -mm. And I have high expectations of what I submit
to -mm. But sometimes something turns out to be a bad idea after all
_long after_ I and the reviewers and testers thought it would be fine.

Besides, the care that I take with my submissions to -mm also does not
change the fact that I submit in different order to you than to -mm.
(As mentioned, reordering could be done to some degree with a respective
number of branches. I don't do it this way currently.)
--
Stefan Richter
-=====-==--- -=-- ==-=-http://arcgraph.de/sr/
--
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at http://vger.kernel.org/majordomo-info.html
Please read the FAQ at http://www.tux.org/lkml/

Re: If you want me to quit I will quit

On Sat, 26 Apr 2008 12:18:34 -0700 (PDT) Linus Torvalds wrote:
> So it's an important part of the process to try to do a good job, and not
> publicizing crap - but it's *equally* important to realize that crap
> happens, and that it's easily *more* distracting to try to clean it up
> after-the-fact than it is to just admit that it happened.
>

Often it takes quite a long time for problems to become apparent. Across a
month or two we end up with things like:

Fact is, this is the way in which developers want to work. That is their
workflow, and their tools should follow their workflow. If a tool's
behaviour prevents them from implementing their desired workflow, it isn't
the workflow which should be changed

Re: If you want me to quit I will quit

On Sat, 26 Apr 2008, Andrew Morton wrote:
> On Sat, 26 Apr 2008 12:18:34 -0700 (PDT) Linus Torvalds wrote:
>
> > So it's an important part of the process to try to do a good job, and not
> > publicizing crap - but it's *equally* important to realize that crap
> > happens, and that it's easily *more* distracting to try to clean it up
> > after-the-fact than it is to just admit that it happened.
>
> Fact is, this is the way in which developers want to work. That is their
> workflow, and their tools should follow their workflow. If a tool's
> behaviour prevents them from implementing their desired workflow, it isn't
> the workflow which should be changed

But that was exactly my point. Bugs *will* happen. Follow-up patches
*will* happen. Don't fight it. Do the best you can do - there's no way
people will ever avoid all bugs to begin with.

And trying to white-wash things later is just pointless and actively
*bad*, when others have already seen and merged the original patches.

Re: If you want me to quit I will quit

Andrew Morton wrote:
> Often it takes quite a long time for problems to become apparent. Across a
> month or two we end up with things like:
[...]
> that's two patches, each with three followon fixes. Very common.
>
> Fact is, this is the way in which developers want to work. That is their
> workflow, and their tools should follow their workflow. If a tool's
> behaviour prevents them from implementing their desired workflow, it isn't
> the workflow which should be changed

Well, some of the questions here, notably
- when to fold patch + amendment patch into a single patch
- when to publish things to whom
are actually about the workflow, not about the tool.

Re: If you want me to quit I will quit

From: Linus Torvalds
Date: Sat, 26 Apr 2008 10:31:36 -0700 (PDT)
> And once you've publicized them, you don't know who has that tree, so just
> from a sanity and debugging standpoint, you should try to avoid mucking
> with already-public versions. If you made a mistake, add a patch on top to
> fix it (and announce the new state), but generally try to not "hide" the
> fact that the state has changed.

I didn't rebase at all this past cycle for networking, and it
helped the people downstream from me _A LOT_.

Anyone who rebases frequently, like I used to, is an idiot, like
me. :-)

Re: If you want me to quit I will quit

On Sat, Apr 26, 2008 at 05:31:29PM -0700, David Miller wrote:
> From: Linus Torvalds
> Date: Sat, 26 Apr 2008 10:31:36 -0700 (PDT)
>
> > And once you've publicized them, you don't know who has that tree, so just
> > from a sanity and debugging standpoint, you should try to avoid mucking
> > with already-public versions. If you made a mistake, add a patch on top to
> > fix it (and announce the new state), but generally try to not "hide" the
> > fact that the state has changed.
>
> I didn't rebase at all this past cycle for networking, and it
> helped the people downstream from me _A LOT_.

Re: If you want me to quit I will quit

On Sat, 2008-04-26 at 13:30 -0700, Andrew Morton wrote:
> On Sat, 26 Apr 2008 12:18:34 -0700 (PDT) Linus Torvalds wrote:
> > So it's an important part of the process to try to do a good job, and not
> > publicizing crap - but it's *equally* important to realize that crap
> > happens, and that it's easily *more* distracting to try to clean it up
> > after-the-fact than it is to just admit that it happened.
> >
>
> Often it takes quite a long time for problems to become apparent. Across a
> month or two we end up with things like:
>
> mm-have-zonelist-contains-structs-with-both-a-zone-pointer-and-zone_idx.patch
> mm-have-zonelist-contains-structs-with-both-a-zone-pointer-and-zone_idx-fix-memcg-ooms.patch
> mm-have-zonelist-contains-structs-with-both-a-zone-pointer-and-zone_idx-just-return-do_try_to_free_pages.patch
> mm-have-zonelist-contains-structs-with-both-a-zone-pointer-and-zone_idx-just-return-do_try_to_free_pages-do_try_to_free_pages-gfp_mask-redundant.patch
>
> and
>
> mm-filter-based-on-a-nodemask-as-well-as-a-gfp_mask.patch
> mm-filter-based-on-a-nodemask-as-well-as-a-gfp_mask-doc-fixes.patch
> mm-filter-based-on-a-nodemask-as-well-as-a-gfp_mask-make-dequeue_huge_page_vma-obey-mpol_bind-nodemask.patch
> mm-filter-based-on-a-nodemask-as-well-as-a-gfp_mask-make-dequeue_huge_page_vma-obey-mpol_bind-nodemask-rework.patch
>
> that's two patches, each with three followon fixes. Very common.
>
> Fact is, this is the way in which developers want to work. That is their
> workflow, and their tools should follow their workflow. If a tool's
> behaviour prevents them from implementing their desired workflow, it isn't
> the workflow which should be changed

Its worth realising that these fix patches contain useful information
too, e.g. they might be by different authors and its also interesting in
some senses to see what fixes were applied to the original patch, why
etc. since it is history and that is what the SCM effectively stores.

This is also happens on larger timescales, a commit goes into some tree,
some regression is found, some future commit fixes that regression,
sometimes over a kernel release or two or more.

My point is that this information can actually be useful and trying to
prune it all out the main tree for aesthetic reasons might not
necessarily be the right thing to do. I agree it can be distracting and
perhaps what we need are tools that can show or hide this kind of
information as an option.

Consider that -stable tree and that if commits were somehow marked as
regression fixes for previous commits, you could run some command and
get a list of regression fixes. I'm a realist and appreciate such output
would need careful manual/human consideration but it would have a real
world use.

On the other hand I agree that the patches in -mm often have stupid
typos etc which aren't interesting in the history but where do you draw
the line?

Re: If you want me to quit I will quit

Richard Purdie wrote:
> Its worth realising that these fix patches contain useful information
> too, e.g. they might be by different authors and its also interesting in
> some senses to see what fixes were applied to the original patch, why
> etc. since it is history and that is what the SCM effectively stores.

Important things can be noted in the patch description. We can also
refer to mailinglist archives and bug trackers in the patch description.

Caveats: Changing the patch description means changing the patch
identity (which different SCMs treat with different rigor). Also, a
number of SCM tools assume that there is exactly one author of a patch.
(Workaround: Track coauthors informally in the patch description.)

....
> On the other hand I agree that the patches in -mm often have stupid
> typos etc which aren't interesting in the history but where do you draw
> the line?

Re: If you want me to quit I will quit

* Linus Torvalds wrote:
> Because if it becomes normal coding practice, now people cannot work
> with you sanely any more (ie some random person pulls your tree for
> testing, and then I pull it at some other time, and the tester reports
> a problem, but now the commits he is talking about don't actually even
> exist in my tree any more, and it's all really messy!).

this is a real problem in practice, so i have a 'merge' repository of
all published x86.git trees of the past. About a 150 tree rebases later
it's still only 500 MB - so we know about the precise identity of every
published commit ID, even long after the fact.
> The x86 tree still does this. I absolutely detest it. Ingo claims that
> his model is better, and I'm pretty damn sure he's wrong. But until it
> starts causing bigger problems, I'll give him the benefit of the
> doubt.

if people basing stuff on x86.git becomes common and the rebasing
becomes an annoyance to them then we'll change this - we already changed
many other practices we had, according to feedback from developers.

Firstly, an architecture tree is kind of special, in that it's very,
very easy to introduce a regression that makes systems not boot up at
all. And that's a showstopper for developers. It's quite different from
a core kernel tree where we can often limp along with regressions. So by
far the most important thing that people value about our tree is
stability (or at least so we think).

So we do have a couple of fundamental things about our workflow that we
think developers value, and i'm not sure how to do that within Git.

For example the act of 'staging' various fixes, and in essence
bubble-sorting topics by the amount of fixes/problems they get. The
lifetime of topics within x86.git is like this, and this roots back to
the -rt queue which for years we maintained in a similar fashion:

- a new, nontrivial topic comes - say kmemcheck. First it's completely
experimental and we dont publish it at all - we keep it observed for a
week or so to get a feel of how much we want that topic and how stable
it is.

- the topic is in it's first stage, the most experimental 'testing'
phase. It's in x86.git/testing branch, right at the tail of every
other patch. It is merged after every existing topic and disturbs
nothing. If it breaks then it's easy to take it out - no harm done to
any other topic. New topics often bounce out and back 2-3 times before
they get past this initial stage of integration.

- the topic is in it's second stage - no high flux development anymore,
it can get into x86/for-akpm. That means it will go ahead of a lot of
other topics and that might mean merging work. It goes through another
level of review - Andrew is picky ;-)

- the topic is in its third stage - x86.git/for-linus - preparation for
push. It's merged ahead of all other topics, merge conflicts resolved.
Another, final review as well. We make sure Acks are collected for out
of tree stuff, etc.

if there's some problem with a topic at any stage, it usually bubbles
back up a level or two. Sometimes the problem is unknown and unfixed so
the topic gets removed completely from the tree (temporarily), until the
problem if found. Since we provide good bisection data, developers can
find problems fast. Sometimes there's fatal review feedback (say Andrew
reviews it too and finds it lacking that moves a topic back to the
'experimental' section completely.

But it gets even more complex: often we append fixes/updates not to the
topic, but to the tail of the queue. We wait a test cycle (24 hours or
overnight test) and when the changes look OK for public release we push
it out. Then when it's OK and enough updates batched up and we trust the
result we move those changes back to their specific topics - and resolve
any conflicts if needed. This basically mixes 'clean topic separation
and ordering' with a short-term 'append only' practice.

but all this inner-queue lifetime is almost invisible to developers -
all the reshuffling is done in an invariant way and we enforce that the
end result does not change when we flip around topics. They might see
topics disappearing occasionally, but given that x86.git carries up to a
dozen topics at a time or more, it's never a true disruption.

But we dont keep the queue public with a known breakage in it. I.e. we
merge easily, but we also find bugs agressively and revert easily. And
an important rule is that the moment we know about a breakage we fix it
immediately either by resolving the bug or by removing the topic, and
release a new version right then.

Another benefit to developers is that we do all inter-topic merges
ourselves. If we picked up two overlapping topics it's our task to
resolve the inevitable conflicts - we dont push that back to the
developers.

And the topic sorting of the queue becomes important in the merge window
as well: that's when the final decision is done whether to merge a topic
or not. This means that topics will again change order - and if there's
some dependency between them (which is very common), that dependency is
resolved.

Quilt gives this sort of workflow flexibility to us - but i never really
found a good Git workflow that matches this type of maintenance model.
But we are no Git wizards so suggestions would be welcome.

Re: If you want me to quit I will quit

On Sat, 2008-04-26 at 13:35 -0700, Linus Torvalds wrote:
> > Fact is, this is the way in which developers want to work. That is their
> > workflow, and their tools should follow their workflow. If a tool's
> > behaviour prevents them from implementing their desired workflow, it isn't
> > the workflow which should be changed
>
> But that was exactly my point. Bugs *will* happen. Follow-up patches
> *will* happen. Don't fight it. Do the best you can do - there's no way
> people will ever avoid all bugs to begin with.

Sorry if I drop in into a thread that's probably too big for me, but I
was having an idea (probably horribly wrong, but I try... don't beat too
hard). I think the main problem is when a fix to a change occurs quite
after the change went in, creating bisection problems (I've had
something like that bisecting an MMC problems some time ago).

Wouldn't be nice to have the concept of patch series in git? Maybe the
sha1 of the first commit of the patch? Then a fix on the series can be
marked as such, and you could teach git bisect to go first series by
series, applying/removing the fix patches as needed... and just at the
end dive into the commits. Maybe it's impossible/too complex/exponntial,
but well, it just a git git idea :-)

This communication contains confidential information. It is for the exclusive use of the intended addressee. If you are not the intended addressee, please note that any form of distribution, copying or use of this communication or the information in it is strictly prohibited by law. If you have received this communication in error, please immediately notify the sender by reply e-mail and destroy this message. Thank you for your cooperation.
--
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at http://vger.kernel.org/majordomo-info.html
Please read the FAQ at http://www.tux.org/lkml/