[BUG] Cannot push some grafted branches

[BUG] Cannot push some grafted branches

There seems to be some bad interactions between git-push and grafts.
The problem seems to occur when a commit that exists in the remote
repo is subject to a graft in the local repo, and we try to push one
of the fake parents.

The problem was first seen on 1.7.12.3 in a private repo, and I could
reproduce it using 1.8.1.rc0, as shown below. 1.7.10.4 seems even
more affected, with something looking like a memory corruption issue.

Re: [BUG] Cannot push some grafted branches

> There seems to be some bad interactions between git-push and grafts.
> The problem seems to occur when a commit that exists in the remote
> repo is subject to a graft in the local repo, and we try to push one
> of the fake parents.

History tweaking by grafts is only visible inside your local
repository and objects are not rewritten, and grafts are not
transferred across repositories. They were invented to be used as a
stop-gap measure until you filter-branch the history before
publishing (or if you do not publish, then you can keep using your
local grafts).

Isn't this well known? Perhaps we would need to document it better.

What you can do is to use "replace" instead and publish the replace
refs, I think. Object transfer will then follow the true parenthood
connectivity and people who choose to use the same replacement as
you do can fetch the replace ref from you (this will grab objects
necessary to complete the alternative history) and install it.

Re: [BUG] Cannot push some grafted branches

> Yann Dirson <[hidden email]> writes:
>
> > There seems to be some bad interactions between git-push and grafts.
> > The problem seems to occur when a commit that exists in the remote
> > repo is subject to a graft in the local repo, and we try to push one
> > of the fake parents.
>
> History tweaking by grafts is only visible inside your local
> repository and objects are not rewritten, and grafts are not
> transferred across repositories. They were invented to be used as a
> stop-gap measure until you filter-branch the history before
> publishing (or if you do not publish, then you can keep using your
> local grafts).
>
> Isn't this well known? Perhaps we would need to document it better.

I am well aware of that, and did not intend to push any grafted commit.
I am attempting to push a well-formed commit, which happens to be used as
a grafted commit's fake parent, and my interpretation is that git reacts
as if it was considering that the remote already had that commit, possibly
because it would not ignore grafts when deciding which commits are already
known to the remote.

> What you can do is to use "replace" instead and publish the replace
> refs, I think. Object transfer will then follow the true parenthood
> connectivity and people who choose to use the same replacement as
> you do can fetch the replace ref from you (this will grab objects
> necessary to complete the alternative history) and install it.

I am only using grafts as a temporary and lightweight drafting area,
before setting the results in stone - although in my case it will be
with filter-branch rather than replace, but the idea is the same. I just
got bitten when attempting to push a valid branch while the grafts were in
effect, when in fact they should have had no influence at all.

In fact, I even looked for a way to specify an alternate (or supplementary)
grafts file for this drafting work, so only well-controlled git invocations
would see them, whereas the others would just ignore them, and could not find
any - nor could I identify an existing way of disabling the use of grafts by
other means than moving it out of the way. In this respect, they seem to be
lacking a few features, when compared to "replace" refs, but they have different
uses, and just using the latter as a drafting area is just not adequate.

I thought about adding support for a GIT_GRAFTS_FILE envvar, which would
default to $GITDIR/info/grafts, or maybe with a more general addition of a
GIT_EXTRA_GRAFT_FILES envvar, but I'm not sure the latter would be that useful.

Re: [BUG] Cannot push some grafted branches

> In fact, I even looked for a way to specify an alternate (or supplementary)
> grafts file for this drafting work, so only well-controlled git invocations
> would see them, whereas the others would just ignore them, and could not find
> any - nor could I identify an existing way of disabling the use of grafts by
> other means than moving it out of the way. In this respect, they seem to be
> lacking a few features, when compared to "replace" refs, but they have different
> uses, and just using the latter as a drafting area is just not adequate.
>
> I thought about adding support for a GIT_GRAFTS_FILE envvar, which would
> default to $GITDIR/info/grafts, or maybe with a more general addition of a
> GIT_EXTRA_GRAFT_FILES envvar, but I'm not sure the latter would be that useful.

My bad on this point: there *is* a GIT_GRAFT_FILE envvar, it is just undocumented.
In fact it is not the only one:

Re: [BUG] Cannot push some grafted branches

> Yann Dirson <[hidden email]> writes:
>
> > .... In this respect, they seem to be
> > lacking a few features, when compared to "replace" refs, but they have different
> > uses, ...
>
> Not reallyl; grafts were old hack whose use is still supported with
> its original limitations; replace is meant to replace all uses of
> grafts while removing grafts' largest warts.

OK, I'll take this into account.

But this situation should probably be make more clear in the docs. Currently,
gitrepository-layout.txt describes refs/replace/ (and shallow) by reference to grafts,
and those are not marked as discouraged-use or anything.

And we may still want the bug fixed, or would we just list it as a known bug ?
At least it does not seem to occur with "replace" refs:

Re: [BUG] Cannot push some grafted branches

> Yann Dirson <[hidden email]> writes:
>
>> .... In this respect, they seem to be
>> lacking a few features, when compared to "replace" refs, but they have different
>> uses, ...
>
> Not reallyl; grafts were old hack whose use is still supported with
> its original limitations; replace is meant to replace all uses of
> grafts while removing grafts' largest warts.

I suppose there's the additional issue that grafts are much easier to
use than replacements if you really only want to replace some parent
lists. With replace you need to handcraft the replacement commits, and
git-replace(1) unhelpfully does not say this, much less gives an example
how to do it.

Re: [BUG] Cannot push some grafted branches

> And we may still want the bug fixed, or would we just list it as a known bug ?
> At least it does not seem to occur with "replace" refs:

The "replace" was designed to "fix" known limitation of grafts,
which is _inherent_ to it; the graft information was designed _not_
to be shared across repositories. The fix was done by by using a
different mechanism to allow propagating the information across
repositories.

So there is nothing further to fix, except that there is a documentation
bug you can fix if you didn't find it documented.

Re: [BUG] Cannot push some grafted branches

> Yann Dirson <[hidden email]> writes:
>
> > And we may still want the bug fixed, or would we just list it as a known bug ?
> > At least it does not seem to occur with "replace" refs:
>
> The "replace" was designed to "fix" known limitation of grafts,
> which is _inherent_ to it; the graft information was designed _not_
> to be shared across repositories. The fix was done by by using a
> different mechanism to allow propagating the information across
> repositories.

I see. But from what I observed (without looking at the source), it looks like
when determining which commits are to be pushed, the grafts file is not "neutralized"
as it should.

> So there is nothing further to fix, except that there is a documentation
> bug you can fix if you didn't find it documented.

Re: [BUG] Cannot push some grafted branches

> Junio C Hamano <[hidden email]> writes:
>
> > Yann Dirson <[hidden email]> writes:
> >
> >> .... In this respect, they seem to be
> >> lacking a few features, when compared to "replace" refs, but they have different
> >> uses, ...
> >
> > Not reallyl; grafts were old hack whose use is still supported with
> > its original limitations; replace is meant to replace all uses of
> > grafts while removing grafts' largest warts.
>
> I suppose there's the additional issue that grafts are much easier to
> use than replacements if you really only want to replace some parent
> lists. With replace you need to handcraft the replacement commits, and
> git-replace(1) unhelpfully does not say this, much less gives an example
> how to do it.
>

Right, replace refs can surely be made easier to use. The requirement to craft a
new commit manually is a major step back in ease of use.

Maybe something like "git replace -p <orig-commit> <parent>..." to just provide a simple
API to the exact graft functionnality would be good. But it would be commit-specific, whereas
replace refs are indeed more generic, and, one could want to rewrite any other part of the commit,
so we could prefer a more general mechanism.

Something that could be useful in this respect, would be an --amend like option to git-commit, like
"git commit --replace". But unfortunately it does not allow to change parents, and it has the
drawback of requiring that HEAD points to the commit to be replaced.

So maybe, if there are no other idea, a simple "git graft" command that would wrap "git replace",
would fill the gap.

Re: [BUG] Cannot push some grafted branches

> On Mon, 17 Dec 2012 09:43:53 +0100
> Thomas Rast <[hidden email]> wrote:
>
>> Junio C Hamano <[hidden email]> writes:
>>
>>
>> I suppose there's the additional issue that grafts are much easier to
>> use than replacements if you really only want to replace some parent
>> lists. With replace you need to handcraft the replacement commits, and
>> git-replace(1) unhelpfully does not say this, much less gives an example
>> how to do it.
>>
>
> Right, replace refs can surely be made easier to use. The requirement to craft a
> new commit manually is a major step back in ease of use.

Yeah, at one point I wanted to have a command that created to craft a
new commit based on an existing one.
Perhaps it could be useful when using filter-branch or perhaps it
could reuse some filter-branch code.

> Maybe something like "git replace -p <orig-commit> <parent>..." to just provide a simple
> API to the exact graft functionnality would be good. But it would be commit-specific, whereas
> replace refs are indeed more generic, and, one could want to rewrite any other part of the commit,
> so we could prefer a more general mechanism.

Yeah I wondered at one point if something like the following would do:

> Something that could be useful in this respect, would be an --amend like option to git-commit, like
> "git commit --replace". But unfortunately it does not allow to change parents, and it has the
> drawback of requiring that HEAD points to the commit to be replaced.
>
> So maybe, if there are no other idea, a simple "git graft" command that would wrap "git replace",
> would fill the gap.

It would not be straightforward to call it "graft" if it uses git replace.

Re: [BUG] Cannot push some grafted branches

> Hi Yann,
>
> On Mon, Dec 17, 2012 at 11:40 AM, Yann Dirson <[hidden email]> wrote:
> > On Mon, 17 Dec 2012 09:43:53 +0100
> > Thomas Rast <[hidden email]> wrote:
> >
> >> Junio C Hamano <[hidden email]> writes:
> >>
> >>
> >> I suppose there's the additional issue that grafts are much easier to
> >> use than replacements if you really only want to replace some parent
> >> lists. With replace you need to handcraft the replacement commits, and
> >> git-replace(1) unhelpfully does not say this, much less gives an example
> >> how to do it.
> >>
> >
> > Right, replace refs can surely be made easier to use. The requirement to craft a
> > new commit manually is a major step back in ease of use.
>
> Yeah, at one point I wanted to have a command that created to craft a
> new commit based on an existing one.
> Perhaps it could be useful when using filter-branch or perhaps it
> could reuse some filter-branch code.
>
> > Maybe something like "git replace -p <orig-commit> <parent>..." to just provide a simple
> > API to the exact graft functionnality would be good. But it would be commit-specific, whereas
> > replace refs are indeed more generic, and, one could want to rewrite any other part of the commit,
> > so we could prefer a more general mechanism.
>
> Yeah I wondered at one point if something like the following would do:
>
> git replace --parent <parent1> --parent <parent2> --author <author>
> --commiter <commiter> ... <orig-commit>

Yes, modification flags, that would only be allowed when the objects are commits,
and would cause creation of a replace commit that's <orig-commit> plus modifications.
We could then reuse the relevant options from git-commit, and add the missing --parent.

But wouldn't it stretch git-replace too much, to add commit-specific behaviour there ?

> > Something that could be useful in this respect, would be an --amend like option to git-commit, like
> > "git commit --replace". But unfortunately it does not allow to change parents, and it has the
> > drawback of requiring that HEAD points to the commit to be replaced.
> >
> > So maybe, if there are no other idea, a simple "git graft" command that would wrap "git replace",
> > would fill the gap.
>
> It would not be straightforward to call it "graft" if it uses git replace.

Well, "git replace" would just be the "implementation detail". The idea would be to keep
the concept of a "graft", and just change its implementation. If we care (and we surely
do not, it's just a thought experiment ;), we could even provide, for pre-replace gits, a
"git graft" implementation that would manipulate info/grafts, together with a docpatch
saying that direct manipulation of info/grafts is deprecated.

Re: [BUG] Cannot push some grafted branches

> Christian Couder <[hidden email]> writes:
>
>> Yeah, at one point I wanted to have a command that created to craft a
>> new commit based on an existing one.
>
> This isn't hard to do, you only have to resort to plumbing:
>
> $ git cat-file commit fef11965da875c105c40f1a9550af1f5e34a6e62 | sed s/bfae342c973b0be3c9e99d3d86ed2e6b152b4a6b/790c83cda92f95f1b4b91e2ddc056a52a99a055d/ | git hash-object -t commit --stdin -w
> bb45cc6356eac6c7fa432965090045306dab7026

Well, I'm not sure this is intuitive enough to be useful to the average user :)
Adding git-rev-parse calls for convenience, and calling git-replace, would make it
a more complete recipe, and we could suggest that as an alias in the collection that's
in the wiki (which is not even linked any more from git-scm.com btw), but imho that
would be hiding valuable information in a dark corner.

Anyway, in this form it will only replace a parent with another, whereas a full
graft replacement should allow to write a different number of new parents instead.
That is, instead of this simple sed, something like:

Well, a short bash script should be more readable and possibly faster, but that's the
idea. Such a script could be a candidate for contrib ?
--
Yann Dirson - Bertin Technologies
--
To unsubscribe from this list: send the line "unsubscribe git" in
the body of a message to [hidden email]More majordomo info at http://vger.kernel.org/majordomo-info.html

Re: [BUG] Cannot push some grafted branches

> On Mon, 17 Dec 2012 13:14:56 -0800
> Junio C Hamano <[hidden email]> wrote:
>
>> Andreas Schwab <[hidden email]> writes:
>>
>>> Christian Couder <[hidden email]> writes:
>>>
>>>> Yeah, at one point I wanted to have a command that created to craft a
>>>> new commit based on an existing one.
>>>
>>> This isn't hard to do, you only have to resort to plumbing:
>>>
>>> $ git cat-file commit fef11965da875c105c40f1a9550af1f5e34a6e62 | sed s/bfae342c973b0be3c9e99d3d86ed2e6b152b4a6b/790c83cda92f95f1b4b91e2ddc056a52a99a055d/ | git hash-object -t commit --stdin -w
>>> bb45cc6356eac6c7fa432965090045306dab7026
>>
>> Good. I do not think an extra special-purpose command is welcome
>> here.
>
> Well, I'm not sure this is intuitive enough to be useful to the average user :)

When I played with git-replace in the past, I imagined that it could be

git replace <object> --commit ...commit options...

that would do the trick.

We could implement it with a git-replace--commit helper script that
generates the replacement commit using the ...commit options... (to be
defined what this should be), and git-replace would just pick its output
(the SHA1 of the generated commit) as a substitute for the <replacement>
argument that would have to be given without the --commit option.

Re: [BUG] Cannot push some grafted branches

> Am 12/18/2012 12:00, schrieb Yann Dirson:
>> On Mon, 17 Dec 2012 13:14:56 -0800
>> Junio C Hamano <[hidden email]> wrote:
>>
>>> Andreas Schwab <[hidden email]> writes:
>>>
>>>> Christian Couder <[hidden email]> writes:
>>>>
>>>>> Yeah, at one point I wanted to have a command that created to craft a
>>>>> new commit based on an existing one.
>>>>
>>>> This isn't hard to do, you only have to resort to plumbing:
>>>>
>>>> $ git cat-file commit fef11965da875c105c40f1a9550af1f5e34a6e62 |
>>>> sed
>>>> s/bfae342c973b0be3c9e99d3d86ed2e6b152b4a6b/790c83cda92f95f1b4b91e2ddc056a52a99a055d/
>>>> | git hash-object -t commit --stdin -w
>>>> bb45cc6356eac6c7fa432965090045306dab7026
>>>
>>> Good. I do not think an extra special-purpose command is welcome
>>> here.
>>
>> Well, I'm not sure this is intuitive enough to be useful to the average user :)
>
> When I played with git-replace in the past, I imagined that it could be
>
> git replace <object> --commit ...commit options...
>
> that would do the trick.
>
> We could implement it with a git-replace--commit helper script that
> generates the replacement commit using the ...commit options... (to be
> defined what this should be), and git-replace would just pick its output
> (the SHA1 of the generated commit) as a substitute for the <replacement>
> argument that would have to be given without the --commit option.

I wouldn't even want a script -- we'd end up inventing a complicated
command-line editor for what can simply be done by judicious use of an
actual text editor. How about something like the following?

+
+EXAMPLE
+-------
+
+Replacements (and before them, grafts) are often used to replace the
+parent list of a commit. Since commits are stored in a human-readable
+format, you can in fact change any property using the following
+recipe:
+
+------------------------------------------------
+$ git cat-file commit original_commit >tmp
+$ vi tmp
+------------------------------------------------
+In the editor, adjust the commit as needed. For example, you can edit
+the parent lists by adding/removing lines starting with "parent".
+When done, replace the original commit with the edited one:
+------------------------------------------------
+$ git replace original_commit $(git hash-object -w tmp)
+------------------------------------------------
+
+
BUGS
----
Comparing blobs or trees that have been replaced with those that

Re: [BUG] Cannot push some grafted branches

> Johannes Sixt <[hidden email]> writes:
>
> > Am 12/18/2012 12:00, schrieb Yann Dirson:
> >> On Mon, 17 Dec 2012 13:14:56 -0800
> >> Junio C Hamano <[hidden email]> wrote:
> >>
> >>> Andreas Schwab <[hidden email]> writes:
> >>>
> >>>> Christian Couder <[hidden email]> writes:
> >>>>
> >>>>> Yeah, at one point I wanted to have a command that created to craft a
> >>>>> new commit based on an existing one.
> >>>>
> >>>> This isn't hard to do, you only have to resort to plumbing:
> >>>>
> >>>> $ git cat-file commit fef11965da875c105c40f1a9550af1f5e34a6e62 |
> >>>> sed
> >>>> s/bfae342c973b0be3c9e99d3d86ed2e6b152b4a6b/790c83cda92f95f1b4b91e2ddc056a52a99a055d/
> >>>> | git hash-object -t commit --stdin -w
> >>>> bb45cc6356eac6c7fa432965090045306dab7026
> >>>
> >>> Good. I do not think an extra special-purpose command is welcome
> >>> here.
> >>
> >> Well, I'm not sure this is intuitive enough to be useful to the average user :)
> >
> > When I played with git-replace in the past, I imagined that it could be
> >
> > git replace <object> --commit ...commit options...
> >
> > that would do the trick.
> >
> > We could implement it with a git-replace--commit helper script that
> > generates the replacement commit using the ...commit options... (to be
> > defined what this should be), and git-replace would just pick its output
> > (the SHA1 of the generated commit) as a substitute for the <replacement>
> > argument that would have to be given without the --commit option.
>
> I wouldn't even want a script -- we'd end up inventing a complicated
> command-line editor for what can simply be done by judicious use of an
> actual text editor. How about something like the following?

Well, while it does the job, it is still hardly as straightforward as the
old "vi .git/info/grafts", or as a single easily-remembered commandline.

I was again thinking the only commandline stuff that does not exist currently in
git-commit is specifying parents. One possiblity would be to add such an
option to git-commit, together with a --replace flag that would cause the
new commit to attached a replace ref (not completely unlike --append, in that
we're doing some non-default action instead of just adding the changes to a
new commit).

But well, I don't think we would want to add to git-commit the ability of playing
with something else than what's in the index/worktree. Abstracting the commit
commandline to make it reusable by a git-replace--commit and possibly other tools
that may want to rw-manipulate arbitrary commits could make sense ?

I do not understand why you even want to go in the harder route in
the first place, only to complicate things?

All you want to do is to craft a commit object that records a
specific tree shape, has a set of parents you want, and has the log
information you want. Once you have the commit, you can replace an
unwanted commit with it.

----A----B----o---- ....

X----Y----Z---- ....

Suppose you want to pretend that X is a child of A, even though it
is not in the real life. So you want to create a commit that

- has the same tree as X;
- has A as its parent; and
- records log and authorship of X.

and then use "git replace" to replace X, right? How about doing it
this way?

$ git checkout X^0 ;# detach
$ git reset --soft A
$ git commit -C X

The first gives you the index and the working tree that is the same
as X, the second moves HEAD while keeping the index and the working
tree so that the commit you create will be a child of A, and the
last makes that commit with the metainformation from X [*1*]. If
you want, you can even tweak the contents of the tree before making
the commit in the final step, or tweak the log message during the
final step.

that is, find an existing commit B that has the desired set of
parents, and amend it with the same tree and the metainformation as
X. This would even work when you want to come up with a commit that
replaces a merge. For example, if you want to pretend that B were a
merge between A and X in the above topology, you could

which would create one merge that has the desired set of parents
(i.e. A and X) in the first two steps on temp branch, prepares the
index and the working tree to match the tree of B, and with that
tree and the metainformation from B, amends that merge. The
resulting commit will be a merge between A and X that has the tree
of B and metainformation of B (with a chance to edit it further, as
I used -c there).

Is this not intuitive enough?

[Footnote]

*1* If you are not tweaking the tree contents, you can do this
all in the index without affecting the working tree, e.g.

$ git checkout HEAD^0 ;# totally random state unrelated to X nor A
$ git read-tree X ;# just update the index to match tree of X
$ git reset --soft A ;# next commit will be child of A
$ git commit -C X ;# and with metainformation from X

After you are done, you can "read-tree $branch" followed by
"checkout $branch" to come back to where you were.
--
To unsubscribe from this list: send the line "unsubscribe git" in
the body of a message to [hidden email]More majordomo info at http://vger.kernel.org/majordomo-info.html