RFC: dpkg patch to using -ffile-prefix-map

RFC: dpkg patch to using -ffile-prefix-map

With gcc 8 being the default compiler in debian now, we should be able
to use -ffile-prefix-map, which should handle *some* of the cases that
BUILD_PATH_PREFIX_MAP is intended to solve. It definitely won't help
with things that embed the gcc commandline into arguments.

But since we've been unable to convince gcc on the merits of
BUILD_PATH_PREFIX_MAP using an environment variable, rebasing the gcc
patches to support it takes a lot of effort, perhaps we should explore
other options.

I've got a dpkg patch which makes use of -ffile-prefix-map. I haven't
found a good test case yet... package that fails to build reproducibly
due to using __FILE__, __BASE_FILE__, or __builtin_FILE() and builds
fast enough that it's easy to test (holger suggested trying "dtkwm", but
I haven't had a chance to try yet). Figured I'd at least publish my
patch to get some review and/or testers.

The patch largely just copy-and-pastes the -fdebug-prefix-map code,
though arguably could obsolete it entirely, since -ffile-prefix-map
effectivly sets both -fdebug-prefix-map and -fmacro-prefix-map. But
having the two features independently allows enabling or disabling one
or the other easily for now.

If nothing else, carrying a patch on dpkg builds *much* faster than gcc,
and rebasing it periodically will be a lot less effort. Though if it
works, hopefully we can get it into dpkg directly.

Re: RFC: dpkg patch to using -ffile-prefix-map

Hi!

On Tue, 2018-07-31 at 08:29:33 +0800, Vagrant Cascadian wrote:

> With gcc 8 being the default compiler in debian now, we should be able
> to use -ffile-prefix-map, which should handle *some* of the cases that
> BUILD_PATH_PREFIX_MAP is intended to solve. It definitely won't help
> with things that embed the gcc commandline into arguments.
>
> But since we've been unable to convince gcc on the merits of
> BUILD_PATH_PREFIX_MAP using an environment variable, rebasing the gcc
> patches to support it takes a lot of effort, perhaps we should explore
> other options.
>
> I've got a dpkg patch which makes use of -ffile-prefix-map. I haven't
> found a good test case yet... package that fails to build reproducibly
> due to using __FILE__, __BASE_FILE__, or __builtin_FILE() and builds
> fast enough that it's easy to test (holger suggested trying "dtkwm", but
> I haven't had a chance to try yet). Figured I'd at least publish my
> patch to get some review and/or testers.

Hah! Just several hours ago I was talking with Mattia over IRC about
the status of this, and ended up cooking the attached patch, which at
his request didn't merge, because he wanted to give it a try first.

And yeah, I also kind of understand gcc's upstream position, that if
you unconditionally embed all build flags into your resulting objects,
then they are really bound to be unreproducible, and as such that's
arguably a bug to fix there probably.

Also AIUI this divergence and the lack of forward porting is the
reason the repro buildds are currently stopped? If so I think getting
something going, even if it might produce some new (or old :)
reproducible problems is probably better than the current situation.

> The patch largely just copy-and-pastes the -fdebug-prefix-map code,
> though arguably could obsolete it entirely, since -ffile-prefix-map
> effectivly sets both -fdebug-prefix-map and -fmacro-prefix-map. But
> having the two features independently allows enabling or disabling one
> or the other easily for now.

I think the semantics in mine are slightly better? :)

> If nothing else, carrying a patch on dpkg builds *much* faster than gcc,
> and rebasing it periodically will be a lot less effort. Though if it
> works, hopefully we can get it into dpkg directly.

From my side, I see no problem with merging something like the
attached patch if it works (I've not even run the test suite on it I
think :).

Re: RFC: dpkg patch to using -ffile-prefix-map

> On Tue, 2018-07-31 at 08:29:33 +0800, Vagrant Cascadian wrote:
>> I've got a dpkg patch which makes use of -ffile-prefix-map. I haven't
>> found a good test case yet... package that fails to build reproducibly
>> due to using __FILE__, __BASE_FILE__, or __builtin_FILE() and builds
>> fast enough that it's easy to test (holger suggested trying "dtkwm", but
>> I haven't had a chance to try yet). Figured I'd at least publish my
>> patch to get some review and/or testers.
>
> Hah! Just several hours ago I was talking with Mattia over IRC about
> the status of this, and ended up cooking the attached patch, which at
> his request didn't merge, because he wanted to give it a try first.

Heh.

> And yeah, I also kind of understand gcc's upstream position, that if
> you unconditionally embed all build flags into your resulting objects,
> then they are really bound to be unreproducible, and as such that's
> arguably a bug to fix there probably.

I get that argument, though I fear that becomes an eternal game of
whack-a-mole.

> Also AIUI this divergence and the lack of forward porting is the
> reason the repro buildds are currently stopped? If so I think getting
> something going, even if it might produce some new (or old :)
> reproducible problems is probably better than the current situation.

Agreed.

>> The patch largely just copy-and-pastes the -fdebug-prefix-map code,
>> though arguably could obsolete it entirely, since -ffile-prefix-map
>> effectivly sets both -fdebug-prefix-map and -fmacro-prefix-map. But
>> having the two features independently allows enabling or disabling one
>> or the other easily for now.
>
> I think the semantics in mine are slightly better? :)

Surely! Glad to see you've done it properly.

>> If nothing else, carrying a patch on dpkg builds *much* faster than gcc,
>> and rebasing it periodically will be a lot less effort. Though if it
>> works, hopefully we can get it into dpkg directly.
>
> From my side, I see no problem with merging something like the
> attached patch if it works (I've not even run the test suite on it I
> think :).

So by default, it would be disabled initially, and then we could
explicitly enable this for the reproducible builds test framework? After
it proves to be working and useful and not disruptive, I presume we
would enable it by default?

Re: RFC: dpkg patch to using -ffile-prefix-map

On Tue, 2018-07-31 at 10:40:08 +0800, Vagrant Cascadian wrote:
> On 2018-07-31, Guillem Jover wrote:
> > On Tue, 2018-07-31 at 08:29:33 +0800, Vagrant Cascadian wrote:
> > And yeah, I also kind of understand gcc's upstream position, that if
> > you unconditionally embed all build flags into your resulting objects,
> > then they are really bound to be unreproducible, and as such that's
> > arguably a bug to fix there probably.
>
> I get that argument, though I fear that becomes an eternal game of
> whack-a-mole.

Right. I guess there's also an argument there, that if you are using a
blacklist then maybe you are doing it wrong? Or doing it at all? Dunno
really. :)

Yeah. That was at Mattia's request too. I'm not sure I'd mind much
setting it by default from the get go. But it might be wiser to let
the repro buildds crunch on this for a bit in case unrelated things
break. :)

> > So by default, it would be disabled initially, and then we could
> > explicitly enable this for the reproducible builds test framework? After
> > it proves to be working and useful and not disruptive, I presume we
> > would enable it by default?
> Yeah. That was at Mattia's request too. I'm not sure I'd mind much
> setting it by default from the get go.

cool too!

> But it might be wiser to let
> the repro buildds crunch on this for a bit in case unrelated things
> break. :)

sure! We've been running this code since last Saturday... so in a while
we should know for sure :)

Re: RFC: dpkg patch to using -ffile-prefix-map

On 2018-08-07, Holger Levsen wrote:

> On Tue, Jul 31, 2018 at 04:54:09AM +0200, Guillem Jover wrote:
>> > So by default, it would be disabled initially, and then we could
>> > explicitly enable this for the reproducible builds test framework? After
>> > it proves to be working and useful and not disruptive, I presume we
>> > would enable it by default?
>> Yeah. That was at Mattia's request too. I'm not sure I'd mind much
>> setting it by default from the get go.
>
> cool too!
>
>> But it might be wiser to let
>> the repro buildds crunch on this for a bit in case unrelated things
>> break. :)
>
> sure! We've been running this code since last Saturday... so in a while
> we should know for sure :)

Which has been over a week now, though slightly modified to default to
enabling fixfilepath.

I've seen it fix at least one reproducibility issue(zbackup), and have
seen the correct flag passed in builds, so I can confirm that it's
working. I haven't seen obvious examples where it breaks anything,
though admittedly haven't looked very closely.

It'd be great to at least see it in dpkg in a default disabled state
soon, and then the test infrastructure can set the environment variable
DEB_BUILD_MAINT_OPTIONS=reproducible=+fixfilepath (or reproducible=+all
? I *think* that's the right syntax?) so the test infrastructure to
enables it...

Would it be safe to set that in the test infrastructure even when dpkg
has no support for it; does dpkg ignore unknown features?

The less conservative option would, of course, be to enable it by
default in sid... :)