Commit Message

This introduces a sanity check for the toolchain, which verifies
each tuning (including any multilibs), producing meaningful diagnostics
for problems, and also provides some higher-level tuning features.
The TUNEVALID and TUNECONFLICT/TUNECONFLICTS settings were not
implemented, and there were some loose ends (like not knowing how
the conflict one was spelled). Listed one or two missing features
in TUNEVALID, also (in a previous patch) fixed the references to
features which didn't exist.
This patch also provides a whitelisting mechanism (which is completely
unused) to allow vendors providing prebuilt toolchain components to
restrict tunings to those based on or compatible with a particular ABI.
Signed-off-by: Peter Seebach <peter.seebach@windriver.com>
---
meta/classes/sanity.bbclass | 67 ++++++++++++++++++++++
meta/conf/documentation.conf | 6 ++
meta/conf/machine/include/README | 4 +
meta/conf/machine/include/arm/arch-armv5-dsp.inc | 1 +
meta/conf/machine/include/arm/arch-armv7a.inc | 2 +-
meta/conf/machine/include/ia32/arch-ia32.inc | 2 +-
meta/conf/machine/include/mips/arch-mips.inc | 6 +-
meta/conf/machine/include/tune-c3.inc | 2 +-
8 files changed, 84 insertions(+), 6 deletions(-)

On Mon, 30 Apr 2012 15:42:44 -0500
Mark Hatle <mark.hatle@windriver.com> wrote:
> Is the above debugging? (I suspect it is) I suggest the following...
Actually, that one's intentional. I originally did it for debugging,
but I thought it was super convenient to actually get a list of the CPU
feature set bitbake thought it was using. We can't tell you whether
you picked the right features, but we can tell you what they are.
So for instance, with a qemux86 and a core2 lib32:
NOTE: Sanity-checking tuning 'x86-64' (default) features:
NOTE: checking for conflicts: m64
NOTE: checking for conflicts with: m32
NOTE: m64: IA32e (x86_64) ELF64 standard ABI
NOTE: Sanity-checking tuning 'core2' (lib32) features:
NOTE: m32: IA32 ELF32 standard ABI
NOTE: core2: Enable core2 specific processor optimizations
I found this really handy, and left it there on purpose. I could take
it out, and/or move it to somewhere else, but I really do like having
that information appear.
-s

On Mon, 2012-04-30 at 15:48 -0500, Peter Seebach wrote:
> On Mon, 30 Apr 2012 15:42:44 -0500> Mark Hatle <mark.hatle@windriver.com> wrote:> > > Is the above debugging? (I suspect it is) I suggest the following...> > Actually, that one's intentional. I originally did it for debugging,> but I thought it was super convenient to actually get a list of the CPU> feature set bitbake thought it was using. We can't tell you whether> you picked the right features, but we can tell you what they are.> > So for instance, with a qemux86 and a core2 lib32:> > NOTE: Sanity-checking tuning 'x86-64' (default) features:> NOTE: checking for conflicts: m64> NOTE: checking for conflicts with: m32> NOTE: m64: IA32e (x86_64) ELF64 standard ABI> NOTE: Sanity-checking tuning 'core2' (lib32) features:> NOTE: m32: IA32 ELF32 standard ABI> NOTE: core2: Enable core2 specific processor optimizations> > I found this really handy, and left it there on purpose. I could take> it out, and/or move it to somewhere else, but I really do like having> that information appear.
I'd change these to debug messages. Users don't need this on the console
on every run. There is already a summary of the active tune options
displayed in the build configuration banner if I remember correctly.
Cheers,
Richard

On Tue, 1 May 2012 12:25:48 +0200
Koen Kooi <koen@dominion.thruhere.net> wrote:
> TUNE_FEATURES = "armv7a vfp neon cortexa8"
I'd like to suggest a separate patch which uses the TUNEVALID info
to explain these, and also explores the tunings of any selected
multilibs, because this is under-informative in some cases.
-s

On Tue, 1 May 2012 11:47:14 +0100
Richard Purdie <richard.purdie@linuxfoundation.org> wrote:
> > This patch also provides a whitelisting mechanism (which is> > completely unused) to allow vendors providing prebuilt toolchain> > components to restrict tunings to those based on or compatible with> > a particular ABI.
^-- this will be referred back to later
> > + if not tune or tune == "":
> "if not tune:"
Heh. I do about 80% of my scripting work in Lua and Ruby these
days. :)
> We'd usually have> > features = (data.getVar("TUNE_FEATURES_tune-%s" % tune, True) or> "").split() if not features:> return "Tuning '%s' has no defined features, and cannot be> used." % tune
Heh. And here, again, I'm used to things in which "".split() would
have given me an array with a single empty string back, same as any
other string with no spaces in it. :)
Looks like I have to do more experiments.
> > + valid_tunes = data.getVarFlags('TUNEVALID') or ""> > + conflicts = data.getVarFlags('TUNECONFLICTS') or ""
> Hmm, shouldn't this be "or {}" ?
Yes.
> > + whitelist = data.getVar("TUNEABI_WHITELIST", True) or ''
> So what is TUNEABI_WHITELIST?
See the last paragraph of the above.
> and what is TUNEABI_OVERRIDE? These should probably get the [doc] tags> you add for other things.
Good catch.
> To be honest I'm really tempted to split this tuneabi stuff out into a> different class. I appreciate the OSVs need something but it probably> doesn't make sense with half an implementation sitting in the core> like this, particularly when the code is near impenetrable like the> above. I'm left wondering what TUNEABI is too...
The big reason I'd advocate for having it in the base stuff is that
otherwise, all the OSVs are going to do their own and do it
incompatibly. The one thing worse than having one incomprehensible
thing is having many incomprehensible things with subtly different
semantics.
> > + bb.warn("Got an unexpected '%s' in MULTILIBS." %> > pairs[0])
> Shouldn't this get added to the errors list?
Good question! I was unable to find anything definite on the topic.
My thought is, if everything has to be multilib:, there's no reason for
the leading multilib: because it adds no information, so presumably
there logically *could* be something else, but I haven't seen any and
don't know about them.
> > + else:> > + if pairs[1] == 'lib':> > + tune_error_set.append("The multilib 'lib' was> > specified, but that causes parse errors.")
> Hmm, it does?
Yeah, that was the thing I ran into when I tried a default config
someone suggested; I thoughtfully corrected lib32 to lib, and libxcb
exploded with two pages of Python stack traces and diagnostics. If
that's not intentional, I can drop this check and we can treat it as a
package bug.
> What we really want to check here is that each pair[1] value is> unique.> Again, I'd just add this to the error list. If someone really wants to> do it, they can disable the checks. This should also happen if tune is> used more than once.
Okay. Good catch on that, hadn't thought of it.
> Just set these and skip "doc" in the above tests. Its ugly but I'd> prefer to have them set than unset.
Okay.
> Can you split this patch into two so we can get the bits that were> below in (they look fine) whilst we work on the above.
Yup.
-s

On Tue, 2012-05-01 at 11:23 -0500, Peter Seebach wrote:
> On Tue, 1 May 2012 11:47:14 +0100> Richard Purdie <richard.purdie@linuxfoundation.org> wrote: > > To be honest I'm really tempted to split this tuneabi stuff out into a> > different class. I appreciate the OSVs need something but it probably> > doesn't make sense with half an implementation sitting in the core> > like this, particularly when the code is near impenetrable like the> > above. I'm left wondering what TUNEABI is too...> > The big reason I'd advocate for having it in the base stuff is that> otherwise, all the OSVs are going to do their own and do it> incompatibly. The one thing worse than having one incomprehensible> thing is having many incomprehensible things with subtly different> semantics.> > > > + bb.warn("Got an unexpected '%s' in MULTILIBS." %> > > pairs[0])> > > Shouldn't this get added to the errors list?> > Good question! I was unable to find anything definite on the topic.> My thought is, if everything has to be multilib:, there's no reason for> the leading multilib: because it adds no information, so presumably> there logically *could* be something else, but I haven't seen any and> don't know about them.
Ah, I see what's happened here. This variable is used to append to
BBCLASSEXTEND where there are other values like "nativesdk" and
"native". Looking at the code, I think you should process
MULTILIB_VARIANTS.
The naming of the various multilib variables could probably do with
improvement in places :/.
Another good sanity check might be to look at the BBCLASSEXTEND values
and ensure they're in the list "native", "nativesdk", "multilib:xxx",
"cross".
> > > + else:> > > + if pairs[1] == 'lib':> > > + tune_error_set.append("The multilib 'lib' was> > > specified, but that causes parse errors.")> > > Hmm, it does?> > Yeah, that was the thing I ran into when I tried a default config> someone suggested; I thoughtfully corrected lib32 to lib, and libxcb> exploded with two pages of Python stack traces and diagnostics. If> that's not intentional, I can drop this check and we can treat it as a> package bug.
I think its a packaging bug. I'd be interested to see the error.
Cheers,
Richard

On Tue, 1 May 2012 21:17:28 +0100
Richard Purdie <richard.purdie@linuxfoundation.org> wrote:
> Ah, I see what's happened here. This variable is used to append to> BBCLASSEXTEND where there are other values like "nativesdk" and> "native". Looking at the code, I think you should process> MULTILIB_VARIANTS.
Ahh, okay.
> Another good sanity check might be to look at the BBCLASSEXTEND values> and ensure they're in the list "native", "nativesdk", "multilib:xxx",> "cross".
Hmm. Adding that as a separate commit.
> I think its a packaging bug. I'd be interested to see the error.
I'll try to get it to show. I'm still getting the x86_64x86_64 thing
first. :)
Should have cleaned up versions of this "soon".
-s