Cross-compiling fails.

Cross-compiling fails.

Cross-compiling sqlite3 fails due to the use of binaries in the build
process that are generated during the build process. Specifically:

mksourceid
lemon

The configure script explicitly offers the "--host=" option to allow
cross-compiling. If cross-compiling is not supported then this option
should be removed, and ideally replace with a short paragraph of text
explaining this explicitly.

An alternative is to add steps to the build instructions explaining that
you need to build these binaries natively first and then put them in the
correct location - however they will probably still get over-written by
cross-compiled versions when the cross-compile build starts so there may
need to be additional changes made.

In addition, when cross-compiling for Windows with the MinGW toolchain,
the code looks for the file "sys/ioctl.h" which doesn't exist for
Windows/MinGW systems. It looks like SQLITE_OS_UNIX is being incorrectly
defined. It doesn't look like a difficult problem to solve and if I get
a reply I will be happy to share any solution I find.

Re: Cross-compiling fails.

To answer your question using autoconf/configure terminology, my build
system is Linux and my host system is MinGW32. That is I am building on
Linux to generate libraries/executables to run under MinGW (on Windows
essentially). I would however face exactly the same issue if I was
building on Linux to run on, say, a RaspberryPi or an Android system.

To avoid confusion I recommend that you remove the option to
cross-compile unless you seriously intend to support it in which case
you will need to make some serious changes to your configure.ac script.
In any case you should remove the "--target" option as that is only used
when building compilers (I am building on system A (build) a compiler
which will run on system B (host) which will generate binaries that run
on system C (target)). The determination of the executable extension is
also broken with some confusion over "build" system extension and
"target" system extension. The script is incorrectly determining that
the "build" system extension is ".exe" by building an executable using
the cross-compiler. It then incorrectly determines that the target
extension is non-existent because it seems to somehow be assigning the
build executable extension to that variable. It then goes on to decide
that because of this the target OS is *nix and then SQLite fails to
compile because some platform compatibility code tries to include a *nix
specific header file. I solved this by simply deleting chunks of code in
the configure script and setting the target extension to be the same as
the build extension. However this is a hack and not a proper solution.

Also, I am not sure your suggestion of generating sqlite3.c natively and
then cross-compiling it separately will work. The generated sqlite3.c
has some platform dependent stuff (selective header inclusion as per
above) in it.

> On 12/4/17, Alastair Growcott <[hidden email]> wrote:
>> Cross-compiling sqlite3 fails due to the use of binaries in the build
>> process that are generated during the build process.
> I recommend that you do
>
> ./configure; make sqlite3.c
>
> or
>
> nmake /f makefile.msc sqlite3.c
>
> depending on whether your host system is unix or windows. Then
> manually compile the resulting sqlite3.c for your target platform.
>
> What are the host and target systems for the build you are trying to do?
>

Re: Cross-compiling fails.

On 12/6/17, Alastair Growcott <[hidden email]> wrote:
>
> To avoid confusion I recommend that you remove the option to
> cross-compile

I'm cool with that approach. Can you suggest a specific edit to the
configure.ac file that will accomplish what you describe?

>
> Also, I am not sure your suggestion of generating sqlite3.c natively and
> then cross-compiling it separately will work. The generated sqlite3.c
> has some platform dependent stuff (selective header inclusion as per
> above) in it.
>

The generated sqlite3.c file should be byte-for-byte identical on all
host platforms. We test for that. And we publish the SHA3 hash of
the generated sqlite3.c file in our release notes. If you find a pair
of host platforms that build different "sqlite3.c" files from the same
sources, then that is a bug. Please report it.

Re: Cross-compiling fails.

Trying to get it to work correctly:

BUILD_CC is correctly determined, but BUILD_EXEEXT is not (unless you
have a Cygwin system). On closer examination, the system is in fact
natively building lemon and mksourceid but because the extension is
wrong, you get an error like:

make: *** No rule to make target 'lemon', needed by 'fts5parse.c'. Stop.

Simply calculating BUILD_EXEEXT correctly should make this bit work, and
cross-compilation would be nice to have. I have now spent way way too
long, while being paid to do other stuff, looking at this. Sorry I can't
do more.

Also TARGET_EXEEXT is calculated correctly. In fact because you are not
building a compiler this value is not needed - simple substitute EXEEXT
everywhere you use TARGET_EXEEXT. E.g. change:

And in Makefile.in change @TARGET_EXEEXT@ to @EXEEXT@. These changes are
untested (no time).

Trying to remove cross-compilation:

At the top of configure.ac is the following text that I recommend you
edit for starters:

# The build process allows for using a cross-compiler. But the default
# action is to target the same platform that we are running on. The
# configure script needs to discover the following properties of the
# build and target systems:

Search the configure.ac for "cross_compiling". Change everything that
needs changing, or remove it. I notice that there is code there to
distinguish the build executable extension from the host one, but it
doesn't work. I spent a while that I should have spent on paid work
trying to figure out how to get that to work and in conclusion, it doesn't.

As far as the autoconf stuff goes, from the configure options it looks
like you have AC_CANONICAL_HOST used, but I only see it in aclocal.m4. I
don't know enough about autoconf to help here - I don't really know what
aclocal.m4 is used for, what it does.

Having re-examined sqlite3.c I think you are right that we can compile
it separately. I think the information needs to be put in a README
somewhere. It will need instructions on which defines and flags need to
be set. Normally configure sets that all up for you.

The way I built it in the end was as follows:

mkdir sqlite_linux
cd sqlite_linux
../sqlite/configure
Interrupt the build early, once lemon and mksourceid have been built
cd ..
mkdir sqlite_mingw32
cd sqlite_mingw32
Edit ../sqlite/configure around line 270 to replace:

However it does look like all I really needed to do was to create a
symlink or something from lemon to lemon.exe and the same for
mksourceid, plus of course fixing the calculation of TARGET_EXEEXT which
just needs to be the same as EXEEXT, or even better just use EXEEXT and
not TARGET_EXEEXT.

> On 12/6/17, Alastair Growcott <[hidden email]> wrote:
>> To avoid confusion I recommend that you remove the option to
>> cross-compile
> I'm cool with that approach. Can you suggest a specific edit to the
> configure.ac file that will accomplish what you describe?
>
>> Also, I am not sure your suggestion of generating sqlite3.c natively and
>> then cross-compiling it separately will work. The generated sqlite3.c
>> has some platform dependent stuff (selective header inclusion as per
>> above) in it.
>>
> The generated sqlite3.c file should be byte-for-byte identical on all
> host platforms. We test for that. And we publish the SHA3 hash of
> the generated sqlite3.c file in our release notes. If you find a pair
> of host platforms that build different "sqlite3.c" files from the same
> sources, then that is a bug. Please report it.
>

> On 12/6/17, Alastair Growcott <[hidden email]> wrote:
>> To avoid confusion I recommend that you remove the option to
>> cross-compile
> I'm cool with that approach. Can you suggest a specific edit to the
> configure.ac file that will accomplish what you describe?
>
>> Also, I am not sure your suggestion of generating sqlite3.c natively and
>> then cross-compiling it separately will work. The generated sqlite3.c
>> has some platform dependent stuff (selective header inclusion as per
>> above) in it.
>>
> The generated sqlite3.c file should be byte-for-byte identical on all
> host platforms. We test for that. And we publish the SHA3 hash of
> the generated sqlite3.c file in our release notes. If you find a pair
> of host platforms that build different "sqlite3.c" files from the same
> sources, then that is a bug. Please report it.
>