Use the file command. It will either
tell you 'mc68020 demand paged' if it is an a.out binary, and give a
longer (self-explanatory) description if it is ELF.

You can also use ldd. If it says
something about a "DLL jump", the binary is in a.out format;
otherwise, it's in ELF.

For statically linked binaries, ldd
reports statically linked (ELF) for ELF
binaries. I have no idea what it says for a.out binaries, because I
don't have anything statically linked a.out any more (at least, not
that I can find).