As everyone knows, the Empire at War Lua files are some weird Lua variant.
Some while ago, I figured out what exactly made these lua files 'weird':

The signature is "\033Lup" instead of "\033Lua"

The indicated version is 0x51 instead of 0x50 (yes, the files are actually in the 5.0 format)

Every function contains an additional integer, after the "lineDefined" integer. I don't know what this integer is for, but it doesn't seem to serve any real purpose when reading the files.

So, read a EaW Lua file (or, as I like to call them thanks to their signature: Lup files), change the signature, change the version and remove the extra integers and you're left with a perfectly valid Lua 5.0 file.

Unfortunately, this is a long way from the Lua source code. I am currently aware of two decompilers: LuaDC and Luadec. Neither of which succesfully decompiled the files.

LuaDC complained about invalid filenames and what not when I started it, and without the source I have no idea what's wrong with it (and I didn't give up easily).
Luadec was able to read the files but it crashed on decompilation. As it turned out (yay for open-source), luadec assumes the information about a function's locals is present, whereas it is, in fact, not (the locals information is optional, and Petroglyph apparently decided to strip that. Can't blame them).

After some l33t hacking :p in luadec I got to work around this by tricking it into thinking it had locals information. Now it actually produced Lua code from the most basic of Lua files. However, when decompiling the more complicated Lua files luadec is unable to make sense of them, leaving me less then impressed with this decompiler.

I've even gone so far as to take a pot shot at writing a decompiler myself, but proper decompilation is hard, even for a 'simple' language such as Lua. Well, it is for me anyway.

On the other hand, I can of course output Lua assembly from the lua files; that's easy to do. But then this has to be manually decompiled into normal Lua code, something I'm sure most modders can't, won't (or even shouldn't) do.

Anyway, I just thought I'd put this information on the forums. Maybe someone who does know a thing or two about decompilation can use this info, or perhaps point me to, or adapt, another existing decompiler.

If you could figure out this one it will save us alot of trouble especialy with the AI.

Thank you for you findings so far. COntinue the good job.

Orao

04-18-2006 10:02 AM

BTW I just forgot to ask you. What files do you edit for scripted action on the welcome screen ?

Thank you.

Admiral Sith

04-18-2006 12:26 PM

Thats in the gameconstants.xml, its actually a replay of your most recent recorded game, a feature that was, unfortunatly, removed. So you get stuck with that same one.

Slocket

04-19-2006 02:30 AM

Well Mike good job so far! This is the REAL editor I am looking for. Can you give us a version of the LuaDec you modified?

Quote:

On the other hand, I can of course output Lua assembly from the lua files; that's easy to do. But then this has to be manually decompiled into normal Lua code, something I'm sure most modders can't, won't (or even shouldn't) do.

I programmed way to much in assembler decades ago when computers were very 'dumb'. Even the decompiled asssembler code can be of great use to do basic modification of integer -say what the AI will use for a raid fleet total number or what it thinks it needs to attack your fleet.

Then by going into the regular files, it can be hexed in. I did this for the DLL in Freelancer to get around some annoying behaviour to allow playing the game in single player mode without the storyline getting into the way.

So if you can give out a copy, I think quite abit of simple modding of the scripts can be done to some extent using the assembler code as a guide.

This well give the game a boost by making the AI smarter.

Slocket

04-19-2006 03:28 AM

Hey, the old demo is written slightly different than the final game release Lup. I manage to follow you advice for the old demo file Lua 5.0 and it decompiled quite a bit until it crashed! lol

I thought you may find this interesting. I only used LuaDec on BuildTurboLaser.lua of the old demo. I change the header from Lua(51) to Lua(50). No other change yet and it decompiled the bugger until it hit the 'end' or that integer thingy. The other thing it seems to have a problem with handling End of Line.

If you can show me what you did with the LuaDec I think with this information; you will have a decompiler of some sort.

I've written a small (command-line) tool that will convert Lup files to Lua files and back: lup2lua (and its source).

Second, here are my (small) modifications to luadec: luadec-0.6-mod.
I only changed two things in the original file, and those changes are marked by my name, so you should find them easily. Running the resulting luadec over BuildTurboLasers.lua results in:

Yet, running it on larger, more complex lua files will crash the decompiler as usual. I've tried to find out why, but I don't understand the laudec code that well and it seems like more than just a superficial problem.

Anyway, I'll create a lua disassembler over the next days, if that'll help you guys.

Slocket

04-25-2006 09:03 PM

Sure, please do. I do not think we will get a LUA editor for now. With a disassembler, we can (the hard way) change any small aspect of the game that many modders may like.

I for one would like to change the way the DeathStar Battle is played out. If it means to manually recreating the script from assembler -so be it. It would only be a few hours work to type up the text script. I assume the EXE will read the expanded form of it.

Thanks for all your hard work Mike!

squeegee

05-01-2006 05:53 PM

Quote:

Originally Posted by Mike.nl

Anyway, I'll create a lua disassembler over the next days, if that'll help you guys.

I had just used winhex to update the sig bytes before I found out about the extra integer's and thus luadec's failure to decompile. Decided to google at that point, and glad I did!

I've written several decompilers over the years, so I'll see what I can figure out about the crash with luadec.

Thanks for the effort you've put into this! What an awesome place to start!

popcorn2008

05-01-2006 06:13 PM

Good work Mike! It would be great to have a community made LUA deassembler until/if Petro releases their own.

Foshjedi2004

05-01-2006 06:44 PM

LoW's Coder Alisama is busy doing Model File Comparisons So We may have a Filefront Community Model Converter and Bone Editor

squeegee

05-03-2006 02:51 PM

I figured out the luadec crash: It's because of the lack of local information.

I coded around it by using _TEMP_ for all local variable names. At least that got me able to decompile everything I've tried it on so far, but make no mistake, it's a kluge (for example, if there are 5 locals, all 5 are named _TEMP_ - not good!).

Email me if you want a copy of the modified source (no place to put it online...).

I will work on a proper solution to the local variable issue next. I just have to do some heavy code reading to understand luadec parse structure...

Athanasios

05-03-2006 03:49 PM

Quote:

Originally Posted by squeegee

I figured out the luadec crash: It's because of the lack of local information.

I coded around it by using _TEMP_ for all local variable names. At least that got me able to decompile everything I've tried it on so far, but make no mistake, it's a kluge (for example, if there are 5 locals, all 5 are named _TEMP_ - not good!).

Email me if you want a copy of the modified source (no place to put it online...).

I will work on a proper solution to the local variable issue next. I just have to do some heavy code reading to understand luadec parse structure...

Can't you set a generator for random naming of local variables?

I have no clue of LUA coding, but in C, C++, Java and such this is doable. Either with matrices of strings or pointers.

squeegee

05-03-2006 04:15 PM

Quote:

Originally Posted by Athanasios

Can't you set a generator for random naming of local variables?

I have no clue of LUA coding, but in C, C++, Java and such this is doable. Either with matrices of strings or pointers.

Actually, my original kluge was pretty stupid. As it turns out, all the crashes I've come across are related to the local vars. All I had to do was before using a local variable check for it being null, and if it was, call the function Mike created. The only other change was to make sure not to free the auto locals on function closure.

So, now, all local variables have distinct names of L0, L1, L2, etc. The best we can do without the local variable information (i.e. not knowing the actual local variable names). The code is quite readable, and I have successfully decompiled all the lua code in config.meg now.

There is one other strange bug in luadec: When a function is declared, the line of code generated is like "name = function()" instead of "function name()". It's annoying, but the code is still readable. I'll look into eventually...

I can make a zip of all the decompiled sources if anybody is interested. As well as the fixed luadec and code if interested.

popcorn2008

05-03-2006 04:50 PM

If you could do us the favor and make a zip of the decompiled scripts and modified luadec it would be great.

It could give us a head start on using Lua.

Athanasios

05-03-2006 04:55 PM

I would suggest to work closely with Mike, so that you break Lua till the end. So that we can even compile models too.

Yet, in the meanwhile, you can upload what's you've done. Just mind to post this to PFF forums too, there are people interested for this there too!

squeegee

05-03-2006 05:18 PM

Quote:

Originally Posted by Athanasios

I would suggest to work closely with Mike, so that you break Lua till the end. So that we can even compile models too.

Yet, in the meanwhile, you can upload what's you've done. Just mind to post this to PFF forums too, there are people interested for this there too!

Agreed, just waitng for a response. luadec has some internal problems, so I am looking into the actual structure of tokenized lua now to see if it's easier to start over or attempt to fix luadec.

From my perspective, luadec is more complicated than it needs to be (it appears as though it actually attempts to walk the code by pseudo-executing it, which is, uh, not the way I'd do it...).

As I mentioned before, I've written several deompilers so I am quite familiar with how to get one up and running quickly, and correctness is the most important thing for a decompiler IMO, and luadec still bails on some simple valid structures (which I just noticed the failure messages in a few of the files I decompiled - although 13 failures out of 260 isn't _that_ horrible for our purposes).

squeegee

05-04-2006 03:40 AM

Ok, I've found what I consider the fundamental issue with luadec: It's not decompiling the binary code directly, but using Lua to load and walk the code. As a result, the internals are quite ugly, as a lot of Lua specific internal states, upvalues, etc. all have to be tracked.

The odd thing is that none of that is needed. The object format is very straight forward, and the mapping from the 35 opcodes to the source is almost direct. Even without the debugging info (function names, local variable names), recreating the original source should be doable.

Anyway, I've been studying chunkspy as well (a disassembler for Lua object code writen in Lua - only 2100 lines, with lots of comments), and I think I am going to work on creating something from scratch over the weekend. I'll post the progress.

This is so much more fun than working on work! :-)

Athanasios

05-04-2006 08:37 AM

Squeegee, mind that decompliling is one step, next step will be re-compiliing our modded lua files. As for opcodes, you should pay very attention so that we dont mess with micromemory (virtual i guess?), but you already know this ;).

If you can do both of these, then you should be given the "golden EaW metal" and ill be the first one to contact admins for this :D!

As soon as we can get our hands dirty with lua scripting.........our way is open ;).

Keep up the good work!

Mike.nl

05-05-2006 01:11 PM

Allright, this is it!

I have finished two tools that:

Converts between the official Lua format and Petroglyph's Lua format.

Disassembles a Lua 5.0 object file.

I used these tools (among others) to decompile most Lua files (91%, lacking some story scripts for now).

Yes, that's right. Lua sources are available.
Along with instructions on how to create custom Lua files for Empire at War.
Get it all here: http://alpha1.dyns.net/eaw/

Lacking a good decompiler, I've manually decompile everything luadec crashed on, and I still have to do the remaining story scripts, but I decided to release the Luas I've decompiled so far so you guys have something to work on :)

Of course, keep in mind the source is uncommented and uses a bit vague function arguments, which is inherent to any decompilation. Perhaps I will one day rename them to something meaningful, based on what they do, once I have time.

Anyway, as usual, if you have questions or remarks (both the good and the bad kind), post here.

squeegee

05-05-2006 01:39 PM

Quote:

Originally Posted by Mike.nl

Allright, this is it!

I have finished two tools that:

Converts between the official Lua format and Petroglyph's Lua format.

Disassembles a Lua 5.0 object file.

I used these tools (among others) to decompile most Lua files (91%, lacking some story scripts for now).

Kewl.

Quote:

Originally Posted by Mike.nl

Yes, that's right. Lua sources are available.
Along with instructions on how to create custom Lua files for Empire at War.
Get it all here: http://alpha1.dyns.net/eaw/

Lacking a good decompiler, I've manually decompile everything luadec crashed on, and I still have to do the remaining story scripts, but I decided to release the Luas I've decompiled so far so you guys have something to work on :)

Ouch. Thanks for taking the time. I also have decompiled all but the 13 files luadec ended up bailing on (I fixed some of the crahes to get that far).

Quote:

Originally Posted by Mike.nl

Of course, keep in mind the source is uncommented and uses a bit vague function arguments, which is inherent to any decompilation. Perhaps I will one day rename them to something meaningful, based on what they do, once I have time.

Anyway, as usual, if you have questions or remarks (both the good and the bad kind), post here.

I am working on a good decompiler. It can decode the binary files for both debug and non-debug already (like what -list can do with luac), so today I am starting the source code generation.

Unlike luadec, this loads the binary object directly, wihtout Lua's involvement. The code is only 500 lines of c++ so far. I should have the source generation working reliably within another couple of days. I've named it luad (I know, how original...).

Once this is done, along with your lup2lua tool, and luac, we should be able to decompile, modify, and recompile the sources!

If you want, we can combine all of the tools into one, so it can be done easier for the non-tech types out there.

Great job!

Athanasios

05-05-2006 02:49 PM

MUST BE STICKIED I-M-M-E-D-I-A-T-E-L-L-Y !!!

Also Mike, do not forget to post this over PFF forums too. The people will go mad!

And someone give those guys a golden medal :D. Mike, i suggest you work with squeege so that we get 100% bug-free and 100% commented lua files.

Check your pm also ;).

[edit] I posted this for you in PFF, just in case you're not registered ;).

Mike.nl

05-05-2006 03:32 PM

Quote:

Originally Posted by Athanasios

Mike, i suggest you work with squeege so that we get 100% bug-free and 100% commented lua files.

I'm pretty sure the code is bug free. I recompiled my decompiled code and the resulting object files was an exact binary match with the original.

And we wouldn't want 100% commented lua files. That would leave no room for the actual code ;)

Quote:

Originally Posted by Athanasios

I posted this for you in PFF, just in case you're not registered ;).

Yeah, I'm not, so thanks :)

Actually, the biggest challenge now is to create documentation. Establish an API. Find out how the Luas interface with the game and how far that interface can be stretched.

And to squeegee:
I'm looking forward to your decompiler. It would certainly be nice to bundle these tools.
Out of curiosity, how robust will your decompiler be? Will it depend heavily on known constructs by the compiler? Or is it able to generate equivalent source code from other, but functionally equivalent code as well?
For instance, the compiler compiles while loops with the test at the end, but what if the test were moved to the beginning? And what if the assembly was more optimized in terms of register (re)usage? The standard lua compiler is pretty brain-dead from what I can tell.

But I do agree, the lua object format is ridiculously simple, that's why luadec never impressed me that much, with all its bloat.

squeegee

05-05-2006 03:58 PM

Quote:

Originally Posted by Mike.nl

And we wouldn't want 100% commented lua files. That would leave no room for the actual code ;)

Comments aren't stored in the object files, so it wouldn't matter. I'm sure you know this, just making sure everyone else does.

Quote:

Originally Posted by Mike.nl

Out of curiosity, how robust will your decompiler be? Will it depend heavily on known constructs by the compiler? Or is it able to generate equivalent source code from other, but functionally equivalent code as well?
For instance, the compiler compiles while loops with the test at the end, but what if the test were moved to the beginning? And what if the assembly was more optimized in terms of register (re)usage? The standard lua compiler is pretty brain-dead from what I can tell.

It will recreate the original source as close as possible; It will generate the exact same object, or it's not done (I mentioned before that IMO a decompiler's most important trait is accuracy).

The lua compiler (tokenizer...) doesn't need to optimize much, as the vm assembly is so simple, and the vm itself can optimize stuff out. If you read some of the vm asm specific stuff out there, they point these things out, as well as other quirks. Most of those doc's are dated in the last 6 to 9 months, so they didn't exist when luadec was written.

My "plan" is to be able to decompile any lua source. Once that works perfectly, then I'll derive a version specific to EAW/Lup, and we can combine the stuff into a single executable. Maybe even provide a simple GUI so non-techies can use it (of course, maybe they shouldn't be modifying the code anyway... ;-).

Quote:

Originally Posted by Mike.nl

But I do agree, the lua object format is ridiculously simple, that's why luadec never impressed me that much, with all its bloat.

Luadec's issue is the writer evidently hasn't written anything of the sort before, and attempted to use Lua's code to do some/most of the work.

What I have is entirely stack based, and thus has no limits on depth of functions, loops, etc. (well, other than the memory of the machine it's running on). By definition, it can only re-generate what the object code contains. That is, if the compile moves things around (which I haven't seen a case of yet, btw), then it will regenerate it that way, as there is no way to know different. Functionally, though, it'd be the same as the original, and the object code will be the same.

Looking forward to getting this done; It's been a lot of fun so far!

BTW I can send the code to anyone who wants it at any time. It'll be public domain or BSD/MIT styled licensed when I get done with it.

Also, Mike, if you want the version of luadec that has fixed most of the crashes, let me know. I have a site I can put it on, I just have to put some work into the web site first (the domain is for email, but the http hosting is included but currently unused).

Slocket

05-06-2006 01:23 AM

A big thank you to you guys! :) You know your stuff very well (this is way above my head). It will be good to have even more versions to work with, in case something came up for redunancy.

As for me, the 'technically challenged', I can play with the AI game mechanics and see what happens! lol

fortune faded13

05-07-2006 02:37 AM

this sounds very good. i have almost no knowledge of LUA or anything of that sort, but im gonna look into it later this weekend when i have time.
way to go mike!

GhOsT-Jedi

05-16-2006 12:36 AM

If I read through all you posted, I could learn somthing, but my god you guys know your stuff, Keep up the good work ;)

Slocket

05-16-2006 03:05 PM

Using Mike's new cool LUA Tools I was able to make a first Enhanced_AI_RAID LUA Module!

I wanted more of a challenging RAID ability for the Rebels. Now for the first iteration the AI will use up to 6 units plus the normal heros and commandos, and have a faster retreat time. Hit and Run tactics really is almost human like and to be feared! :)

Later you can change what the AI considers for target priority and task force mix. It really does allow you change the AI behavior to be smarter!

I am (have to) re-install EAW and making sure it is fully functional under patch 1.4. Once released, modders should try out the LUA scripting to make changes to the AI behaviour for the better after seeing what a HUGE difference it can make.

Slocket

05-17-2006 05:25 PM

Just to let you know, the patch 1.4 gives you the decompiled source LUA to use!

I am testing it to see if the game likes it if it is modified.

squeegee

05-17-2006 05:31 PM

Wow. I will re-decode the meg's and check it out. Thanks for the heads up!

Slocket

05-18-2006 01:13 PM

The modified Lua will work in game. You can also add extra lines to it like C language. I was 'copy paste' command blocks to add extra stuff IE a Huge Rebel Invasion Fleet when it decides to "Crush.lua"...and it is awesome.

The RIAD_AI_BOOST works fine and is ready to go. I tesed it. Now fear a Rebel raid for sure! lol

squeegee

05-18-2006 02:47 PM

Quote:

Originally Posted by Slocket

Just to let you know, the patch 1.4 gives you the decompiled source LUA to use!

I am testing it to see if the game likes it if it is modified.

Really? Where? After I moved out the decompiled sources I created from luadec, and re-extracted the new meg files, the LUA files are still in the same luap format as before. Where did you find the source for them from 1.4?

Mike.nl

05-18-2006 05:35 PM

I'm not sure if Slocket was correct by saying patch 1.4. It could be from the map editor as well.

Anyway, look in GameData\Mods\Source\Data\Scripts.

squeegee

05-18-2006 06:03 PM

Quote:

Originally Posted by Mike.nl

I'm not sure if Slocket was correct by saying patch 1.4. It could be from the map editor as well.

Anyway, look in GameData\Mods\Source\Data\Scripts.

That directory structure doesn't exist with just the 1.4 update (specifically GameData/Mods), so it must be the map editor, which I haven't got around to yet. I'll go grab that next.

Thanks for the clarification Mike!

Admiral Sith

05-19-2006 04:14 PM

Map Editor = Mod Tools. Including a exporter for ALO format :P

squeegee

05-19-2006 04:18 PM

Quote:

Originally Posted by Admiral Sith

Map Editor = Mod Tools. Including a exporter for ALO format :P

Yeah, I knew that, I had let eaw dl and install the 1.4 update itself, so I didn't get the editor. I have it now, and all the scripts are in source form.

So much for needing a decompiler. I am still working on it, but work has been getting in the way (if only I worked for google...:-).

With mike's lup converter and the source, we should have everything needed to make changes to the ai now.

Michael Plant

05-20-2006 06:51 AM

We don't need a converter, I tried replacing the lua files in gamedata\data\scripts folder with the ones that came with the map editor, and the game still worked.

Slocket

11-09-2006 02:14 AM

I am trying to decompile the new Forces of Corruption LUA with Mike's tools. Been awhile and his site for instructions is gone.

I think it is the same as EAW. Now where did I place his LuaDec 0.6 program. Any else have a copy?

EDIT: Found the files, just have to look at the URL home. The download section was moved.