If this is your first visit, be sure to
check out the FAQ by clicking the
link above. You may have to register
before you can post: click the register link above to proceed. To start viewing messages,
select the forum that you want to visit from the selection below.

The "New" Frontier of M2TW Modding: Memory Editing

Hi all! Recently, I started analyzing the M2TW: Kingdoms' executable in order to have a better understanding of the way the engine works for what concerns the way it parses and uses data files (how unknown options and values affect the game and how volatile data is stored into the memory) and the causes of its more common bottlenecks, bugs and misbehaviors. A mastodontic task that, little by little, is producing excellent results, especially for the knowledge of the modding community.

Then, suddenly, I realized that if I wanted to go further into this (fixing the game engine or modifying its behaviors) I would have to find a way to do it without breaking the game's license... and I found it: memory editing. Let me explain you this: in fact, as we all should know, it's strictly forbidden by CA to alter the game executable, as stated in its eula:

You agree to only use the Game Software, or any part of it, in a manner that is consistent with this License and you SHALL NOT:

{...}

(f) reverse engineer, derive source code, modify, decompile, disassemble, or create derivative works of the Game Software, in whole or in part (except as the applicable law expressly permits, in which case all and any modifications, adaptations, improvements, etc. shall belong to, vest in and be the exclusive property of Sega and/or its licensors on creation, in any event);

{...}

(i) create data or executable programs that mimic data or functionality in the Game Software.

The same rule applies for almost every software house that produce closed source applications but it's not always easy to understand what kind of limitations it imposes. In my point of view, for example, the declarations "reverse engineer" and "create derivative works of the Game Software" should immediately ban all what concerns modding because discovering the structure of binary files (meshes and textures, for example) absolutely fall into the reverse engineering category. The same goes for the statement "create data or executable programs that mimic data or functionality in the Game Software" because edited files can be considered data that mimic existing software files and the Launcher replacements that can be found in this forum can totally be considered executable programs that mimic functionality of the original application. Right?
While we could waste our time discussing for the next few months about the legitimacy of binary data modifications and launcher replacements without finding a concrete and unanimous solution (it could just be considered ok, we can be sure about the fact that, by all means, releasing a modified executable could lead to very disagreable consequences (signally, a legal action from CA).

And here is where memory editing comes into play: the first thing I learned when I started working with debuggers, decompilers and disassemblers is that while it's totally forbidden to edit executables, it's 100% legit for a user to edit its own memory without any consequences. Memory editing is a grey zone in which, nowadays, there are still no real rules and this is due to the fact that all what is being put inside the virtual memory of a computer switches its ownership from the productor of a packed entity (the executable compiled by CA) to a consumer of volatile data (yourself). It's your memory, you bought it, you can do whatever you want with it.
When executables are launched, they are loaded and mapped into the computer's memory and, there, their instructions can be edited as will without breaking any law. This is the correnct and only path we must follow to circumvent the eula's restrictions and bypass any limitation. How? Using a custom launcher/patcher that creates a fully deployed game process in a suspended state, modifies some small parts of it in order to change its behavior and finally resumes its execution. This one, to be exact:

At present, the application has been created for my personal use only and has never been distributed, but it received a go ahead from the forum's administration. Here is their verdict, that has been sent to me via PM:

Originally Posted by Gigantus

Hi Zarathos,

the issue has been discussed in depth and the decision\guideline is as follows:

You have a go ahead from GrnEyedDvl (TWC owner) and Jack Lusted (CA member). No altered exes, and no game exes provided, but altering values in the memory is ok.

If you have further questions, do not hesitate to contact me.

Greets
Gig

By all means, patching the executable can lead to infinite opportunities to improve the engine (adding new features, fixing bugs, improving existing features, removing limitations and so on) of a game that left its development stage long, long time ago and it's no more being improved and maintained by its creators, but that still has a great and very active modding community and is still being sold (blink blink). Here is what I achieved so far:

changed the EDU units limit from 500 to 1000 (just some memory allocation adjustments and pointer shiftings remain to be done);

changed the soldiers limit from 4/100 to 1/150 (if I'll find the multiplier being used when you set the CFG option unit_size to a value different than normal we will finally be able to have real one-man units);

made the executable aware of large memory addresses on 64 Bit systems with at least 6 Gb of RAM;

optimized the game engine for single core execution.

All I need is collaboration. Analyzing an executable is a very difficult and time consuming task to accomplish that requires a high knowledge of software architecture and assembly code. Anyone willing to help me is more than welcome provided he knows how to use the necessary tools (IDA Pro, OllyDBG or another similar debugger and an hexadecimal editor).

Re: The "New" Frontier of M2TW Modding: Memory Editing

Remember help can come from many ways, if you know someone who understand about coding this type of thing, call it, show him that thread, go my fellows, go and spread the good news...

Here goes my 50 cents:

THINGS TO CHANGE ON THE .EXE

-Height limit for terrain (battlemap is flattered beyound 80+ height, thats why battles above mountains will load totlly flat battlemaps)
- Age limit (elves should live forever!)
-Faction limit
-Culture limit
-Testudo formation.
-Single man per unit.
-Trolls/beasts to hit gates (like elephants did in rome 1)
-Enable arrow slots firing from siege towers again (it is a working feature in rome1)
-Ajust screen resolutions for widescreen monitors.
-Remove settlement core type icon from ui of unique settlements.
-Remove stratmap models from settlements showing even under fog on stramap (it actually shows vanilla med2 models even if your settlement has a custom model).
-Allowing certain settlements to not require siege engine to be attacked even if its level is one of that needs it (rivendell for example is a large city but has no gates).
-Creating a new unit type besides cavalary, infantry etc, that would not respect pathfinding both on battlemap and campaign map (for flying units!)
-Creating a new infantry unit type that would not be able to climb ladders or walls (trolls)
-Acivating the settlement view feature from Rome.

Re: The "New" Frontier of M2TW Modding: Memory Editing

I'd like to help. I study computer science specialized in low level programming and embedded systems. I've written programs in assembly (specifically for the ARM Cortex-M0, but I can easily learn other instruction sets) and am pretty handy with C/C++. Things like memory allocation, how pointers work exactly, hexadecimal representations, endianness, principles of object oriented software architecture, etc are common knowledge for me. I don't have much personal experience with reverse engineering game engines, but I've followed the development of the Elder Scrolls script extenders pretty closely in the past four years or so, so I've got an idea of what's involved. I know I can do this kind of thing, especially with the occasional help from someone more experienced. Let me know if you want to have me.

Btw, will the development be completely open source? Will it use source control like Mercurial/Git/SVN?

Re: The "New" Frontier of M2TW Modding: Memory Editing

Well I don't know anything about these stuff, I only know of this Large adress aware tool that supposedly makes games use up to 4gb or RAM. In the screen you posted those are just things anyone can edit in the cfg, but if you say it's possible to increase hardcoded limitations like the 500 units or the number of factions maybe, I can't wait to see such a thing with my own eyes! I wish you good luck then.

Re: The "New" Frontier of M2TW Modding: Memory Editing

Many thanks for your replies guys, this is a very nice motivation boost for me

Originally Posted by Maegfaer

I'd like to help.

{...}

Btw, will the development be completely open source? Will it use source control like Mercurial/Git/SVN?

You are more than welcome. The last time I did this kind of tasks I was between 16 and 19 years old... many, many years have passed and, meanwhile, I forgot many things and even more things have changed in programming technology.
The source code of the KingdomsPatcher will NEVER be publicly released for the following reasons:
1) the purpose of this project is to create a sort of M2TW: Kingdoms 1.6 unofficial patch, so the patcher must be able to work on every PC and on every system in the same way if we want it to be available for everyone still playing the game (exactly as official patches work)... if everyone is going to make its own patcher to modify the game executable the way he prefers, soon or later, the community will be overflowed by patchers that modify the hardcoded limit to different values for different reasons and this fact will pose major compatibility issues and a lot of confusion;
2) speaking with people about this, I noticed that everyone is pressing to solve his own issues: mappers want mapping problems to be fixed, modelers want better graphics and more complex models, scripters want more commands, and so on... but the real problem is that without a focused effort, this project will go nowhere;
3) every single modification made to the executable has to be done cum grano salis: it has to be tested and tested and tested again but, most important, has to take into account the limits of a single CPU engine with very old graphic libraries and old school programming approaches... I'm pretty sure that releasing the source code of the patcher will lead to people pushing the hardcoded limits to a level that cannot be tolerated by the game engine, compromising the stability of this work.
What I can grant instead is that everybody, in the development team, will have full access to my IDA Pro database containing:
- the disassembled executable with a lot of members (arguments, enumerators, methods, static instances, structures, variables, virtual function tables, etc...) renamed in a coherent and human-readable way (just an example: sub_008EEF56 has been renamed to ParseEDUFile);
- many inline comments that explain what the executable is doing and how;
- the binary patches being applied to the memory mapped version of the executable (it's just a simple txt-like file with every single line in the form of <OFFSET>:<OLDBYTE>:<NEWBYTE> which is being used by my KingdomsPatcher also to modify the process into memory by just converting the offsets to a value relative to the process' image base address);
4) if we let people know how we modify the executable, soon or later modified executables will pop out leading to bad consequences, maybe for us too.

Originally Posted by Vipman

Well I don't know anything about these stuff, I only know of this Large adress aware tool that supposedly makes games use up to 4gb or RAM. In the screen you posted those are just things anyone can edit in the cfg, but if you say it's possible to increase hardcoded limitations like the 500 units or the number of factions maybe, I can't wait to see such a thing with my own eyes! I wish you good luck then.

What you can see is an application I started to develop as a normal launcher replacement for my personal use: of course, all those options can be manually set into the CFG file, but changing the game options before launching it in this way is much more simpler and quick.
Anyway, this is not the point. That launcher is now being used as a base for my patcher: it changes the CFG file options, it starts the game with the selected mod and then it also patches its memory before resuming the main thread. Of course... executable modifications have no related buttons because they are pretty standard and the memory editing process is totally invisible

Originally Posted by Rorarii

fantastic, would be nice to have upgraded graphics but that's probably out of the question.

Memory editing sounds very short term until it's saved, then I suspect it breaches the law?

Memory editing will change the executable instructions once it has been loaded and mapped into the RAM. It's a very short term modification, yes, because unlike modifying the executable itself (which is illegal), you need to do this every time you launch the game and, as soon as you close it, every change will be lost.
And yes... upgraded graphics are almost impossible to be achieved. This would require the use of new graphics library being injected at runtime into the assembly and a major rewriting of every rendering process. I don't even want to think about how much work this would require, it's a nightmare.

Re: The "New" Frontier of M2TW Modding: Memory Editing

Memory editing is not against EULA or any law, so that's why you were allowed this. Many moons ago there was some Spanish modder who did the same thing and even modified the battlefield expansion. He was Agostinos or Agartino something and there's a thread for that MMM modding sth in the medieval 2 mod workshop.

If you need help with this project then i suggest you contact wilddog and tellos. They might be able to help you out nad they have the proper skills.

Re: The "New" Frontier of M2TW Modding: Memory Editing

Does multicore support can be enabled? What are the next of your plans?

Multicore support cannot be enabled. It's almost impossible, because it's linked to the architecture of the whole executable and the way it works. Next plans?
- Completing the EDU limits increase.
- Increasing hidden resources limit from 64 to 128 (I already spotted the necessary ASM opcodes to modify).
- Creating better exception handling messages to help modders discovering the errors quickly.
- Fixing major CTD issues.
- Improving height maps management with heights being truncated to 80.
- Increasing the regions limit from 200 to 300 (but probably not the map maximum size)
- Much more...

Originally Posted by Brune

-Will it be possible to have a free camera and an auto-resplenish system like in the warscape total war?

It can be done. In fact, almost everything except improving the game rendering and enabling multicore support can be done... it all depends on how many people will join me for this project, how fast we will discover things and how the game engine has been programmed by CA.

Originally Posted by Ishan

Memory editing is not against EULA or any law, so that's why you were allowed this. Many moons ago there was some Spanish modder who did the same thing and even modified the battlefield expansion. He was Agostinos or Agartino something and there's a thread for that MMM modding sth in the medieval 2 mod workshop.

If you need help with this project then i suggest you contact wilddog and tellos. They might be able to help you out nad they have the proper skills.

Congrats and good luck with your project..

I know that thread (by Argantonio), I almost know it by heart. It was very helpful for me, unfortunately it was created for M2TW and not for Kingdoms, so some things have been changed meanwhile by CA developers.
I'll try to contact those guys, I hope we can set up a small working team of at least 5 persons, because if we can split up the tasks and work together we should go very fast.

Re: The "New" Frontier of M2TW Modding: Memory Editing

Multicore support cannot be enabled. It's almost impossible, because it's linked to the architecture of the whole executable and the way it works. Next plans?
- Completing the EDU limits increase.
- Increasing hidden resources limit from 64 to 128 (I already spotted the necessary ASM opcodes to modify).
- Creating better exception handling messages to help modders discovering the errors quickly.
- Fixing major CTD issues.
- Improving height maps management with heights being truncated to 80.
- Increasing the regions limit from 200 to 300 (but probably not the map maximum size)
- Much more...

Too bad about the multicore support, but these things you plan sound great, especially fixing ctd issues and improving error reports in the log to find the problems easier sound fantastic! I really hope you will manage to achieve such things and release it

How would the compatibility with mods be? If with this patcher I select the mod I want to start will it work with any kingdoms mod or will it require some changing some mod files?