Python Upgrade Coming Soon

I want to give you a heads up that in an upcoming release, we will be upgrading the version of Python used by the game. We currently use Python 3.3.5 and will be moving to Python 3.7.0. Once this change is released, all compiled script mods will completely stop working. Note that in this new version of Python, the .pyo extension is replaced by .pyc, see https://www.python.org/dev/peps/pep-0488/

We know this will have impact on your script mod authoring pipelines so wanted to provide notice in advance so you can be ready for the change when it arrives. To assist, I’ve attached the Python scripts from the currently released version of the game (v1.46.18) recompiled under Python 3.7.0.

If all goes as planned, this change should arrive in November.

Note that there will not be any fundamental changes to tuning data, other than the typical types of changes we make to tuning for patches.

Comments

Short term this is going to be such a headache, but I think in the long run this is really good news. Kind of ironic though that I was just telling someone recently that EA would likely never update the Python version due to the number of mods it would break!

As Turbo says, THANKS FOR THE WARNING and for including those recompiled scripts so we can try and figure out how we're going to reverse engineer those in advance.

Any chance we get a more concrete idea of the time frame as you know more about when this bomb will drop?

It's very unfortunate to see that this could be the death of many great new script mods. The lack of a reliable decompiler for Python 3.7 means that a lot of essential sources are missing and without them there is no way to figure out how many elements of the game code work.

Is it possible that there is some documentation that we can get that explains all of the files and functions? Is there anything that can help us now @SimGuruModSquad?

@LittleMsSam Compiling existing scripts won't be an issue, there will be some changes required to some batch files but that's minor.

@SimGuruModSquad The problem will be getting the EA scripts we refer to for new mods. So far the best decompiler we've been able to work with for Python 3.7 is uncompyle6 and that just chokes and dies on any seriously important script files. Many critical files that would be a required reference for a script modder are simply no longer accessible (unless someone finds a better uncompiler, but they are mostly unmaintained since Python 3.5 bytecode changes).

game_object.py, object_tests.py, sim_info.py and many other critical files we'd need to reference are simply cannot be decompiled at this point. Unless something changes, this may well spend the end of script modding for me personally and likely others - I for one have no desire to continue to support mods when I can't look at script changes that break those mods. And certainly any new scripting would be unthinkable for me.

Unless somehow EA can reverse their viewpoint on releasing the core scripts to modders for reference, or we can find a reliable method to reverse engineer those files - something which EA has been aware of us doing and quietly approved of since the game came out.

Not a modder, so maybe this doesn’t “count” but I’ve been with SIMS since day one. If those of us who rely on Mods to make our games fun lose out because they can’t fix the scripts easily, you’ll see a drastic drop in gameplay. I love the work ya’ll put into this game, and the work the modders put in too. Please make it easy for them to fix this issue when you release the update!

Do not panic yet, folks! EA notified us of this in advance because there were bound to be unforseen issues. We have to give them time to digest what those issues are and determine what their best response to the issue is. They're being proactive for a reason!

I've been cutting back on script mod usage -- pretty much rely on MCCC and UI Cheats though for both gameplay and townie creation/fine tuning. It's been great going in and easily adding extra traits like the ones from Parenthood to fill out personalities on the premades. I hope that the change in Python version doesn't kill those -- as @PIsForPerry said, there'll be a lot of players who'll give up on the game if they lose script mods especially MCCC. Posted the link to this on the MCCC Discord server to give @Deaderpool a heads up.

Yep. Aware of it and, like has been said here previously, I'm just hoping that this means that going forward EA will give us the decompiled scripts like they did on this post when a new release comes out. We've always had to decompile the core-game scripts to do script modding. If going to the new version breaks our decompiler (which it does), then it would be nice to know that they're going to take care of us going forward by providing that.

Otherwise, compiling to a new version and all, we can make work. Those of us still around, anyway.

Yep. Aware of it and, like has been said here previously, I'm just hoping that this means that going forward EA will give us the decompiled scripts like they did on this post when a new release comes out. We've always had to decompile the core-game scripts to do script modding. If going to the new version breaks our decompiler (which it does), then it would be nice to know that they're going to take care of us going forward by providing that.

Otherwise, compiling to a new version and all, we can make work. Those of us still around, anyway.

Small correction, the file sgms posted here is still compiled code. It remains to be seen if they'll provide uncompiled code but here's hoping. 🤞

Yep. Aware of it and, like has been said here previously, I'm just hoping that this means that going forward EA will give us the decompiled scripts like they did on this post when a new release comes out. We've always had to decompile the core-game scripts to do script modding. If going to the new version breaks our decompiler (which it does), then it would be nice to know that they're going to take care of us going forward by providing that.

Otherwise, compiling to a new version and all, we can make work. Those of us still around, anyway.

Small correction, the file sgms posted here is still compiled code. It remains to be seen if they'll provide uncompiled code but here's hoping. 🤞

Nice to know. I haven't unzipped it or done anything with it yet. Planned to do that tomorrow. Thanks!

@SimGuruModSquad
If you still aren't willing to give us the uncompiled script sources, could you reconsider giving us the documentation?
I am assuming that you generate documentation for internal use, and I can't see how it would be worse than giving us the TDESC's.
Although I haven't delved much into script modding myself, my guess is that having access to the documentation is far more useful to us than having access to decompiled code.

@Scumbumbo
Remind me if I'm wrong, but weren't we unable to decompile a lot of files in the beginning, but then someone went in and fixed a decompiler?
We probably could look into why uncompyle6 chokes on the files and maybe fix the problem there.
As our community uses these tools so much it would be nice if we gave back some, if we can that is.

To everyone else:
I'd suggest that anyone that Isn't going to be working on decompiling the scripts keep from posting here, not because your opinions aren't valid, but because if you derail the conversation it's less likely that we will come to a good solution. Consider making a new thread instead.

@SimGuruModSquad
If you still aren't willing to give us the uncompiled script sources, could you reconsider giving us the documentation?
I am assuming that you generate documentation for internal use, and I can't see how it would be worse than giving us the TDESC's.
Although I haven't delved much into script modding myself, my guess is that having access to the documentation is far more useful to us than having access to decompiled code.

If not decompiled code, the best they can do is give us an outline of the code (by wiping the actual code) with the developer notes. But something tells me that there might not be any useful (for us) notes in there. I might be wrong about it but the programming staff hasn't changed much for the last 4 years which means it's a very minimal documentation. Looks like a lot of sections of the sources is dealt with by individual programmers since they include their names there, so they don't need that much documentation in the first place.
And we can generate an outline like that ourselves, but it won't be anywhere near as helpful as the actual source.

Remind me if I'm wrong, but weren't we unable to decompile a lot of files in the beginning, but then someone went in and fixed a decompiler?
We probably could look into why uncompyle6 chokes on the files and maybe fix the problem there.
As our community uses these tools so much it would be nice if we gave back some, if we can that is.

The decompiler for Python 3.3 was actually designed for 3.2 but then edited for someone somewhere to work with 3.3. I don't think it was the Sims community that got it to work, it was the case of using the decomplier for the right version since they are named the same. And if I'm wrong, just by looking at uncompyle6, we are dealing with something a lot more complex and editing will be challenging.

Oh, man. And I was just getting into scripting. Swatting up and slowly botching a few attempts. Well scrap that now for a pointless endeavour. Locked in tuning purgatory for a while longer, I guess.

Well, as Scumbumbo pointed out, early days yet, EA putting out the feelers, and thus may well tune their final approach to fully consider the script modders who have put so much hard work in over these last four years.

I imagine healthy game sales figures do depend on it. Don't bite the hands that feed you, nor the ones holding up your chair. So...

@fogity That "fix" you are thinking of back then for unpyc3 was iirc a pretty simple one, just ignored an error by trapping the exception and moving on as best it could. Produced some gibberish, but hey that's life.

Andrew and I have both tried some similar hacks in the same vein without any really notable results.

I'm going to run everything through decompilation tomorrow and summarize exactly what results we end up with and provide those to SGMS. Andrew and I were just chatting with them on Discord, I won't post the gist of the convo here as I don't want to put words in EA's mouth obviously. All I can say here is that releasing sources is unlikely (as we expected), but they seem committed to helping us work out issues with decompilation.

At a guess, I'd say roughly 75% of files seem to uncompile properly using uncompyle6, but those that fail are quite important ones as a general rule. Some that uncompile have some invalid syntax in them, particularly those with complex nested expressions. My tests I run will involve recompiling the uncompiled sources to check their validity.

Feel free to PM me on DP's Discord channel if you want some more unofficial info, I imagine SGMS will be posting here as well once anything is official.

@SimGuruModSquad - thanks a lot for chatting with us, it is good to know definitively that you will assist in-so-far as you are able to.

@Scumbumbo
Although I really shouldn't be working on the sims, I have fallen down the rabbit hole again. I might be interested in looking into trying to fix the issues in uncompyle6. Although python is not my language of choice I do have knowledge about compilers and such.

@simgurumodsquad I have those results you wanted for the uncompyle6 tests. I just ran it on the files in simulation and core as we don't really care about base or generated.

1,261 (73%) files uncompiled and tested 100% accurate.
296 (17%) files uncompoiled but could not be recompiled due to syntax errors.
169 (10%) files threw exceptions during the uncompile run, of those 10 were due to pressing Ctrl C as those files would get stuck in an infinite loop. The remaining 159, at least all the ones I saw, threw the assert failure due to not being able to properly parse the kwargs ending up with a negative positional arg count.

I can send you a copy of the results, which would include the failed uncompiled files and the log that was generated with all exceptions and messages about failed verifications, just let me know how you want me to get that to you.

Meanwhile, Fogity and Turbo have been trying some stuff with pycdc and I'll likely try again to get a working copy of that compiled so I can look into that.

Right now I am Modless and CC-less. But in the past I had a lot of CC and even dabbled in mods. So I have faith in the modders and cc makers- they are an intelligent & creative group, and I think they will adapt..On another subject This change I believe has to do with the whole new terrain changing tool.I don't know much technical stuff But I hope this leads to us being able to make our own Worlds/neighborhoods- like in Sims 3 with the Create a world tool...I have faith in our Modders they will adapt !!