Topics - Josh @ Dreamland

We're looking to merge #1875 (New Events) into master. We've tested it pretty thoroughly, but there's still a chance it will break something; this recodes a huge chunk of the event system. We will be merging it into the EGM feature branch today, and expect it to hit master on or before the completion of EGM.

If you have time, and would not mind testing this change out, please issue git checkout NewEvents and give the system a go.

Please report any issues either in this thread or on the pull request page on GitHub. If no one says anything before EGM is complete, these changes will be merged and we'll deal with the fallout, then.

This is a quick heads-up that we're in the process of switching our YAML parsing over to yaml-cpp instead of doing the reader in-house. This has been coming for a while, so you probably already have yaml-cpp installed. If not, please run install the yaml-cpp package using your favorite package manager.

I have some happy news to share regarding some changes to the ENIGMA codebase. Cheeseboy, Robert, and I have been making some relatively big changes to the infrastructure of this project, lately. You (I hope) haven't noticed so far, because these changes are mostly internal cleanup, code health, and productivity changes. Since we fixed the Travis-CI build a few months ago, Robert has added an AppVeyor build. With the new assurance that a change will not break other platforms, we have been cleaning up the codebase and moving things around. We have had minor regressions, which is why I'm in the process of writing a regression test suite on top of our build tests. But for the first time in this project's history, all contributors are submitting code to branches instead of directly to Master. Our build has been green since we fixed it a few months ago. This is huge news for us, but doesn't directly affect users.

The downside is, of course, the minor regressions. And we're about to have much bigger changes.

What you need to know:Some dependencies will be added to the project. We are going to stop baking GLEW into our source, for example. LateralGM will also be updated. We will provide instructions on how to obtain the latest dependencies when that time comes. I expect us to just ship a new portable zip that everyone will need to update to.

A more comprehensive list of changes:

In the coming weeks, we plan to completely change the way LateralGM (and other IDEs) communicate your game data to ENIGMA. We will use a flatbuf-based system instead of EnigmaStruct, which will be removed.

As I mentioned, Cheeseboy has removed the GLEW source from our source tree. You will soon need to obtain it yourself (probably by downloading a new portable zip). Advanced users can install it through MSys on their own.

ENIGMA now writes a .desktop file on Linux alongside built games. This serves as a launcher for the game, and enables users to more easily set their own game icon.

Expect the forum to enter maintenance mode. You will be unable to post for some window of time during this migration.

Do not be surprised when this happens. I will only activate maintenance mode during the database migration. This should last something like half an hour—The export will take fifteen minutes, but migrating the ENIGMA DNS to point to our new IP will take longer. I'm going to see if I can change the assigned IP over instead of updating DNS records. I will keep this thread up to date.

I recommend using our shiny new SSL connection for signing into the forums from here on out. I will look for a way to enable this by default.

We don't want the default forum browsing to be secured, though, as you will just receive warnings that the content users post (images, avatars, etc) will be from third-party sites.

However, I made some arrangements, and these domains and the certificate are now paid through 2020. The website, however, is not. Right now I have launched an IRC bot, which the host said was okay despite the terms of our VPS lease. I will continue to renew this VPS as long as they allow both of these to run. I will also be, at some point in the next century, setting up a build bot here. If at any point this becomes "too much load" for our VPS, we will be moving to a dedicated server.

Google Summer of Code signups are around the corner. Are we, as a group, interested in participating? I am considering sponsoring either ENIGMA or NaturalGM, or even both, depending on which project is able to generate an workable list of tasks for contributors.

The problem with ENIGMA's participation is that its contributor base is already mostly college-level help, so the majority of tasks that would be easy to delegate to new help are already completed, while the remaining work requires more skill and background info on what ENIGMA is and what it is supposed to be.

That said, looking over the wiki page from last year, some of the workpoints seem nearly feasible. The problem is that no one has started any of them. If we want help, we have to be serious about having room for participants to actually contribute code. We can't be allocated a new contributor and expect them to figure out what ENIGMA is and just start tacking entire new systems on. Work points such as "Implementation of an Asynchronous Networking System" are entirely too much work for someone who is unfamiliar with the project. Does the Network_Systems folder even work like the other systems folders? How will a new contributor even know they are on the right track?

If we are to participate, we need tasks that begin with ten lines of code and end with implementing an entire system. There needs to be room for a contributor to start small and grow in the project. I can't endorse ENIGMA with such course-grain tasks. This is the reason I stayed out of the GSoC planning last year, and if we can't generate a better task list this year, we'll hit the same fate.

We can improve the "Implementation of an Asynchronous Networking System" task by creating a reference list of what we actually expect this system to look like, and offering an example game which, given these functions, should be able to be played over a network. We can't expect a new contributor to write the game that tests his or her new functions. That's way too much work for someone who has never heard of the project.

Related: Do we even have a simple dev wishlist, anymore? I occasionally hear about X huge thing that is completely broken and everyone has been working around, but I have essentially no idea what people want to see out of this project right now, aside from a new compiler. It seems to me that we have no little tasks; the project is waiting on someone to "just go through and fix everything." Does anyone care to prove me wrong?

To be brief, if we want to participate in GSoC, we have to make sure we have tasks that are actually suited to someone who has never heard of the project. Otherwise, I guess we can sit this one out.

I'm going to spend the rest of today getting the plugin building on my current dev machine. Ism gave the green light last week for a backwards-incompatible change to the EGM format, and then volunteered to handle backporting in the form of converters. Maybe that's okay for now, but I want to avoid another incarnation of the problem to avoid wearing Ism out, at very least. It's sad; even GMX is writing plain-text for its rooms and objects; we have some catchup to do.

While it seems that the massive changes that were flooding in over the summer have wound down (to a degree), I am still skeptical of this, and so I am changing my compiler-related plans to accommodate all the contributions that rolled in over the summer, as well as any new ones.

There are a couple changes I started rolling out in the other compiler branch that are becoming harder and harder to merge in as changes continue to be made upstream.

I was in the process of gutting the extension system and the relevant compiler chunks to try to splice in a hook system (see my notes on the Wiki from years ago) when I realized that the warnings I corrected in the compiler were starting to trickle back in, and that (on a very related note) Robert made an absolute war zone of write_object_data.cpp trying to add support for inheritance. I stepped back to address that, and realized that the best move is for me to make all of my non-parser compiler changes in master and naively hope that they propagate painlessly, which has been the development policy for a while, now.

In other words, gentlemen, it's Josh's turn to break things carelessly. I know it isn't ideal, but I've been too reserved about these changes in the past. I have merged enigma-dev/#819 after its (minor) review, as it seems not to seriously break anything important. If I can get you guys to stay on top of my future pull requests, we can probably make it through this next week without anything serious failing. I will exercise the maximum amount of diligence possible given our lack of a regression test suite, which I will probably begin to set up while I'm at it (see my current implementation here).

You will see my first commit sometime tonight or tomorrow. It will do three things, listed here in order of increasing impact:

Fix the warnings in module_write_fonts.cpp and small warnings elsewhere

Separate write_object_data.cpp into multiple helper functions and recode as needed to make non-hideous

Move ALL C++ codegen and module export logic into a new class, compile_context. This has already been done in the compiler branch.

The purpose of the move is so we don't have contention on globals in the event that we need to run two compiles or two parses. This is less important than the smaller, more-intricate move which will be made in the commit after that one, which will move all logic involving interaction with the parse system into helper functions. This includes the following:

All logic which directly requests the parsing of code.

All logic which requests information about user-defined variables.

All logic which attempts to reflect or coerce type information or code metrics.

The connecting factor, of course, is that the parser is somehow involved in all of those operations. The connecting symptom is that ENIGMA sucks at all of those operations. The reason for this move, therefore, is to make it so that once this is all said and done, the new parser will be a drop-in replacement for the old. My original approach was to do the replacement in a branch where I would not have to touch the old logic, but that's just not going to cut it for the reasons I've already hinted at.

I am going to finish mincing write_object_data.cpp, now.

TL;DR: Keep an eye out for my future pull requests. Please check them for breaking changes as they roll out.

I realize that Robert has been trying to implement a CLI, but this has been on my plate for a while. In fact, when I heard of his intentions, I published a repository I'd started to GitHub for reading files and general file processing groundwork. This includes a UTF8 implementation of std::string (still in progress) and a class for iterating files. Right now, the file iteration class supports zip iteration and filesystem iteration, but it has only been implemented on Linux. The zip iteration should work directly on Windows, provided someone can install libzip in MinGW. Today I coded what should resemble the Windows implementation. I have left it commented since it is not tested.

I meant this procedure to be for Robert, but he seemed generally uninterested in the prospect. I would appreciate it if a Windows developer would do the following:

I'm going to attempt once more to contact a third party for help with this licensing fiasco. I'm going to give everyone an opportunity to weigh in on what I'm sending before I send it; this is our official request for help.

I have sent a very similar letter to this one in the past, but I fear it did not adequately convey our issue, based on the response I received. Here is the text that will be sent this time:

Quote

Greetings,

I am writing on behalf of all contributing members of the ENIGMA Development Environment (http://enigma-dev.org/ , https://github.com/enigma-dev/). We maintain a free-software game development platform designed to simplify creation of various games while offering sufficient features to create games of any size.

Heretofore, all of the code in our project, including that in the game engine, has been GPL-licensed. Now, while our developer base is still relatively small, we are discussing a means by which our users could have more license freedom in the games they create. As our tools not only link against the GPL game engine code we provide, but in fact integrate segments of GPL code into their game code directly, all of our users are presently bound by the terms of the GPL, and so cannot release a closed-source version of their games.

What our team is looking for is a means of allowing our users to close-source their games in a way that no contributors biding by our license can take legal action against them, but to ensure that in all other cases, linking against our code causes the resulting module to use our license.

What we do not want is to become the next WINE vs CrossOver Office; that is, we do not want someone to exploit our license to maintain a closed-source version of our engine with exceptional improvements. If someone improves our engine, we want to be able to pull in those improvements, or at very least see how they did it and mimic.

The mantra of this operation is "Prohibit ENIGMA forks which prohibit ENIGMA forks." This "prohibiting" is supposed to be done by enforcing the full terms of the GNU General Public License. In short, we want the terms of the GPL to apply, but to allow users to choose their own license for their own games created in the engine.

The immediately obvious answer to our problem is to select a license such as the Mozilla Public License, which seems to allow just that. The issue is, this license would allow a third party to extend ENIGMA using their own proprietary code, which could then draw users away from ENIGMA, and get them hooked on these closed-source additions. We therefore want extensions to ENIGMA itself to be covered by this license/exception or by plain GPL, to ensure it remains free for everyone.

The crux of the problem is that the only time we do not want the GPL to apply is when a user actually releases a game made in this engine.

That said, which license/exceptions could we use? Would there be merit in a custom exception?

Thanks much,Josh Ventura

If this request fails, I will embrace the MPL or a similar license for the sake of forward progress.

tl;dr: we run CentOS 5; our software is so out-of-date, we are not affected by Heartbleed. Of course, you probably don't connect to us via SSL, anyway.

Heartbleed is an OpenSSL exploit that enables hackers to listen in on what should be secured connections. It's a terrible vulnerability that can lead to the leak of all sorts of sensitive information; in our case, passwords. Since most users connect over HTTP and are probably therefore not using a very secure password, this isn't an issue for most of our users. For those who do connect to us over HTTPS, you're safe, anyway, because the vulnerability is with a newer OpenSSL than we'll ever have. This has a number of downsides, but the upside is, we didn't even have to patch for this exploit. So your HTTPS passwords have been safe, and our SSH connections have been safe.

And to clarify, yes, I am still alive; I haven't lived for ten consecutive days in the same state for almost the last month, now, but I am finishing getting moved into my current apartment where I am now holding a job at Google. That said, afternoons and weekends are mine unless I break something.

ENIGMA (Specifically EDL) is full of stateful functions: functions which operate based on a global state. Obvious examples include file_find_first/file_find_next. It also contains functions which operate on resources not owned by ENIGMA. Examples of these include file_bin_open, surface_create, and the list goes on. Still more functions change the overall game state in ways that are conducive to leaks, such as ds_list_create, ds_map_create, ds_grid_create, sprite_add, sound_add, background_add...

You might have figured it out by now, but these functions all have one thing in common: they don't agree with game_save. In Game Maker, game_save and game_load can dump all state information and re-load it in a matter of milliseconds. But this excludes work done by the functions above (or at least the majority of them). I can have ENIGMA generate state dumps for objects, but I can't have it generate them for all functions. Least of allsurface_create.

This is problematic because game_save is a fantastic placeholder for custom save mechanisms, and its emulator-like dumping is great for debugging. In fact, that's the major reason I am interested in it right now. Rusky and I were discussing a crackpot lunatic's great idea for game debugging, and for some reason, I'm dumb enough to want to give integrating it into ENIGMA a try. But I can't do that (despite Rusky's insistence otherwise) without a working game_save/game_load.

The easiest way I can devise to handle the situation is to create a namespace for state restoration functions. So in the header that declares file_find_first, you'd see something like this:

string file_find_first(string pattern, int attr);namespace enigma {namespace dump_state { vector<char> file_find();}namespace load_state {size_t file_find(char*);}}The compiler would then include a call to everything in enigma::dump_state in game_save, and a call to everything in enigma::load_state in game_load.The call to dump_state::*() returns a vector of bytes to write to the save file. The call to load_state::*() accepts a pointer to file content (ideally memory-mapped), reads in the state information in its own dump format, and then returns the number of bytes read. Thus, it's critical that each function write only what it will read and reads everything it might have written. Otherwise, it will ruin all the other state loads.

Now, in the case of file_find_*, it's pretty easy to dump the current filename and just iterate back to it. But in the case of surface_create, dumping all the surfaces to a file might be a large, cumbersome task that a user would want to disable. And in the case of sprite_add, keeping track of all added sprites is more code than some users will want in their game. That said, I think it would be frugal to make sure this logging code can also be toggled off, with a preprocessor. Later on, I'm not opposed to a Game Load event.

With a little luck, game_save will be powerful enough to do what I am planning. If not, we may need to use some more preprocessors and include code to keep track of state changes in a more minimal fashion. I'll worry about that later. For now, game_save is something isolated enough to be concerned with.

I never make changelogs; this is a personal mistake I'll attempt to work on. I am using this thread to document notes for developers and changes to the new compiler framework. I will place these changes in the Wiki after the fact. Excuse the Wiki list notation in the meantime; there's no sense in my formatting it with SMF's syntax if it's just going to be * in the end.

Notes:* Files in the CompilerSource/JDI folder are meant to be read-only. This is a verbatim copy of the JDI repository; please make any changes there. We might end up renaming this to JDI-read-only to comply with the precedent set by other projects.

Changes:* EnigmaStruct is no longer the main argument to compiler functions; it has been wrapped in a new class, compile_context.** This class also contains the global object parsed_object *EGMglobal; the other parameter to most compile routines.** This class also contains FILE *gameModule, the other parameter passed to most resource writing routines.* There is now a separate resource_writer class to handle—you guessed it—writing resources. This is not only so we can support multiple languages, but also so we can support multiple resource export schemes (eg, internal, external). This is generated by the compile() method of the corresponding language adapter.* Compile functions are now all snake_case, not some_SnakeCamel_Bastardization

Changing:* The way JDI interfaces lexers and contexts needs changed—these are independent concepts.

I'm going to stop playing with blocks today and try to get this last JDI error fixed, at which point ENIGMA's engine will parse without error on Linux. I was going to boot up Windows and test it, but then I realized I don't feel like doing that at all. Plus, if I have people do this, I can test on multiple systems.

I'm going to assume that the preprocessor works without error, and so for the purpose of cross-platform testing, supplement GCC's. What I need from you is to do the following:

1. Open a terminal.2. cd to your ENIGMA folder.3. cd ENIGMAsystem/SHELL3. g++ -E SHELLmain.cpp <YOUR COMMAND LINE BUILD PARAMETERS, AS PRINTED IN LGM> -o preprocessed.cpp4. Upload preprocessed.cpp here as an attachment, or to dropbox, or if it's small enough (which is unlikely considering you people won't stop including STL headers from main), pastebin it.5. Link me if necessary.

This will allow me to test JDI on arbitrary machines without booting or having access to any of them. The only thing that won't be tested this way is the preprocessor, which, as I said, should be working fine.

When this is done I'm going to go ahead and merge NewTemplates into master, at which point I will recommend integration into ENIGMA, but I will not advocate it or otherwise facilitate it, as it means writing the new compiler around it can finally happen.

I am particularly looking for a Windows dump, but any platform is welcome—the more, the merrier.