thoughts on scripting language design, use case research

Menu

It has been a busy time for me recently due to a project that was expected to push certain limits, though mainly in the area of 3D game development with SGScript. I was making a game.

Total development time for the game - two weeks. Some more time had to be spent on technology upgrades. The result is, of course, a very solid experience, which you can try yourself on a Windows PC.

There are some issues with gameplay and visual communication with the player in the game itself, I’m fully aware of those and a plan has been prepared, should I wish to push the project further.

Though I am still slightly amazed that it actually worked. I’ve had a few close calls, all of which were successfully bypassed, and the related bugs have been fixed to prevent it from happening again. Had I not known the technology before, the project would be impossible to make. It isn’t anymore.

To make it work on a mid-level laptop, there had to be some serious optimizations, such as software occlusion culling, due to a lack of time to make things completely right (with lightmapping). Having dynamic lighting all over the place, as well as bloom and distortion effects doesn’t help at all with a laptop. They can’t stand overdraw and I initially had quite a bit of it.

Model exporter code broke tangent vectors on just some surfaces, effectively disabling normal mapping on them. Bug was found and fixed, and players of the game can enjoy the delicious textured roughness on most surfaces.

The version of Bullet I used is actually shipped with a bug! Managed to find and fix it in time. It reversed the normal vector of intersection on certain sloped intersections, effectively pushing bodies through triangles at some places. It was a horrible, serious, game breaking bug that had effectively reduced my options on certain projects more than one time already. I simply had to find it, and I was in huge luck that someone had done it before me.

There is a spotlight culling bug that wasn’t fixed in the first demo version and I’m not sure that it is in this one but it’s rarely visible and it’s fixed in SS3D now. Basically, all spotlights had convex hulls generated with origin taken from the first spotlight only. Screen-space AABB overhead helped the bug to stay hidden, from me as well as others.

Game math library needed a huge upgrade with 3×3 matrices and quaternions.

There were some bugs in sgs-bullet resource handling, type ID system and before this game, it didn’t have support for convex hull shapes. I also found out that it was way too easy to create a memory leak by referencing self through entity in userData of a rigid body and hard to actually find out that this is the problem.

The result is that I now have a great 3D engine that continues to improve and is expected to be used on yet another game-changing project soon. It doesn’t support character animation quite yet but is partially equipped to handle that already, the work should be finished whenever I have time for that. Same goes for lightmapping and environment mapping and the level editor.

New TODO items:

make “object island” finder or API for it (it would locate and dump/return interlocked object sets that can’t be accessed from anywhere in SGScript). For manual, one-time GC work.

New plans: Multiplayer games! Not sure how far I’ll get with this but it’s worth a try since I’m interested in trying to make it work once again (have tried several times before with big gaps in between, it’s hard to get it working, let alone get it done right!).

Hopefully this is the first of many more editors built with this system.

As for the current TODO list (sgs-ui):

transform gizmo – something that would help move things around in a 2D/3D world

color picker – at the moment I’m not quite sure what’s a good way to make this; it needs lots of big images for proper sampling for which there’s currently no generation system prepared

item gallery/picker – probably going to extend the file browser for this, there already is an option to implement a custom file system for it

tile picking/editing template control – tilemap editors need this for tiles, and it just so happens that I have a tilemap editor in development

Side note – I did get to implement the “??” operators. Finally writing that code works as expected. Didn’t make much sense otherwise, with || and several versions of “first_not_null” functions.

And for SGScript, since I have completed most of the previous TODO, some new ideas have appeared.

DSL (domain-specific language) generation toolkit. I don’t like the idea of threads in a language as the C part of it is very uncontrollable. Can’t break past it without tremendeous preparations and restrictions. Instead, I would expect that a basic DSL compiler would be made that compiled instruction lists to special functions. It should allow for all kinds of out-of-order execution. More research & experiments are yet to be performed, though. I currently have no idea where this would lead.

Toolkit support polishing. When I was setting up my SDKs on another PC, I noticed a few holes of varying sizes that need to be patched up.

I’m also looking forward to putting together a 3D engine/editor combo as soon as possible. And that’s all for now.

Upgrading C++/BC (from that nearly ancient TODO list). GC-marking, dumping, will need the new object method system too (must be optional for users).

Preparing to release multiple add-ons as binaries. Next thing to do here – ensuring the makefiles are OS X – compatible.

sgs-ui layout system redesign, data editing control, components. Should result in a noticeable performance / modularity / usability increase. Example: it will finally be extremely easy to add hover animations. I’ll show some videos when all that’s done.

Post-LD game testing took some time so the list is rather short. I intend to allocate less time for that from now on.

Web applications these days are taking increasingly more space on the market and attention from billions (if not more) users worldwide. They have been there for quite a long time and while the overall level of complexity is amazing, it is also preventing those applications from ever truly being the best, UX-wise, and thus are not capable of replacing native applications neither completely nor partially for most use cases.

Due to browsers becoming increasingly common, they have been made to work as application delivery platforms. Because of that, user experience suffers. Offline availability, performance, input responsiveness, graphical fidelity and uniformity across platforms – all of these things are compromised for the sake of getting something on the screen quicker.

It’s hard for me to watch this happen. UX is too important to let this quantity rush mess it up. This is why I’m trying to identify in what ways web applications have succeeded and outperformed native applications and finding ways to turn that around or at least minimize the difference.

Last week I was working intensively on a delivery method for native applications. This method is centered on an application that acts not only as a game archive and setup system but also as a platform for instant extract-launch-clean workflow support. For the user, it means that only two or three actions at most are necessary to run something – download the application, open it, and optionally click on a button to start it.

NAILER is the name. Native Application Installer / Launcher. v0.4 supports building a self-sufficient executable that can deliver an application and let people run it from anywhere, anytime, provided that they have some free space in their temporary file folder. Currently only Windows is supported since a lot of stuff has to be done in completely different ways for other operating systems. How fast the support will come depends on you. I’ve implemented the MVP, it’s time for you to decide how useful you think it is.

In closing, all I want to say is – let’s test it and see how useful it is. I’ve found lots of unexplored space while building the system and I would like the chance to explore it – support the project by using it and I will make it even better.

I’ve taken some time to update my Ludum Dare game with some checkpoints (check the v1.1 version if you want to see them in action). Felt like I had to do this not only because people asked for it but also because it’s a very important thing to have for any bigger project. And because I’ve never attempted to do something like full state serialization for a SGScript game yet.

So I’ve wrote some code, and it didn’t go quickly enough, I felt like I hadn’t covered all use cases so I decided to write a plan first.

1

2

3

4

5

6

7

8

9

10

11

12

13

14

15

16

17

18

19

20

21

22

23

24

25

26

27

28

29

30

31

32

ENTITY SERIALIZATION

--- unserialization process ---

> restore all entities

> do not reload heavy data (textures, meshes) if possible => do not simply destroy and recreate such entities, try to preserve them

unserialization may...

- create a new entity (no matching entity exists on the level that would be eligible to step in)

- destroy an existing entity (entity could not be matched to any one to be recreated)

- update an existing entity (entity could be matched as a good enough fit for updating)

entity should not be recreated if possible so a reuse-testing function is added, it checks if the entity has the same type and it loads the same mesh;

entity has both parametrized and internal description data;

a “singleton” entity (no parameters, created once, never rebuilt, no more than one instance) is serialized – reuse-testing function only checks the type.

Even though it was simple to implement, I’m glad I didn’t do it at the jam. It took some planning and well over 3 hours (split over two days) to plan the design, implement it, fully test it and learn to use it well.

Before I proceed with the review, I’d just like to mention – this is the game I made. You might want to have a look at it before reading further to have some context when I talk about the specifics.

What went right: development estimates, luck and controlling chaos & unfinished code.What went wrong: lack of a fully developed engine / editor combo, deployment process.

Development estimates and luck – there’s not much to say about those. I always had spare time on my hands to fix whatever popped up, there was no stress to get something done in time, no major feature cutting. Might be partly related to the lack of a fully developed initial plan. I just went in with my tech and hoped I could get an idea of the gameplay halfway in the process. And that did happen.

Controlling chaos & unfinished code – what I had at the beginning was a completely unfinished engine. No working physics, not even fixed-key input handling or cursor locking/relative mouse movement, no pause screen, no 3D-related entities (sprites, lights, meshes), no character physics, no cutscene system. Things so many people take for granted to be working and polished when they begin working on a Unity or Unreal Engine based project.

It was very easy to write all that, provided that there was (as in my case) some previous experience with all of those things. It was also easy to add game action hooks to the engine (like non-entity tick/input events). I also dropped the original entity system in favor of a much simpler one (each entity is defined by a creation function), fixed some bugs in the engine too.

A note regarding code simplification: Every dynamic language takes a lot of weight off your hands with dynamic (re-)binding, runtime (re-)evaluation, first-class functions. My advice: trust those systems. It will save you a lot of code generally spent on the premise that any such system could fail to be robust enough. They almost never do.

Lack of a fully developed engine / editor combo - as much as I liked to write Bullet Physics wrapper code and fix bugs in my engine, I would prefer to spend that time somewhere else. Or maybe not, but others definitely would. People would also prefer (and here I join them) to use an editor to add things to scene, not copy & paste coordinates out of Blender. It’s what I did for the position data. Writing any other non-spatial/visual data is just fine, though. I would in fact prefer writing to fancy controls. But not things that need immediate visual feedback.

Here’s a few more things that should’ve been available to me:

Specular lighting – regardless of the type, it requires strong visual control. I have lights from far away leaking reflections through walls. I like unattenuated (by distance) specular because it’s the only kind that looks good and real. But it’s much harder to control in large scenes without shadows.

Occlusion culling – it was actually in development but not quite finished yet. It would probably help control those lights as well as improve frame rate and allow to increase levels of detail a lot.

Mesh data extraction from the exported file format – I actually had to write this one while working on the game, otherwise there would’ve been no physics, and no game.

Last but not least – support for other platforms. I really wanted to get more people to play this. Right now, it’s not happening.

Deployment process – there’s no errors like the ones you don’t get on your machine. The release started off with a “missing libgcc_s_dw2-1.dll” error. GCC was nice enough to screw with me at some point and didn’t let me know exactly when that happened. These things should be statically linked by default, though I should’ve checked all DLLs myself. It’s always too easy to forget that.

What followed was a silent failure. Most likely something simply wasn’t extracted from the archive but the engine should’ve notified the user. So that’s a bug, and it’s fixed now. Someone else couldn’t start the game without any explanation, that could’ve been the same bug. Or another one…

I received another report about a “variable of type ‘null’ cannot be called” error. Classic, means that the function doesn’t exist or the variable is not a function. Checked the file at the specified line – SS3D_CreateRenderer – SS3D addon silently wasn’t loaded. I’m going to upgrade all include messages to errors (SGS_ERROR) to fix that. Optional includes will be possible with either pcall or @include("...") (error suppression on function call). I should’ve handled such errors manually, though.

Edit: In fact, it would be best to not link optionally available DLLs statically. When I finally get to making multiple renderers for SS3D, it should be possible to run the OpenGL renderer if the Direct3D one isn’t available or at least notify the user about the action to take automatically.

So, what we can conclude from all that? Dependency issues are generally one-off (with some recurrence) so there isn’t really any need for automated tools. As for the rest – the value of a language strongly depends on the amount and value of bindings it has.

And that’s all I need to know to determine the direction of development.

Before I state the successes that were already achieved, I just want to point out that the game is not finished yet, though well on it’s way to be delivered after 12 hours or so. Here are the blog posts about the game / my participation in LD30: 1, 2, 3 and 4.

As the preparation for LD30 went on, I became increasingly interested in making something really big. Something 3D.

The original idea was to make a third person platformer but as the theme was announced, I was not sure if I could pull it off. After all, the 3D addon to sgs-sdl does not yet support skinned meshes, which would certainly come in handy for the playable character.

What I did know and hold on to was the idea to quickly push out an incomplete 3D physics engine wrapper. In well under 24 hours of total investment (not an uninterrupted period of 24 hours, mind you, I’m not *that* crazy), a usable Bullet Physics wrapper was made, with a discrete dynamics world, rigid bodies and ghost objects, box/sphere/capsule/plane/triangle mesh shapes, raycasts, convex casts and possibly something else I’m forgetting now. Anyway, check the /bin and /src directories in the link above to see exactly what’s been implemented.

C++ binding compiler was slightly modified in the process to support inheritance. Without it, I couldn’t have done what I did and that’s not a part of the success. But it is a product of trial by fire, a process which I believe strongly in.

So, bottom line – one dare done successfully, one yet to be completed, a new library was created and SGScript was upgraded to support more use cases. I look forward to the moment when all of what I’ve used in these few days will be fully documented and available for others to use, with much less effort than what was required from me.

As I’m preparing everything to get some quality stuff out (think native modules), not only I have to get documentation generation system in order for it (which I’ve almost done anyway), it is also important to improve other aspects of production, such as binary file generation and testing. Regarding those things, four distinct improvements have been implemented recently.

#1: build preparation script has support for partial behavior overrides

This basically means that it’s possible to change commands that will be executed for full rebuild / follow-up cleaning and a few other things. This enables the build script to be reused for projects that include SGScript, as shown here.

#2: makefile has been split for reuse of the core features

Reusing the platform-specific stuff handling code allows to reduce the amount of boilerplate code for additional modules, making things a lot clearer. Not a particularly big deal but removes a considerable amount of weight that would’ve been otherwise become a huge problem in the future.

#3: testing system is made to support running from specified folders

This has the benefit of being able to reuse the original SGScript test framework for other projects, making it easy to quickly see, at any moment, if the code is running as it should. Regression tests are a safe bet if you intend to deal with each bug only once, made easily available by this change.

#4: SGScript version checking macro

This thing should reduce the amount of version mismatch-related crashes to a very round number, zero. These things may happen if different versions of SGScript reside in current directory and the PATH environment variable. Generally the missing symbol errors should pop up but as the API becomes more stable over the time, it might be better to avoid relying on such luck.

So there you have it, four simple changes with substantial benefits. What can be inferred from this is that the first fully produced and implemented SGScript module should appear rather soon. Others are expected to follow that example. All goes well, by the end of the year lots of modules with tests, full documentation and binaries should be available.

This post possibly should be called ‘Meta-objects, part 3′ but the focus has slightly shifted. This time we’ll look at the upcoming features of SGScript that will allow property access customization.

1

2

3

4

5

6

7

8

9

10

11

globalcolor_iface={};

functioncolor_iface.__getindex(k)

{

if(k=="x"||k=="r")returnthis[0];

if(k=="y"||k=="g")returnthis[1];

if(k=="z"||k=="b")returnthis[2];

};

functioncolor(r,g,b){returnclass([r,g,b],color_iface);}

c=color(2,3,4);

printvar(metaobj_get(c),c,c.x,c[1],c.b);

This code creates a class (metamethod-enabled object with a meta-object attached, if you care for the exact term), based on an array object, overriding its property retrieval to return indexed items based on specific names.

This reminds me… the storage could be more efficient on these things (arrays require 2 allocations – one for the object, one for the resizable list of variables), some day I might add a fixed array object if it proves to be a valuable investment.

Even though the concept is simple, it wasn’t possible to do this from scripts, only from the native environment. However, it is mostly a convenience feature, due to the vast amounts of user code being run here, it should not be expected that this is going to be fast enough for everything. At most, in terms of performance, it can only save memory by allowing to avoid copying data and put all kinds of links to that data instead.

But that’s not all. It is also possible to set properties in a similar way:

1

2

3

4

5

6

7

8

9

10

11

12

13

14

15

16

17

18

19

20

21

globalcolor_iface={};

functioncolor_iface.__getindex(k)

{

if(k=="x"||k=="r")returnthis[0];

if(k=="y"||k=="g")returnthis[1];

if(k=="z"||k=="b")returnthis[2];

};

functioncolor_iface.__setindex(k,v)// <<<<<<<<<<<<

{

if(k=="x"||k=="r")this[0]=v;

elseif(k=="y"||k=="g")this[1]=v;

elseif(k=="z"||k=="b")this[2]=v;

elsethis[k]=v;

};

functioncolor(r,g,b){returnclass([r,g,b],color_iface);}

c=color(2,3,4);

c.r=5;

c[1]=7;

c.z=9;

printvar(metaobj_get(c),c,c.x,c[1],c.b);

This piece of code enables setting the same nonexistent properties that could be retrieved. And this is where the really good part comes in: it is possible to intercept property assignments to allow custom code to be executed. This allows for various useful things to be implemented, such as:

This is useful for localizing behavior of each property in the code (put related stuff together) rather than grouping code by system interfaces, which can be unintuitive at times. And the mm_(g|s)etindex_router functions guarantee two things:

you won’t need to write all that routing code over and over again (but you can if you really want to)

whenever a JIT virtual machine might be done, special optimizations could be attached to those functions to improve performance

So pretty much every angle has been covered at this point. As I’m writing the post, more changes and bugfixes are applied, all of the mentioned features are rather thoroughly tested.