Saturday, February 7, 2015

Me, myself and Delphi

Let's talk about programming, as it has been a while. A programming love-story that is, Fifty shades of Grey material. Ok then. Right now I'm taking some bold steps; migrating from Delphi7 to Delphi XE. Back to the future, but then the other way around. As much as I like Delphi7 -we go back a log time as you will soon find out- it got surpassed by others. As every aging married man has to admit sooner or later; your wife isn't as pretty as she used to be anymore...

Puppy love
It must have been six- or seventeen years ago when I first met her. So far, my relations didn't go very well. There was that first kiss in kindergarten from QBASIC, where we made a text-based game with its famous PRINT commands. But I cheated on her, by tweaking the "Lingo" game timer from 7 to 70 seconds. That must have been the last time we spoke.

When I got a little bit older, I tried to date C++. That didn't work out too well either. She was the type of girl that comes with a thick manual. Always complaining about her syntax, nagging about linker errors, undeclared kisses. Impatient teenagers as we were. It was fun as long as it lasted. Or actually, it wasn't fun at all. In short, I couldn't read English books, nor did I speak C.

I always wanted to become a "programmer" (couldn't even spell it back then). Which is somewhat strange in itself, as I'm more an Alpha than a Beta. Numbers, Math, physics, chemistry... not really my cup of tea. I wasn't too bad at it, but my chemistry was more with, erh, hmm, "making stuff". And then I mean in a creative way, because my hands aren't even capable of holding a hammer. Drawing things, making up fantasy worlds, that kind of stuff. I didn't become a programmer because I liked brain-crackers or solving problems with logic. I figured in order to make (design) games, you had be able to program them. Well, if it takes writing zeros and ones on a computer, then so be it. Whatever it takes to realize my fantasies.

In that context, the first QBasic or C++ dates were a bummer. Nope. There are no ready-to-go "draw cool game figure" or "play shotgun sound effect" instructions. Browsing through the books I lend from the library, it seemed the highest possible accomplishment was a DOS console application that queried the user whether he was an asshole or not, using ScanF commands. Thanks, but no. As a teenager, I have more important things to do than studying books for just that. Like... drinking beer. Or playing games instead of trying to make them.Dad received QBasic and a big bunch of programs + other course material... on cassette tape! No idea how the hell he did it (I must have been 11 or so) but somehow he managed to stream the tapes onto our 286 computer.

Going to college
But there she was. In 1998 we went to "Hogeschool Brabant" (now Avans in Breda I believe), just to have a look and get an idea of possible follow-up studies. Fortunately, for my age I was still kind of confident about the (game)"programming" thing, so I took a look at the computer studies section that day (though architecture also seemed fun and almost pulled me out).

She smiled at me, behind the glass computer screen. With the help of a step-by-step paper, we were to make a slide-puzzle program. From what I remembered then, QBasic looked like a blue pig. C++ looked as if she was always on her period; sulky, full of warning messages. But here was Delphi -3, I think, forgot to ask her number. She looked like an angel. Or no, like that Greek mythological bitch... oh yeah, Delphi or something. What made her so pretty, was the lack of a smack-in-your-face code editor, and a kiss-on-your-cheek IDE. A what? Integrated-Development-Environment silly. Instead of reading a fucking book in order to make a "how old are you? Press any number" DOS console program with dozens of lines, I just dragged & dropped buttons on a canvas. I never forgot the term "Property Inspector" since then, as it would allow me to edit colours, sizes, positions, captions, and all kinds of cool things. Without coding a single letter! Exactly what a not-so-Beta guy like me wants.

In the meanwhile, other kids from my class were complaining about how boring it was to follow shit step by step from a paper. But I was in trance. This program would be my portal to making games. Just drag & drop a sprite on the form-canvas! Of course, after some steps we had to double-click a button, and suddenly I was inside her. In her code-editor I mean. But, still being high in the air, I made my decision: I would marry that girl.

Just say yes
And so it happened. My father bought her for three goats, and not much later she got pregnant from "project1.exe", a revolutionary image viewer program. About 60 other kids got born. Or well, most were prematurely aborted really. But to name a few of our bastards:

It's a shame we never took pictures of our children. And erased hard-drives. Yes, we had rough times,
but we loved each other no matter what. Going to that same "Hogeschool Brabant" later on, I was tempted by others. A bit of C/C++ again at collegue. And Java, what a ho. The entire school had her. But I wasn't too charmed by her slutty "console.output.write( string.stringFactory.makerClass('Screw you world.') )" way of saying things. Oh, and there was Clean. A "Functional programming language" like Haskell. What a nerd. Functional my ass, I couldn't even write the "How old are you?" console application with it. No, despite the temptations, Delphi and I trusted each other.

The truly old (2D game) shots are gone mostly, but I did found this. Back then you could get pretty good results with Milkshape models and lack of photo-realistic shaders.

What made her so special, besides her awesome IDE front, was her easy going slang (Turbo Pascal). Not that cryptic and far more forgiving than C/C++. And unlike Java or the likes that come with dozens of rules, methods, and restrictions, Delphi gave me some much needed freedom. Do as you like, she would say to me. I could use pointers if I pleased, and we didn’t rely on virtual-machines or runtime packages. The best of both worlds; flexible and advanced as C++, yet easy and flirty as Java.

Midlife crisis
Later on we started making a living together. Besides making half-finished game babies for fun, we now would make money by developing business software. Database programs, weighing applications, document generators, you name it. My heart was still with games, but face it; chances are small you’ll make a living out of it. Having my responsibilities as an adult man now, I had to earn a buck to support our family. Fortunately, Delphi always stood on my side. Alpha or not, I got pretty good at programming, knowing her thoroughly.

Our most ambitious venture so far must have been a 3D game engine, which resulted in Tower22. Together with a few business program, our real children. Some of them already flew out and left home. Tower22 is a raging ADHD teenager that requires lots of attention though. And frankly, I’m a bit tired of it.

Not of Tower22, don’t get me wrong. But of Delphi 7’s attitude. Often tired, not keeping up, wearing old fashioned grandma clothes. Maybe it’s not fair, but I feel she is just not cooperating anymore. We lost that spark. And I would be lying if I’d say I didn’t flirt with others at the office. Visual Basic .NET for example thought me Kama Sutra with better debugging options, and always up-to-date support for 21th century technology. And Qt convinced me that C++ didn’t have to be dull and impossible.

So I swallowed some pride and sentiments, and started dating a smoking hot new chick; her younger sister! Delphi XE 3. Delphi7 doesn’t know yet, she still washes my socks and cooks supper. But honestly… I already packed some of Tower22’s Teddy bears and Pajama’s, as we secretly move over at Delphi XE’s house during the weekends. “Business trip for work”, I would say to D7. But who am I kidding?A D7 shot of the T22 Map editor. And with "Classic Style" code editor coloring of course. Didn't completely forget about QBasic! My friends would *always* see this blue screen whenever they entered my room.

A better future?
Finding a better home for our son Tower22 isn’t easy. Obviously he will miss the warmth and security Delphi7 gave all these years. I thought about moving over to C++, but that would be too much of a shell-shock. Having to re-program the whole kid… no. Let’s give the Delphi family another chance.

So I copied the entire Tower22 folder and made new XE project. You can clearly see this little girl caught up with the present. Kids these days… Just take a look at the IDE for example. I’m entering a Space Shuttle cabin with all those new panels and buttons (but less space for the heart; the code editor). Plasma Thrusters engaged Spock. Another exciting feature for me –working a lot with vector math- are record functions and operator overloading.

Yet it’s funny I’m still seeing quirks and characteristics of her older sister. Or well, funny… more like encountering some of the old shortcomings again. Having to include each and every unit again and again (instead of automatically including child-units of a top-level unit like C does). Namespaces were introduced, but now “import all” like Java can do with “import OpenGL.*”? Or how about the somewhat wacky not very helpful auto-code-completion? Qt does a better job. Oh well. It must be in the family DNA. Or maybe I’m just not experienced enough yet.

Unicode tears
Our first big argue was/is about UniCode strings though. Of course, before just dumping my beloved Delphi7, I did some research on the internet. Googling Delphi XE, hacking her Facebook, watching her on Tinder… For the sake of Tower22’s health, I was concerned about how much effort a code migration would be. “No problem!” “Just do it!” was the bottom line. Ok, ok, if you say so…

And, how did it go? Any problems? Let me try to describe… It felt as if eating logs and going to the bathroom. As if fighting with a swarm of penguins barehanded. As if an elephant baby crawled back into me, had to give it rebirth, and it came back without a nose. And three ears.

No, it didn’t go without a struggle, Delphi XE crying, and me yelling at her. And shoot, we’re still not finished! She keeps saying AnsiString is so 2000 and one, I keep saying everything was better with AnsiStrings back then. I’ll admit, hanging around with Delphi7 for so long may have affected my judgement. Don’t forget, I’m an old pervert dating this young cheetah. I know computers have more memory, and AnsiStrings - having 1 byte per character only – have a limited set of characters. But… do I really need Chinese characters for twice the price?

She would defend by saying Delphi still does support ansiStrings, if I really want to. “Just typecast them old nagger”. Just typecast… just typecast, clearly you have no idea what you’re talking about, woman! You never grew up with limited memory, old DLL’s and binary files, spoiled brat!

Baby? Ah no, I’m sorry. I didn’t mean to - please baby, now don’t cry. You’re not ugly. But please, you have to understand that:

• Tower22 uses many different binary files
• Tower22 uses many (self-coded and not self-coded) DLL’s
• Tower22 used –and forgive me if that was a mistake- “String[somelength]” types.
• There is no proper equivalent! “AnsiString[32]” won’t work.
• You can typecast all you want internally, but ping-ponging UniCode strings towards a DLL that expects AnsiStrings or arrays of Ansi chars just doesn’t work! (and vice-versa).
• Struct or type sizes when involving the Delphi standard “strings” will give a different size than it did in Delphi7
• This will give invalid reads or writes when dealing with binary files.
• The compiler does not throw an error when forgetting to (properly) typecast from one to another.
• The whole game + tools and DLL’s is about 400.000 lines of code. Good luck finding the issues.

I ended up with making -as if we were making an old C program without any string library- a bunch of string conversion functions and tried to locate where they would be needed through the entire code. Not saying that I’m inclined to keep using ansiStrings, but the first goal of migration is to make things working again. First a solid basis, then we can chat about replacing old types and methods for newer ones.

Group-sex
All in all, I feel bad for 7, but in the age of 64-bits, more and more non-Windows platforms, and moreover the guarantee of not falling too far behind, I think we made a healthy choice. It sucks that Tower22 crashed during loading now, but give me a few days and it will work again. I hope. On the bright side, it forces me to have a clean-sweep. One plus about having to scan the whole code for those stupid Ansi/UniCode clashes, are the visits to older parts. It’s like cleaning the house; you’ll find all kinds of crazy things, long-lost items and other fond memories.

One major advantage of Delphi XE is the Project Group feature. Modern programming environments usually support multiple projects, so you can swiftly jump between projects, and debug all your way through. Delphi 7 didn’t allow to open multiple projects (though you could jump into a linked DLL while stepping), but XE fortunately does. It may not sound like a very big deal, but to me it actually is. Doing this upgrade isn’t only about just updating for the sake of updating. Not only do I want to clean out old garbage code. I want the re-structure the engine in more logical parts / modules.

And yes, that involves a lot of rewriting, but once I’m on the motherf*cker, I’ll go warp-speed. You see, typing code isn’t the major time consumer. Googling how-to-code, reading papers about graphics, trial & error, and fixing mistakes (in shitty code) is what really sucks up time like a black hole. Remember that many of the T22 code was written while learning. Never did shaders. Never did FMOD or Newton physics. Never really thought about flexible entity systems, or multi-threading. For a “first version”, it’s actually quite amazing that it works pretty well. But if I had to re-do it, I would definitely do it a whole lot better. Smarter. And that’s what I’m about to do.

First of all, I will do it slowly, step by step. At all times, the game must remain “working”. For example, one of the first (and simplest) moves is to separate the whole common types and math portion. I made a separate module that holds all the Vector, Matrix, String and other types, plus all of their functions. Dot products, Matrix multiplications, string help functions, and so on. Once up and running, I’ll remove the old functions and types from the “old code”. The same will happen for drawing (OpenGL) functions, Cg, and so on.

I’m not 100% confident of how to modularize though, but maybe some Delphi readers here can give directions. Initially I thought about making a DLL for each module. Thus for example, a “Utiltities” module for all the basic types and common (math) functions mentioned above. The nice thing about DLL’s is that you can reuse them, in another environment. For example, you could attach them to your Visual Basic or C++ program. But three major disadvantages are:

• Slight overhead for calling another module (note that math or draw calls happen a lot!)
• Not sure if I can export OOP (classes, records with functions, et cetera). AFAIK, DLL’s can only export functions, simple types, or Interfaces. Could be wrong though.
• Harder to debug when dealing with shared memory / released pointers, et cetera

A more logical move would be static libraries. You include a library, and it gets baked into your project as if it was just a part of one and the same project. Now here is where my knowledge gets flaky. I believe C has *.A or *.LIB libraries for that. But what is the Delphi equivalent? I guess it is “BPL Packages”, which can either be “design time” (static), or linked at runtime, like DLL’s.

Currently I experimented by making this “Utilities” library as a separate package project. Then the other host project (“the game”) would simply include its units. It works, but I feel somehow that I’m doing it as it should be. For one thing, can’t I just install that package instead of having to add a search path to all of its units?

Well, details. The most important part is that the code will get a reboot – hopefully without bringing the game “offline” too often- resulting in better, cleaned, more compact, and more logical code. And also, having more & smaller modules, a better ability to let other Delphi people help me on certain modules. Would Delphi XE be a better choice than staying with Delphi 7, or moving over to C++? Impossible to tell, and maybe it doesn’t really matter anyway. What matters is a working product in the end. But so is keeping up with the technology, instead of sticking in your old dusty comfort zone.

14 comments:

Sounds like me, talking about Delphi. I managed to migrate my projects to C++ Builder 6.0 back then and wrote a parser to translate Newton physics header to Delphi, since my first demos with this library were amde with Delphi 7 + GLScene for the 3D rendering.After porting to C++ Builder, the step towards VC++ was pretty much a breeze. I'm still using C++Builder for my visual level editor, but the game and the engine itself is in Visual C++. It's just a dll with a plain old C-style interface, just like a win32 API dll :)

Packing your stuff and moving to another house is always a hassle. It takes time, money, and a broken spine. But, at least it allows to have a critical look, sort your stuff, throw away what isn't needed, and re-decorate. In other words, usually the code comes out better :)

Yep,refactoring the code almost always helps to get it to a better state. It even helps understanding and refreshing the memory, what the code does and why it does it that way. I would suggest you embrace C++, not really suggesting going to DirectX.What OpenGL version the game is "written around" at the moment ? I guess it's OGL 2.1 ?

Got my daily portion of C++ at work already (harvester touchscreens and such), so for the sake of having fun, not having to rewrite too much, and to keep up with Pascal as well, I decided to stick with Delphi. Though I did consider C++ / Visual Studio honestly.

OpenGL2x headers were/are used, but that's another thing to clean up. Getting rid of all the "glVertex" calls, and replacing it fully with VAO / VBO, et cetera. Though, apart from performance, it doesn't really matter which GL version is used in my experience, as the GPU/shaders are doing most of the stuff these days.

I'm thinking about switching from Cg to GLSL shaders btw, but again, it will be a load of work and it tightens me more to OpenGL. Then again Cg sounds a bit dead to me. Few updates / tutorials.

Delphi itself helps programmers to create great stuff, because it stays on their way as little as possible. I remember the Newton forum and the greatest demos were made with Delphi. I think that the RAD tools help a lot too. To my experience prototyping is very easy, when you have a full blown RAD IDE on your side. I myself deserve almost anything to C++ Builder and Delphi when it comes to my projects and even my game. I cannot imagine developing my game without the visual tools I sketched with the great VCL. It helped me reload meshes, textures, shaders on the fly, change parameters on the fly, making a property editor for every object on the level editor etc. If I were to use MFC and Visual Studio only, I would be lost by now, probably. GLSL is indeed tightens you more to OpenGL, but at the same can open your application to other OGL powered platforms as Android, WebGL etc. On the other hand, as you have already pointed out, these days GPU/shaders are doing most of the stuff and it was never been that easy to port to another API. For example porting to Direct3D ( if you already have all you shaders written in Cg) would be relatively easy, if you get rid of old OGL fixed function approaches like glBegin();Sometimes I feel porting my stuff to this and that platform, but in the end I press myself in the corner and decide(once again) that the priority of most importance is getting the game demo finished first! Of course, walking down the road with that thought in my mind, I try to step carefully and thinking not to tight myself in a way that would prevent me porting the code in the future.BTW, what Global Illumination technique are you using ? It looks great. To what degree it is dynamic ? Does it needs pre-processing ?

Multi-platform was never a target here because well... let's just first try to get it accomplished on a Windows PC for starters! Often hobbyists like me get over-ambitious and come up with all kinds of not-so-relevant goals for their engine... making it a never-ending project.

Then again switching techniques (or upgrading like now) does happen. If I were to use D3D instead, GLSL would be a pain in the ass. Having wrappers rather than libraryX functions scattered all over the place is definitely a plus then. As said, I'm not over-concerned about this flexibility, but sometimes the thoughts of using dated stuff haunts me. Delphi7, OpenGL, OpenCL, Newton, Cg... My guts say C++, D3D, CUDA and such are more popular / a step further.

GIA whole arsenal of techniques has been used, mainly semi-dynamic techniques but none of those really satisfied. Too slow, too much artifacts, too ugly. All in all too much sweat for relative poor results. And I believe A+ engines such as CryEngine or Unreal keeps making those conclusions as well for Voxel-Cone-Tracing, Light Propagation Volumes, and so on.

So I did a step back about 6 months ago, using pre-baked probes (basically 6 color cubeMaps). A lot of probes are placed all around the place, and measure incoming light (multi-bounced), and store it in a bunch of 3D textures that can be accessed later on when rendering walls, objects, particles, or whatever.

The amount of incoming sun/skylight is stored as a factor so it can be multiplied with the current "time of day" light setting, making it somewhat variable btw. So its still a little bit dynamic. As for swiching lights on/off on a corridor (T22 is mainly indoor), I still have to think of something.

Great, I like it when other valuable people make same decisions as me. That suggests me, that maybe I'm on the right track. I also settled down on using light probes spread around the level. I do not store the information in a 3D texture to save memory on spots where there are no lights. Instead I put light probes on places I need, render cubepams and convert them to spherical harmonics to save even more memory. At runtime I render those probes as deferred lights, passing spherical harmonics coefficients to the shader. You can sample those as cubemaps, using a direction vector. I also implemented a simple lightmapper as fallback solution and for mobile.

You could switch your old D7 wife for this (also not so new) but younger C# wife. But soon you would realized that although you have a pretty wife, she is quite dumb. She does not like kinky stuff( pointers, memory allocating memory), she imposes on you some strange rules (everything must be a object) and your old wify is two times faster cleaning home.

VC++ word is not perfect either. Just few issues which I have with it in terms of IDE:

- No easy way of detecting memory leaks (EurekaLog in Delphi works great).- No range checking for arrays.- No refactorings! (it is to hard to implement, you can download semi supported plugin with very basic options)- No indexing when debugging - MyVector[5] gives you error message in immediate window.

No experience with C# here, but my experience with its cousin VB.NET is that it works nice for -of course- Windows applications. Porting .NET to another Windows based platform (say Win CE / Mobile) is easy, and overall the .NET framework reduces development time. Being able to let the debugger jump from one program into another (even webserver based progs) is certainly useful.

But I wouldn't use it for lower level stuff, such as a game engine. Too much rules, no pointers indeed, plus I just don't want the Virtual Machine principle for high performance programs. I want to be in control here, not some framework.

Never used VS a lot for C(++) based programs, but Qt instead. C++ on itself is an old, grumpy, harsh, stuborn grandpa. But a mighty powerful one. Qt compensates here by offering useful (but optional) libraries, and a better IDE, making life a lot easier. Also the Valgrind memory analyzer helped me more than once finding stinky holes.

And now it's Delphi XE3 turn to impress me. So far it didn't, but that's just because a lot is still the same ... which is a good thing in this case. The project-group feature, struct functions/operators were very welcome at least. And I hope to develop an iOS or Android program with Delphi one day, because Eclipse+Java is a pile of buggy shit. Unfortunately, I probably need a newer Delphi version for that though.

Delphi's true problem is its ridiculous price. Why not offer cheap/free "Lite" versions, like the VS Express editions? Delphi's future relies on popularity, and therefore needs to feel comfortable with the new generations of programmers (students, hobbyists) to finds its way back into IT business. But what student, hobbyist or small business is willing to pay thousands of dollars for a programming IDE? Exactly.

Well, maybe I can put a few coins in the bag... by opening the Engine22 (used on T22) source for all you Delphi fanboys! Not sure if/when/how, but ideas are in the make...

Delphi gets better and better with each release. I was able to do an Android app after two days of learning in XE 6. Unfortunately there is no improvement on the low level stuff.

"Delphi's true problem is its ridiculous price."

You know what is even more ridiculous? That the price is not the same in all countires! Every member of the community will tell you that Delphi needs free version but they seam to be deaf in this matter. However there is a "lite" version called Delphi Starter Edition. It costs around 200$ if I am not mistaken. But I doubt that any single student will spend even 10$ for software.

However, do not forget that Microsoft does not live from VS. They could give away for free VS and nothing would have happened to them. Delphi from the other hand needs sold copies to survive. Embarcadero simply is afraid to offer a free version because that could meant that nobody will buy Delphi.

Sure Embarcadero needs to make a living. But compared to other packages, it just is a lot of money. And likely I'm not even using 20% of their offered features (C++, Firemonkey, Database tools, and so on). Then at least make a configurable / stripped version.

A free express edition is a gamble indeed. But if I were a student/hobbyist who enjoyed Delphi, I would recommend it to my boss/work/professional projects. Now it seems that none of those "newcomers" would ask for Delphi, simply because they never gave it a try. People usually stick with the packages they learned during their studies or hobby days.

I bet many people didn't always pay for their Windows, Office, Photoshop or other (expensive) packages. Technically that's a crime, but at the bright side those people will likely pay their buck once they get older, use it for work, make a living with it, et cetera. Because they know it's worth it.