Display posts from previous

Sort by

Really happy to report that it's been a tremendously productive two weeks! In the last log I spoke extensively about our various mechanisms for getting bleeding-edge performance out of a magic blend of LuaJIT and C. Since then, we've been building more and more 'real stuff' using those systems, working our way to release-quality code! The general theme as of late has been "wow, it just works!" Of course, the longer version is "wow, after 1000s of hours invested in technobabble and aggressive banging-head-on-desk followed by burning the codebase and restarting, things finally, at long last, just work!"

---

(I'm not going to separate by Josh & Adam this time since we were both working all over the place.)

Basic Game Objects & Components: Working!

Last week was the first time that we were actually able to build and use some real components and game objects purely out of our CTypes system (see last devlog if you don't know what that means). It worked beautifully. Right now we've only got System, Asteroid, Ship, and Nebula objects -- we're scaling up slowly as we nail down what we want the gameplay code to look like / how we want to be able to manipulate said objects in Lua. We've also got some basic components -- Motion and Transform, in particular. The former handles kinematics, the latter keeps track of position/orientation, etc. These are your bread-and-butter components for a component-based game engine (along with something like Drawable/Renderable, Inventory, etc.), so naturally we want to test the most ubiquitous components first.

The real stress test of our CTypes system came when we packed thousands of asteroids into a system, doing a rather naive update loop on them, and performing full rigid body dynamics on them (meaning I stuck a motion component in the asteroid definition and allowed every single asteroid to have accurate linear and angular dynamics -- something that we're probably not even going to do for release, because I still contend that dynamic asteroids don't add a whole lot of value..but hey, mod away). The speed was unreal. The bottleneck was, by a huge margin, the rendering (no LOD models yet). The actual update logic is so fast that, if I turn off rendering, I can get up to about 30,000 asteroids before I drop below 60 FPS on my machine. Again, that's a full rigid body dynamics step happening on 30,000 moving, spinning asteroids @ 60 FPS. And that is using very naive update logic, no cleverness about putting objects that aren't moving to sleep, no threading, etc. Adding in cleverness is only going to push that number even higher Anyway, I guess I can stop bragging about our performance now It's just that...I spent SO long battling FPLT -- and sometimes seriously doubting that I'd be able to give LT that 'epic' scale that I so badly wanted -- that...surely I deserve a little leeway...

Dynamics: Fast, Accurate

I mentioned the rigid body sim above, but it's worth mentioning it as a stand-alone point. To be perfectly honest, I've never had accurate physics code in LT. I mean, the linear part was accurate. Kind of. But angular dynamics have always been extremely hacky in LT. But those days are over, and we now have a genuine physics integrator with accurate angular dynamics (i.e. applying impulses at various points on objects results in correct torque & force). Honestly it's not really a big point, as you likely won't notice many cases of fancy angular dynamics in LT. But I think the takeaway is that this stuff was written in Lua (yes, the physics step is in Lua...!) and runs well within the acceptable range for us to support a massive simulation.

I suppose it's also worth mentioning that mods could certainly leverage this to 'go further' with the physics in LT. You all know I'm a fan of realism right up until it starts cutting into fun, which, in my experience, happens quite early-on. I don't intend for ship thrusters to use real impulses, because I don't want players to have to spend hours trying to properly mass-balance their ships to get inertial characteristics that aren't impossible to fly with. But for those of you who love that good old abused phrase "real Newtonian physics" -- well, mod away. I already wrote the real physics for you

Adam's BSPs are Stupidly-Fast

Several times over the past week I've had to turn around and tell Adam that his BSPs are just grossly over-performant. In our current LT sandbox, we can click objects to select them, view data about them, orbit them with the camera, etc. Selection requires raycasting the scene, which, in turn, requires raycasts against the BSP trees that provide the acceleration structures for meshes. Currently I am not using a broadphase structure, so we are literally raycasting against every single asteroid / ship in the scene. Really, it shouldn't even work. It should just melt the computer when you have thousands of complex objects in the scene. But it doesn't. In fact, the fps hit is so low on my machine that I forget how negligent I'm being.

The biggest place that raycasts are used (biggest in terms of 'most likely to cause performance problems') is in checking to see if a projectile weapon hits an object. Pulses, beams, railguns all use raycasts. Missiles are slightly more complex but still do some raycasting.

I guess where I'm going with this is: we can probably support like 10000000 million projectiles in the world at once. Or something like that. Basically, REALLY BIG BATTLES. 100 ships is going to be a cakewalk, apparently

Josh's Nebulae are Stupidly-Fast (and Pretty!)

I spent some time after-hours one night revisiting my most-prized algorithm: the nebula generator. I made it roughly 10x faster while also improving the visual fidelity AND making the generator more consistent, such that it yields less "really bad" systems. It was a fun night. We all know I'd die if I went more than half a year without touching 'nebula.glsl' But seriously, I'm happy about this, because nebula generation was one of the most expensive parts of system generation (by quite a lot). It's now much more reasonable, especially on less-powerful GPUs.

Render-Time Interpolation of Simulation State

We implemented it. This is a major technical component of games that need to have tight control over the game world simulation (like those that want to simulate universes). I've spoken about this in devlogs long, long ago. The idea is that simulation of the game world happens at a fixed rate, independent of the frame rate. This makes simulation more stable and typically more performant as well. It also brings a major challenge: your simulation state is no longer in-sync with your render loop, so now you have to 'guess' what the simulation state would look like at the current render time. To do so, you simply keep one frame's worth of historical state, and interpolate between the last and current state to obtain a smooth, linear approximation that gives the player the perception that everything is nice and continuous on-screen.

Major game component, done.

Dev UI

We've built (well, mostly Adam) a really handy developer UI to help with poking and prodding the game as we work on various parts of it. We decided that the real UI should be one of the last things we do (this was guided by the realization of how much time I've sunk into UI that later became useless when underlying game mechanics changed). Until then, though, we've got some nice programmer-artsy widgets to see what's going on.

Multiplayer: Just for Fun

Last Friday I implemented multiplayer. Just for kicks Don't get excited (or worried), it's not a feature and we're not shipping with it. By multiplayer, I mean I implemented a very naive peer-to-peer network architecture so that Adam & Taylor (a good friend/fellow programmer at the tech park) & I could all throw asteroids around in the same star system. We would find the biggest asteroid that we could in the system, then set our cameras to orbit it, then fight to push/spin it in various directions with impulses. Not exactly MMO-of-the-year material, but it works.

So, the UDP socket API in our engine works, and I've proven that it's possible to implement (some degree of) multiplayer with it. I hope the LTMP modding team is ready! :V (Also, I was on linux and they were on windows, so it works across different platforms, as you would expect of a decent implementation.)

---

Conclusion:

Everything is coming together. The road has been thoroughly and meticulously paved, after oh-so-many setbacks, and we're FINALLY getting to spend almost all of our time in Lua at this point, just pushing forward...which is a really good sign! Game constructs are being built and carefully probed for performance along the way, ensuring that everything is done 'the right way' so that (ideally) we don't end up with any nasty performance-related surprises in the end. That which is being built is being built to last. Thus far, all of that effort we sunk into performance is REALLY paying off, because the work is moving super quickly.

I know it's probably difficult to appreciate how much progress the above points represent -- especially when screenshots don't look appreciably different than they did several months ago in the early days of LT in LuaJIT. But if you've been reading, you know the truth: under the hood, it's a whole different ballgame. This time, the muscle under the hood is enough to get us to release. I'm genuinely excited about it (and let me tell you, it's not easy to excite someone who's spent the last five years doing little more than staring at vim )

From here, we just keep at it. Every day, more game. Every day, more LT. Then one day...

...you know

PS ~ Some eye candy just because. Nothing you haven't seen before -- pretty nebulae, decent asteroids, bad ship algorithms...probably too much bloom But I know you guys require food every now and then

I stuck a motion component in the asteroid definition and allowed every single asteroid to have accurate linear and angular dynamics -- something that we're probably not even going to do for release, because I still content that dynamic asteroids don't add a whole lot of value..but hey, mod away

Okay, fair enough that not every asteroid is fully dynamic, but does this mean orbital mechanics for planets and stations and zones of asteroids as a whole is now possible? Even if interpolated at LOD? Adding it for every asteroid might not add much, besides being able to lob asteroids at the enemy like a catapult! but having a system where the planets fly around in different orbits around a star, the planets have moons and the moons have orbiting stations? that WOULD add a lot.

we can probably support like 10000000 million projectiles in the world at once.

10,000,000,000,000 projectiles at once...

"From Pearl Harbor to V-J Day the Industry-Ordnance team furnished to the Army and 43 foreign nations 47 billion rounds of small arms ammunition, approximately 11 million tons of artillery ammunition, more than 12 million rifles and carbines, approximately 750,000 artillery pieces and 3/2 million military vehicles. this is estimated to be about 40% of the total war production on all sides"

So in the entirety of World War 2, about 100 billion projectiles were produced. And LT can handle 100 TIMES that at once... Holy Heavenly Fcuk!
So lets say 100 v 100 ships. each ship has an average of 10 guns, each gun can produce 100 rounds per second and the rounds live for 1 minute.
200 * 10 * 100 * 60 = 12 million = 0.00012% of engine capabilities...so basically 100 vs 100 is like a high school food fight in comparison to the big battles...
Is LT going to have Epic Battle Simulator sized fleets? Each ship helmed by a semi-intelligent AI flying in proper formations with some understanding of tactics and strategy?

Adam & Taylor & I could all throw asteroids around in the same star system.

Asteroid billiards! Brilliant!

Last edited by Hyperion on Thu Aug 31, 2017 9:59 pm, edited 1 time in total.

If you're trying to fill the multiverse, and you're not willing to consider the entire creative output of humanity as a starting point, you're wasting your time.

I needed some new LT themed desktop wallpapers for the next few months anyways. Thanks Josh!

Also, cross platform multi-player, very impressive. I look forward to seeing what Silver is able to do with this functionality. Is the team size back up to 3 or was Taylor only helping for a limited time?

Josh, old boy. Do you have Lua interfaces for the UDP sockets? Do you have TCP sockets?

Not that I can't just use UDP and re-implement the minimum TCP requirements.
It'd get slight more data/sec through, but really TCP handles a bunch of shit that just isn't worth re-implementing until after the core of the multiplayer mod is sorted.

we can probably support like 10000000 million projectiles [Baile optimistically reads 'projectiles' as 'guided missiles'] in the world at once. Or something like that. Basically, REALLY BIG BATTLES.
100 ships is going to be a cakewalk, apparently

Y'know.
When the day comes, and that virgin binge of LT has been had..
I'm probably going to have a cry after that. The joy overload variety.

Okay, fair enough that not every asteroid is fully dynamic, but does this mean orbital mechanics for planets and stations and zones of asteroids as a whole is now possible? Even if interpolated at LOD? Adding it for every asteroid might not add much, besides being able to lob asteroids at the enemy like a catapult! but having a system where the planets fly around in different orbits around a star, the planets have moons and the moons have orbiting stations? that WOULD add a lot.

Well, the problem with orbital mechanics / moving stellar bodies has always been about the collision detection side of the physics engine. It takes a hell of a broadphase structure to be able to handle moving objects that differ in scale factor by 6+ orders of magnitude without dying a fiery death. So I would say that's a different technical problem entirely. But I do think it's one that our broadphase structures will be able to handle in some way or another!

(Frankly, though, I'm more interested in stations being able to move than planets.)

Is LT going to have Epic Battle Simulator sized fleets? Each ship helmed by a semi-intelligent AI flying in proper formations with some understanding of tactics and strategy?

Like I've always said, we'll push it as far as we can until it breaks Where the actual number lands is dependent on many factors still to come. It will and in excess of 100 v 100, though

Josh, old boy. Do you have Lua interfaces for the UDP sockets? Do you have TCP sockets?

Not that I can't just use UDP and re-implement the minimum TCP requirements.
It'd get slight more data/sec through, but really TCP handles a bunch of shit that just isn't worth re-implementing until after the core of the multiplayer mod is sorted.

Yes sir yes sir. Lua bindings for all the engine constructs, sockets included. And yes, you can choose between UDP & TCP.

“Whether you think you can, or you think you can't--you're right.” ~ Henry Ford

Well, the problem with orbital mechanics / moving stellar bodies has always been about the collision detection side of the physics engine. It takes a hell of a broadphase structure to be able to handle moving objects that differ in scale factor by 6+ orders of magnitude without dying a fiery death. So I would say that's a different technical problem entirely. But I do think it's one that our broadphase structures will be able to handle in some way or another!

(Frankly, though, I'm more interested in stations being able to move than planets.)

Okay, I thought this might give a cool, but ultimately unnecessary addition, I too am more concerned about stations moving than planets.

Is LT going to have Epic Battle Simulator sized fleets? Each ship helmed by a semi-intelligent AI flying in proper formations with some understanding of tactics and strategy?

Like I've always said, we'll push it as far as we can until it breaks Where the actual number lands is dependent on many factors still to come. It will and in excess of 100 v 100, though

That's actually exactly the answer I was looking for. I was actually being a bit hyperbolic to check the state of pragmatic josh Good to see who is in control. But good to hear what once pushed the system to its limits is now a breeze. This now means that we could see rather populated systems in comparison to what we were seeing before, right? I remember you once saying that the number of AI wasnt really limited so much by the AI coding, but by the rendering and physics coding. Where once 120 NPCs in system was supposed to be about as populated a system as you would see, now it sounds like 120 may be more of a backwoods system, and the core worlds of an empire could see somewhere in the thousands? again, still much speculation being done, but if populations could be increased by a factor of 10 to 100, thats the difference between systems feeling like

If even some places not only feel alive, but are positively buzzing with activity that feels alive? I can't imagine a game that compares, not even an MMO, not even GTA. And for LT, that's a hope, not an expectation.

If you're trying to fill the multiverse, and you're not willing to consider the entire creative output of humanity as a starting point, you're wasting your time.

Also the nebulas looks beautiful. I think it's a good idea to work on the under-the-hood stuff, but If i may suggest something to improve on the graphics side, i think the asteroids could have better lighting / shadows, also the ships looks kind of square and bulky - compared to say Elite Dangerous.

Anyways, keep up the great work.

Last edited by erf on Fri Sep 01, 2017 11:22 am, edited 2 times in total.

You all know I'm a fan of realism right up until it starts cutting into fun, which, in my experience, happens quite early-on. I don't intend for ship thrusters to use real impulses, because I don't want players to have to spend hours trying to properly mass-balance their ships to get inertial characteristics that aren't impossible to fly with.

I spent a little time "enjoying" the mechanics used in the earlier builds of Star Citizen which entailed fighting to control your damn ship against all the thruster activity. If LT still gives me the sort of Freelancer control that the LTP version gave me I will be a very happy pilot.

I spent some time after-hours one night revisiting my most-prized algorithm: the nebula generator. I made it roughly 10x faster while also improving the visual fidelity AND making the generator more consistent, such that it yields less "really bad" systems. It was a fun night. We all know I'd die if I went more than half a year without touching 'nebula.glsl' But seriously, I'm happy about this, because nebula generation was one of the most expensive parts of system generation (by quite a lot). It's now much more reasonable, especially on less-powerful GPUs.

I'm always pleased to read of improvements in your "most-prized algorithm", Josh, but is there any possibility you could devote some time to your less prized algorithm for ship generation? It's not hard to see from the screenshots that the spaceships haven't improved much. I know it's not your favourite part of the game but please try and give it some love. I admit I've been spoiled by the work Star Citizen have done on the design of their ships but you don't have to look at really big budget games to realize that your designs are somewhat wanting. Screenshot one is a good example.

As I won't be seeing some of the neat (IMO) Freelancer features which have been omitted from LT can we at least have some awesome ship designs to go with your awesome space landscapes?

Edit: And yes, I did note that you acknowledged that your algorithm for ships is bad but that won't fix it.

You all know I'm a fan of realism right up until it starts cutting into fun, which, in my experience, happens quite early-on. I don't intend for ship thrusters to use real impulses, because I don't want players to have to spend hours trying to properly mass-balance their ships to get inertial characteristics that aren't impossible to fly with.

I spent a little time "enjoying" the mechanics used in the earlier builds of Star Citizen which entailed fighting to control your damn ship against all the thruster activity. If LT still gives me the sort of Freelancer control that the LTP version gave me I will be a very happy pilot.

I spent some time after-hours one night revisiting my most-prized algorithm: the nebula generator. I made it roughly 10x faster while also improving the visual fidelity AND making the generator more consistent, such that it yields less "really bad" systems. It was a fun night. We all know I'd die if I went more than half a year without touching 'nebula.glsl' But seriously, I'm happy about this, because nebula generation was one of the most expensive parts of system generation (by quite a lot). It's now much more reasonable, especially on less-powerful GPUs.

I'm always pleased to read of improvements in your "most-prized algorithm", Josh, but is there any possibility you could devote some time to your less prized algorithm for ship generation? It's not hard to see from the screenshots that the spaceships haven't improved much. I know it's not your favourite part of the game but please try and give it some love. I admit I've been spoiled by the work Star Citizen have done on the design of their ships but you don't have to look at really big budget games to realize that your designs are somewhat wanting. Screenshot one is a good example.

As I won't be seeing some of the neat (IMO) Freelancer features which have been omitted from LT can we at least have some awesome ship designs to go with your awesome space landscapes?

Edit: And yes, I did note that you acknowledged that your algorithm for ships is bad but that won't fix it.

this!!! been seeing the same ship blocky designs since the LTSL days. Cmon josh - give the ships some love so that your awe inspiring screenies wont have those blotches (ie current ship placeholders) that look like a fly after you've used a swatter on 'em - but some semblance of cool ship archetypes. i'm really curious to see what that inventive mind of yours will churn out on ship designs. even ships edition 0.1 (as opposed to 0.0 today) would be terrif