I'm pretty tech-ignorant, so I'm looking for some expert opinions. The Kickstarter for Camelot Unchained states that their goal is to "Maintain an absolute minimum of 30 FPS in battles of up to 500 people." That's why they're building a custom engine: no one else, apparently, has built an available engine that can do this. My question is, is this even feasible with modern technology?

It sure is, but all of your Cores will eventually MELT if you don't have an amazing Cooling setup. ...the reason for this, is that no other MMORPG's are built from the ground up to 100% max out all 8-16 cores but CityState's Q&A vloggers assure us that real multithreading is one of their top priorities.

It will probably still suffer Hitching issues related to SSD limitations and some projectile/collision lag between players in different Sectors (server-clusters) but it should no longer be the case that we're just sitting there watching an absolute slideshow while half of our CPU cores sit at 20% capacity

Yeah, we do have to be smart about our CPU/GPU usage but we haven't melted anything yet except grilled cheese in our toaster oven so that's a good sign!
You're right about the impact on multi-threading if we do our jobs right. Multi-threading is a giant PITA, which is why most engines aren't multi-threaded. There are languages out there that make multi-threading easier, but they aren't ready for us to use in this project. As multi-cores has become a must-have tech, I expect that will change over time.

Nevermind, I'll just wait until they show me they can do it. I'm a founder so I'll get to see it first, no point in arguing over it. If we're sticking to the facts, which is what MJ likes to do, then no they have not successfully tested CU with 500 players running the latest spells/combat abilities, VFX, new animations, and player character armors/weapons.

CU as promised by MJ is supposed to do much more than what we've seen in the alpha so far. And a lot of the things still missing are things that would put stress on the servers and the client computers.

Will they be able to do it? We'll see.

Well, we have done some of that (player characters with weapons and armor) but not with the VFX/new animations. But, as you know, I've been saying the same thing myself. Getting 1.5K Bots to run around a tiny space with almost all of them in your viewing frustrum is a major accomplishment because of the networking code to support that. On the rendering side, once we add VFX we expect to see an FPS drop (duh right?) but we also haven't begun to really optimize the engine yet nor give players the kind of options we expect in terms of graphic settings.
Have we accomplished something special with the code so far? Yes. 1.5K networked Bots is something relatively unique when you pair that number with a server-based physics system (which was active at the time) and a renderer that can handle that. On my machine (GTX970, standard PC, nothing special) with vsync off I was at 60 FPS with 644 Bots in front of me. With 1361 Bots, I went down to 30 FPS.
Now, again, does that mean we're there yet? No, 1361 times, no. But what we have accomplished is not too shabby, not too shabby at all.

Ill believe it when i see it , live in game , Noone has managed this yet without problems .. Very skeptical that they can

Don't blame you one bit. I've said the same thing to people. Skepticism is not a bad thing when it comes to new technology and/or Kickstarters. And since we are making both a KS-ed game and new tech, folks have twice the reason to be skeptical. That's one of the reasons you've seen so little in terms of hype/marketing/ads from us. As always, time will tell.

graphic quality might be a concern on that but generally speaking it should be zero problem, the only question is how many polygons to do you want on your models.

Polycount isn't main cause of the bottlenecking, it's just the leading contributor to all the other Bells and Whistles which make up "Eye Candy CREEP" in modern games. ...Variables like: Deformation, Particles, dynamic physics, attachments, particles attached to the attachments that attach to your model; and the reason for this is that all of the variable for various ways that Light can interact and be emitted and/or refracted off these Eye Candy effects has to be communicated through the CPU's lower level cache. This is because MMORPG designers are LAZY about things they don't understand and among all the things they deal with on a daily basis, this is one the the things they understand the least. ..they don't go about designing world aethertics & grouping encounters or anything else in a way that tackles these issues at their foundational levels. They just throw demands to make "pieces of flare" at the Art departments and cross their fingers that everything will run at 30 frames when they get 20-30 ppl on the same screen together. SCALING has never been one of their strong points and it never will until MMORPG'ers become a bit more discerning about choosing performance over flashy traillers and fancy looking horse armors

FYI, we're not that lazy. Slow? Maybe, but not lazy. These are the things and kind of discussions that we have here at the offices. And our VFX artist is highly technical so we are pretty good on that front. And fortunately for us, we're not focusing on eye candy on the way some other games are because that's not our target audience. But as always, time will tell.
...continued

CU has tested 1000 man battles so 500 is considered an easy target for them.

No, it is not an easy target. All the tests so far were backend focused, with barebone graphics and limited network testing capabilities.

They still have very long way to go....

Actually, you are wrong about the network testing capabilities and barebone graphics, though you are right that we aren't there yet. Our network testing stressed the network traffic the way that 1.5K players/Bots who were running around, colliding with each other (handled by the server, not client) would have stressed the game when it goes LIVE. FYI, the Bots stressed the system more than 1.5K players who were just running around but not fighting. OTOH, as per above, we need those Bots firing off abilities to really stress the system of course. And in terms of the barebone graphics, they were running on the Autumn Biome forest which was being rendered behind them. We don't cheat on tests.

CU has tested 1000 man battles so 500 is considered an easy target for them.

No, it is not an easy target. All the tests so far were backend focused, with barebone graphics and limited network testing capabilities.

They still have very long way to go....

I'm sure even they (Mj and CO) would agree there.

500 players is an easy target for us network-wise, 500 players all firing off abilities is our next big test. We had that before with our old ability system now it has to work with our new ability and VFX system.

CU has tested 1000 man battles so 500 is considered an easy target for them.

No, it is not an easy target. All the tests so far were backend focused, with barebone graphics and limited network testing capabilities.

They still have very long way to go....

You clearly haven't followed the game's development closely. They are already testing near release triangle and particle counts. You likely confuse textures with graphics that impact server side ... they do not. They impact the client pc.

On top of the current updates to the VFX that runs more efficiently than when they even ran those bot tests, the land mass the 500 player battle is on runs on it's own server, the physics system runs on it's own server and the vfx system is separate as well hence why the pet classes use it instead of standard models like mobs and characters. They don't impact the game's performance on the same network layers. So many of the things you likely believe impacts the 500+ player counts in fact do not. This is the point of the custom engine.

The net code is what is being tested already. It is why they took 2 years to make the engine. It is why they test with up to 1500 bots that run like clients logged into the game to emulate real player connections along with real players. This is not "backend".

Mark and the main engineers have no troubles believing they can run 500 player battles. Not one worry at all.

You are correct about the tri-count on player characters, armor, and weapons. The new VFX system is currently in testing right now, we had our first test of that with players yesterday and last week and more are coming next week.

----

All,

Thanks for your responses here, including the skeptical ones. FYI, I don't blame anyone for being skeptical. I said that during the KS and I'm still saying it. We are trying to do something nobody has had been able to pull off. So, as always, time will tell. Thanks!

There's something inherently awesome about the guy running the company regularly answering questions himself on a gaming forum. I really, really want this game to succeed.

Thanks, I really want this game to succeed as well. As a matter of fact, I have lots of reasons, millions of them. And thanks for the kind words, I do appreciate them.

I've almost always posted on forums whether back in the old times on BBSes or then on GEnie and other networks. The only times I really stopped posting entirely was for a while at EA, when I just need a break or if people are just being really rude/nasty/threatening. Most of the time, and on most places, I really enjoy posting, even when it comes to talking to folks who might be a tad cross with me. As long as it stays polite, even with criticism, I try to stick around. During our KS I posted here and a number of folks said that I'd be gone once the KS ended and never return, that I was only doing it to get more Backers. Well, I'm still here and not begging for bucks so I must actually enjoy this. In truth, I wish I had more time to post about other things but not happening, unfortunately.

I have to admit, after logging in over 4th of July I see the game has improved by quite a bit. Really impressed with the looks of new armors & weapons, animations are better, and general graphics look good with new shadow effects. I know I can't say much so I'll just say, I want to play this game. I wish we could see the end of the tunnel and get the game done, because I want to play it. But it's just not quite there yet.

I have to admit, after logging in over 4th of July I see the game has improved by quite a bit. Really impressed with the looks of new armors & weapons, animations are better, and general graphics look good with new shadow effects. I know I can't say much so I'll just say, I want to play this game. I wish we could see the end of the tunnel and get the game done, because I want to play it. But it's just not quite there yet.

Hopefully the hard parts are done or close to being done.

Thanks and you are correct. We are not there yet but if you look at how far we've come in terms of the graphical improvements, we are definitely a lot closer to the end of the tunnel than to its beginning.

In terms of gameplay, the ability system is starting to work the way we expected it would when we went into re-abilitation. Yeah, it has taken longer than we thought it would but fortunately we (Backers/CSE) can look at the game and see significant improvement every month. That pace will begin to get more visible as we continue to transition to more gameplay features rather than tech features over the next few months.

I know how hard it is to keep faith when we are late but the thing is, when I said we were building the core building blocks of our engine, I meant it. Now that they are coming online, the biggest challenges, tech-wise, are being checked off our list. That will make a heck of a difference over the rest of the development schedule. And unlike other MMOs, we will have the tech in place to support the battles that were part of our KS pitch. We could have gone the other route and it would have worked great in the short-term and screwed us big-time at the end. Fortunately, we went down the right path and as you saw, that approach is in the process of being validated.

Thanks again, especially for checking out the game again on the 4th. I can't wait till we can have a test with players, Bots and lots and lots of arrows/fireworks of all types. That should be amusing as heck.

After some tech hurdles with the new systems it seems they are back on track with heavy testing. Recent tests using full textures and VFX on more recent builds had successful 1500+ bots (emulating externally logged in player accounts) in close quarter full combat testing high rate casting fire spells. They even had over 2k bots at once point.

All of this at playable frame rates on a GTX 970.

Still lots of work yet to be done to meet their beta goals but looking good and progressing!

Actually, you are wrong about the network testing capabilities and barebone graphics, though you are right that we aren't there yet. Our network testing stressed the network traffic the way that 1.5K players/Bots who were running around, colliding with each other (handled by the server, not client) would have stressed the game when it goes LIVE. FYI, the Bots stressed the system more than 1.5K players who were just running around but not fighting. OTOH, as per above, we need those Bots firing off abilities to really stress the system of course. And in terms of the barebone graphics, they were running on the Autumn Biome forest which was being rendered behind them. We don't cheat on tests.

Mark

Am I?

Were the bots run from 1500 individual workstations outside of local network?

Yes, graphics is barebone, unless last game you have seen is 15 years ago :-p

Regardless, I still think all the graphics isn't in place yet and will take quite a lot of effort actually to balance performance/visual wise. I even think it is more difficult than all the back-end stuff.

Actually, you are wrong about the network testing capabilities and barebone graphics, though you are right that we aren't there yet. Our network testing stressed the network traffic the way that 1.5K players/Bots who were running around, colliding with each other (handled by the server, not client) would have stressed the game when it goes LIVE. FYI, the Bots stressed the system more than 1.5K players who were just running around but not fighting. OTOH, as per above, we need those Bots firing off abilities to really stress the system of course. And in terms of the barebone graphics, they were running on the Autumn Biome forest which was being rendered behind them. We don't cheat on tests.

Mark

Were the bots run from 1500 individual workstations outside of local network?

1) Bots are “headless clients” running on separate AWS instances. This means that on different instances (servers), we are running 1500 separate clients. They are headless, because in our case, they aren’t attached to real players at keyboards, monitors, etc.

2) Bots are almost indistinguishable to the server from players. They are started up from player characters we created just for that purpose. As such, they have full access to all the skills, abilities, etc. as players.

3) Because Bots = players, the server treats them the same way as players (except they don’t get to ask for refunds!) and as such, consume the same amount of network traffic as players. So when a Bot is firing off an ability, it has to send the same message to the servers that players do and in return, are sent the same amount of information from the server.

4) Bots, in general, consume a bit more network traffic than players since they don’t spend any time talked, go off for “botty-breaks”, etc., like normal players do. In some tests we actually had them chatting so they could test the chat system.

5) Bots are not “NPCs” because those things are usually controlled from the same server(s) instance(s) and are generally not networked like players. We will, of course, have NPCs too but no NPCs were harmed in the making of these screenshots!

So, in terms of whether are Bots are the same as players in terms of Internet/network traffic and also client stability (they can crash too), etc., the answer is 100% yes! [...]

So, in terms of whether are Bots are the same as players in terms of Internet/network traffic and also client stability (they can crash too), etc., the answer is 100% yes! [...]

And that was my point - it is not.

Bots are great testing tool and they can test servers very well but when it comes to network, it is very raw testing.

Sitting on local network means you test with best possible network conditions you can get and do not need to deal with things like latency when your client is thousands of kilometers away - thus limited network testing capabilities.

I am not trying to say it is a game changing difference or just be negative for the sake of being negative, just pointing out the journey from technology tests to polished game with real clients is very, very long. Imo, much longer than people realize.

Bots are great testing tool and they can test servers very well but when it comes to network, it is very raw testing.

Bots are a piece of software, hence the devs can make them as 'demanding' as they want and need, i.e. the bots can be programmed with an artificial delay (for testing ping) or more bandwidth-consuming than the average user client (which they currently are!).

So, in terms of whether are Bots are the same as players in terms of Internet/network traffic and also client stability (they can crash too), etc., the answer is 100% yes! [...]

And that was my point - it is not.

Bots are great testing tool and they can test servers very well but when it comes to network, it is very raw testing.

Sitting on local network means you test with best possible network conditions you can get and do not need to deal with things like latency when your client is thousands of kilometers away - thus limited network testing capabilities.

I am not trying to say it is a game changing difference or just be negative for the sake of being negative, just pointing out the journey from technology tests to polished game with real clients is very, very long. Imo, much longer than people realize.

That's not how spinning up AWS VMs works. You can spin up a VM from any region in the world so you can replicate distribution in your network. We do something like this at work on a much smaller scale to test our web services and prototype demonstrations.

An AWS VM is always outside your network, even when you create a VM cluster, because the VMs themselves are only region specific. You can tie them together with a virtual network but that's not the same thing as creating an "on local hardware" network.

There are some real world stress conditions you can't test with a headless client. If it's truly headless in the sense that the graphical client isn't running then that system load and performance isn't being widely tested.

Just trying to live long enough to play a new, released MMORPG, playing POE at the moment.

Fools find no pleasure in understanding, but delight in airing their own opinions. Pvbs 18:2, NIV

Don't just play games, inhabit virtual worlds™

"This is the most intelligent, well qualified and articulate response to a post I have ever seen on these forums. It's a shame most people here won't have the attention span to read past the second line." - Anon

Even though I'm not one to financially back these projects I really hope Mark and the team pull through. So many people looking for a new home... if one passionate team can keep their focus on the game and the players.. just one. Make the game great and you will make all the money you need. Focus on making money and we all know where that leads.

Even though I'm not one to financially back these projects I really hope Mark and the team pull through. So many people looking for a new home... if one passionate team can keep their focus on the game and the players.. just one. Make the game great and you will make all the money you need. Focus on making money and we all know where that leads.

"Make the game great" is sadly both extremely hard to do and sometimes just not enough.

I hope they pull it off as well, but the longer it takes the more interest players lose.

So, in terms of whether are Bots are the same as players in terms of Internet/network traffic and also client stability (they can crash too), etc., the answer is 100% yes! [...]

And that was my point - it is not.

Bots are great testing tool and they can test servers very well but when it comes to network, it is very raw testing.

Sitting on local network means you test with best possible network conditions you can get and do not need to deal with things like latency when your client is thousands of kilometers away - thus limited network testing capabilities.

I am not trying to say it is a game changing difference or just be negative for the sake of being negative, just pointing out the journey from technology tests to polished game with real clients is very, very long. Imo, much longer than people realize.

Because you continue to post without doing a shred of research yourself (and 2 people rated your post as insightful lol), this is a snip from Mark posting about what bots are and how they are used here:

"1) Bots are “headless clients” running on separate AWS instances. This means that on different instances (servers), we are running 1500 separate clients. They are headless, because in our case, they aren’t attached to real players at keyboards, monitors, etc.

2) Bots are almost indistinguishable to the server from players. They are started up from player characters we created just for that purpose. As such, they have full access to all the skills, abilities, etc. as players.

3) Because Bots = players, the server treats them the same way as players (except they don’t get to ask for refunds!) and as such, consume the same amount of network traffic as players. So when a Bot is firing off an ability, it has to send the same message to the servers that players do and in return, are sent the same amount of information from the server.

4) Bots, in general, consume a bit more network traffic than players since they don’t spend any time talked, go off for “botty-breaks”, etc., like normal players do. In some tests we actually had them chatting so they could test the chat system.

5) Bots are not “NPCs” because those things are usually controlled from the same server(s) instance(s) and are generally not networked like players. We will, of course, have NPCs too but no NPCs were harmed in the making of these screenshots!

So, in terms of whether are Bots are the same as players in terms of Internet/network traffic and also client stability (they can crash too), etc., the answer is 100% yes! Otherwise, if they weren’t, we’d be, to put it mildly, lying and even though you and I disagree about some things, I hope by now you and our Backers know that I refuse to lie, I just rather deliver bad news. FYI, I wanted to have these kind of Bots for testing in WAR. Unfortunately, that was vetoed. Fortunately, this time my partner/co-founder Andrew couldn’t have been more in favor of it and was a strong proponent himself. And thanks to the other engineers at CSE and the considerable time spent getting them up and running, they are really incredibly useful for these types of tests, then, now and forever.

FYI, interesting historical note, the concept of using Bots for this kind of testing goes way, way back, even to the origin of the MUDs. One of the earliest members of Mythic Entertainment, and a fine gentleman who worked for me at AUSI (my first company, maker of the world’s third MUD, Aradath), Darrin Hyrup, had a similar system to stress test our MUDs. Very helpful then, incredibly helpful now. For example, In Beta, we’ll be using them every few weeks to beat on the current build to make sure that nothing bad has crept into the code networking-wise. No matter how good the programmers/designers/artists, occasionally things go into the game that looks fine but when systems are pressured, bad things can arise. The Bots aren’t a cure-all, but used properly they can cut down on the number of times things seem to work great when 40 people are playing during testing but go to complete crap when 4000 people are playing the same build when it goes LIVE."

I do not believe this will help you however. No matter what others say, you do not believe them even when they directly answer your questions. You continue to argue without knowing the fundamentals of the subject matter.

Just trying to live long enough to play a new, released MMORPG, playing POE at the moment.

Fools find no pleasure in understanding, but delight in airing their own opinions. Pvbs 18:2, NIV

Don't just play games, inhabit virtual worlds™

"This is the most intelligent, well qualified and articulate response to a post I have ever seen on these forums. It's a shame most people here won't have the attention span to read past the second line." - Anon