Kitely attacks slow scripts

There are many reasons why a region might be slow to load, but one of the hardest to address has been that of slow scripts. OpenSim doesn’t make it easy to discover which particular scripts are contributing the most to lag — but that is about to change.

Kitely has fixed a long-standing problem in OpenSim where there was no useful script performance information in either the Statistics Bar or the “Debug” tab in the “Region/Estates” dialog window when users pressed the “Get Top Scripts” button.

Oren Hurvitz

“Until today, the Script Time shown in the Statistics panel was always 0,” said Oren Hurvitz, Kitely’s co-founder and VP of R&D, in an announcement earlier this week. “That statistic had never been implemented in OpenSim.”

That data is now available to Kitely users, and region owners should shoot for a value of under 1 ms, he recommended.

Top scripts dialog window. (Image courtesy Kitely.)

“Some worlds might legitimately need more Script Time, if they have many scripts or many animations, but usually if the Script Time is higher than a few ms then you should try to optimize some of the slowest scripts in your world,” he wrote.

The “Get Top Scripts” button now works as well, and shows how much time has been spent on each script in the previous 30 seconds.

The slowest scripts can then be fixed, deleted, or temporarily shut down if, say, there’s a big event going on at the region.

Once this feature has been tested, it will be donated to the OpenSim community, he said.

Ilan Tochner

“We felt this feature was too important to keep private,” Kitely CEO Ilan Tochner told Hypergrid Business. “We’ve therefore decided to help people on all OpenSim grids combat this problem by contributing our latest new feature to the OpenSim project.”

Tochner added that this particular feature is important because of its potential to improve the entire hypergrid.

Maria Korolov is editor and publisher of Hypergrid Business. She has been a journalist for more than twenty years and has worked for the Chicago Tribune, Reuters, and Computerworld and has reported from over a dozen countries, including Russia and China.

Alex Ferraris

Sorry but we have that in AviWorlds. 🙂 Always saw the scripts there and not zero. Maybe i am not understanding.

Please read our blog post. There are two places where script-related statistics should be displayed. One is the Statistics windows (CTRL-Alt-1) and the other is the Top Scripts window. OpenSim 0.8.2 doesn’t include an implementation for the former (our patch will add that). While the later (Top Scripts window) statistics are present in OpenSim 0.8.2 but have various implementation problems which greatly reduce their usefulness for finding script load issues. It isn’t enough to show some numbers it’s important that the numbers that are shown will be accurate and actionable.

The article quotes “Until today, the Script Time shown in the Statistics panel was always 0” – that is the Statistics window you open with CTRL-Alt-1 which indeed doesn’t include script runtime in OpenSim 0.8.2 (try it now on your own grid).

One of the problems with the Top Scripts implementation in OpenSim 0.8.2 is that it didn’t include indicative data about all the scripts that require the most amount of time to run. For example, a script that takes a long time to run (and creates lag) but is called only once per 15 seconds may not get included in the top scripts list under certain conditions. Alternatively, a script that takes very little time to run but is called thousands of times per second (and it too creates lag) may not be included at all because each call by itself is insignificant compared to the time it takes to handle the other scripts being called in the sim. Our patch addresses these scenarios and more.

Bryan French

We are talking about 0.8.2 aren’t we? I have found the Top Scripts
window to be very accurate, but we aren’t running 0.8.2 either. In my
Statistics window it does say Scripts Run 0.000%, but it does say on the statistics there are
68 scripts running and all 68 show up on the Top Scripts even the ones running 0. When it comes to scripts on our grid we will only allow scripts we have tested and that run under 1.0ms. That is why you will not find a blink script on our grid. They run continuously and over 1.0ms

If you look at the OpenSim code that implements this panel you’ll see that what it currently calculates for Top Scripts is far from accurate. The fact that many people currently trust what they see is one of the main reasons we decided to contribute this patch.

TribeGadgets

Nice add but is your welcome centre ‘which is a highly optimized world’ really running this Total Frame Time? 88ms+ (quote and snip from your blog)
I would worry if my ancient box running a small 0.8x simona was that bad.

OpenSim 0.8.2 includes patches from the MOSES project which significantly improve the statistics you see in your viewer. The frame time you see reflects the actual Sim FPS that OpenSim runs at. The numbers you see if you are running OpenSim 0.8.1.1 or earlier are multiplied by a constant that the developer that first implemented stats added in order for them to look like the numbers people were used to in Second Life.

In other words, the stats in our screenshot represent the maximum Sim FPS you will see once you switch to OpenSim 0.8.2 (unless OpenSim core decides to increase it). In any case, the base Sim FPS hasn’t changed it is just that now the Statistics panel shows the accurate numbers your sim is running at.

> OpenSim 0.8.2 includes patches from the MOSES project which significantly improve the statistics

Can you point me to where these are because to my knowledge they still only exist as patches attached to Mantis records and have not been committed to the code base. Of course Kitley may have deployed these, but it is not generally available.

Correct, they sent 3 patches and the first one being committed slipped under my radar. It was committed at the end of April under “Added in all metrics for Phase 1 except for Time Dilation.”
I think I tired paying attention to it after a good deal of fuzz on the dev mailing list.

Not sure where phase 2 is, and phase 3, which is the network stats, is still under review after they submitted the final patch on August 5.

It’s a function of the FPS you wish tho have. At 11 FPS you get around 90.9ms per frame. If the FPS is a bit higher (around 11.2-11.3 FPS) then you get the numbers you see in the screenshot, i.e about 88.9ms per frame. If you read historic conversations in the OpenSim dev mailing list you’ll see that the OpenSim target FPS was arbitrarily decided years ago. I’m quite confident it could be increased, but while doing so would increase system load, I’m not sure it would contribute much to actual user experience (people seem to be happy with OpenSim’s performance as it is).

I agree with you that performance in general is where people are happy about it (at least when the dust have settled and they are rezzed in to a scene.)

I was just a bit puzzled that the new Frame Time FPS produced a number almost identical to the Physics FPS. I was under the impression that (as you say) the desired Physics FPS is set at a 11 target, but that the scene FPS was independent of the Physics FPS.

Thanks for the refresh. I remember having read it before, I just had not entirely realized that Physics PFS = Sim FPS with a target heartbeat at 11 for OpenSim.

From observing the sims a bit more I see they fluctuate between 89 and 98 ms where at the high end they probably stall a bit on mesh loading and decoding. (meshes are always physics unless set to phantom).

I run them on mono 4.0.2.5 and one of the things about mono 4.x is that it is pretty aggressive on purging memory so perhaps I should look into that. Simulators that typically ran around 2GB with mono 3.10 now run with process memory of less than 512 Mb.

I guess the reason why LL has let the physics component for meshes weigh as much as it does in the LI calculation is because they have exactly the same relation between physics simulation and Sim FPS. They just don’t break it out in their stats much.

Actually on the marketplace there should be tick marks for if an item is 100% mesh, partial mesh and if it was uploaded with a separate physics model or not. The difference on the sim can be quite substantial just for a single mesh if it has an optimized physics model over the “default” generated out of the mesh geometry at upload.