Archive for OpenSim

All current outstanding requests for InWorldz OAR exports are complete.

In August of 2018, when the news came that InWorldz would shut down, I made a personal promise to the residents: I would do everything I could to ensure that any users who had any content rezzed in any region at shutdown time would be able to request, and be provided, a filtered OAR export archive of their content.

Now in July of 2019, after 11 months of near-continuous part-time efforts, I have completed the lengthy list of outstanding requests for OAR exports. Because the content was being taken off-grid, in order to respect the promises made to creators when they joined InWorldz, we needed to provide filtered OARs — only the content owned and created by a given user, or a “whitelist” of alts and collaborators providing their consent for export. This first required that a lot of specialized code be developed for the OAR loading process, and enhancements made to support not only the filtering aspects, but “drilling into” the Contents of rezzed objects, and the Contents of any objects nested within those objects.

It wasn’t just a matter of filtering objects with a subset of those in the region; when you’re looking 5 levels deep into the Contents of an object and 2 out of 5 of the Contents items are permitted for export, it means a new object must be created, and the other 3 Contents items must be replaced with placeholders — and then that new object must be substituted for the one referenced by (inserted back into) the object, which in many cases will trigger that object in the Contents list to be substituted as well, with another new object that has one or more Contents items replaced… all the way back up the parentage of objects to the one rezzed in the region. Also, most asset references needed to be checked and potentially scrubbed according to the whitelist of users. The code that scanned objects for asset IDs had no support for the newer InWorldz “Thoosa” asset storage, just the old OpenSim asset formats, and almost none of the code had support for Thoosa assets.

Just the coding itself was a tremendous amount of work, and if I had been aware of the effort required there is no way in hell I would have attempted it. I would have considered it far beyond that promise of “everything I could do” (reasonably). But I didn’t know that, dug in, and eventually completed the work anyway.

Then began the work of producing the actual filtered OAR export files for users.

Actually producing the OAR files was the easiest part. Trying to get information from users, such as a definitive lists of avatars to include, with authorizations from the email addresses that was used to register the accounts, and a list of specific region names, well that required a lot of user interaction. And interacting people slow down work tremendously. To date, it has required just under 900 email messages, often detailed and lengthy.

There was also the matter of user privacy. If “Joe User” collaborated with “Sally Avatar”, I couldn’t just provide the email addresses used to register those accounts to each of them and ask them to work it out. In many cases former project collaborators had no way to communicate, and I’d need to do that communication myself on behalf of the user requesting the OAR export, in order to protect those email addresses entrusted to InWorldz.

Even worse, I’d occasionally get an email with a confirmation of export authorization from what appeared to be the correct user, but not from the email address we had on file for that avatar account. So I could not accept such consent, and I could inform the user that it was the wrong email address, but for privacy reasons I could not provide the correct email even to that user.

Near the end I began sending a request for confirmation to the officially-registered email address in the hopes that it would reach the user and they could simply reply. But for those who moved on with email accounts, it was a battle between getting it done and privacy, and privacy was always paramount, especially since email addresses often revealed real-life identities behind the avatars.

If you would still like to receive an OAR export file:

I know some former InWorldz users are learning of all this process very late, so I’m still willing to provide OAR files for those who have missed out so far.

If you are still interested in a filtered OAR export of a former InWorldz region, please send a request to jim AT gridmail DOT org. The request should include:

the avatar name(s) of your account(s),

the avatar names of any creative collaborators who will confirm via email that I have their consent to include their content in an OAR export to you, and

the region name(s) of any regions where you had content rezzed when InWorldz was shut down in July.

Setting your OAR file expectations:

Please be aware that if you were not much of a creator of content, your OAR file will be heavily filtered. This means a lot of plywood boxes for things you purchased from other creators for use in InWorldz. That said, if there are third-party objects where you have your own content, for example a texture-changing photo frame where you added your own snapshots, the photo frame may be replaced by a box, but your snapshots would survive in the Contents of that box.

Also, an OAR export file is not really meant for end users. Second life has no knowledge of them, nor do any viewers (including Firestorm). OAR files are intended to be provided to grid owners for loading into an existing (empty) region on that grid. The only real exception to that, which isn’t really an exception, is if you are running your own Halcyon or OpenSim server (including Sim-on-a-Stick), you can do it yourself with the “load oar filename.oar” command. But there are many subtleties, including content Owner and Creator substitution that are best left to a knowledgeable grid owner. Also note: as far as I am aware, only TagGrid has a specialized process for attempting to preserve Creator info, other than the OAR owner, when loading an OAR file.

Back in July of last summer, there was a fast frenzy to save the contents of every running region as a new OAR file containing its own assets. There wasn’t time, or server code, to properly support filtering. But there was a limited amount of time, a few days, where there was an opportunity to save the contents of rezzed regions.

When it comes to exporting the content from InWorldz for former residents, I’ve been thinking of three milestones:

OAR-related server enhancements (COMPLETE): Complete the software changes to allow deep filtering of content based on the Creator and Owner of each prim, each Contents item, and that of any nested objects. This is the “whitelist”, a list of avatar accounts to include. The changes applied went much farther than I could have predicted, and as such it is much more advanced than other OAR filtering code, allowing everything that could possibly be saved to be included, even if the parent prim or embedded Contents item was not created by a whitelisted user. This work was completed months ago, although small improvements continued to be made each week.

Region OAR exports(COMPLETE) : Complete the processing of the list of requests for specific regions, for specific users. I am very (very) pleased to report that, with a couple of special exceptions, we reached this second milestone last night. A total of 166 OARs have been produced, so far, with about 15 more outstanding where I await the user. (The remaining cases are regions where the user requests more objects be included and we await confirmation of authorization to include that content from the creators in question.)

Sandboxes/other (BEGINNING NOW): All the other miscellaneous region requests where there are many many requests for smaller quantities of objects from the same region(s). This generally (and almost totally) refers to InWorldz Sandbox 1-6 regions, although we have some requests for owned content from the InWorldz Events regions and other sponsored ones.

As of last night, the first two are complete. If you have a request for a specific region (other than the Sandbox, and sponsored regions of the third point above), please let me know (by sending an email to ‘jim’ AT ‘gridmail.org’) because it means I am not aware of it.

This work (both of those first two points) were major efforts representing months of work in my spare time. But it was important to me to not just leave everyone without anything; as I said before, the house was on fire but we were able to grab a few things as we ran out, and it was time to give those region contents back to the owners of those wonderful creations. And I could not just leave Elenia on her own to try to get all of this done. We lost a lot. I hope we also managed to save a lot.

Milestone 3 remains. That is, processing of any requests for Sandbox regions and other sponsored regions with shared content. I’ll post another update when it’s complete.

I will begin work very soon on providing an OAR file for each Sandbox region where a user has contents, but only for everyone who requests one. There are far too many of them to try to send them out unsolicited, not knowing who to include in that specific users whitelist, or if their registered email address is even still valid.

I will automatically include any alts registered under the same email address.

I do not have very many requests for content from Sandboxes. Unless there is a sudden burst of requests, it should only take a few days to complete the Sandbox work and reach Milestone 3.

Going forward: Once the Sandboxes and sponsored regions are processed, I will continue to take requests, either for OAR exports from the Sandbox regions, or any other region, and process those on-demand. Should be about a one-week turnaround on getting an OAR export, with the backlog cleared.

OAR file deletion: I’m not sure how long I will keep the OAR files available (it costs me money each month), but I promised everyone I’d keep them available for at least a month, so what I can say is that I will ensure they are available at least until July 1. After that, they could disappear at any time. So download them before then if you may want to access them beyond the end of June.

Sandbox regions: Below is the list of the owners of every. single. prim on all of the Sandbox region OAR backups taken that week in July. (Note: Sandbox 3 seems problematic, there is only one owner, in one OAR backup.)

If you see “(REQUESTED)” after a name, it’s a request that I’m aware of for content from the Sandbox regions. If your name is listed and it doesn’t have that mark after it, and you want an OAR made of the content on the regions (even if it’s only one object), send me the email to the address above request it. And indicate if there is permission obtained or coming from other avatars to include. As I said, I’ll automatically include your alts on the same email address.

The two numbers at the beginning are the number of linked objects, and the total number of prims (LI) for those objects:

A quick update on the “export OARs”. These are the OAR files I have been accepting requests for, intended to be personal OAR exports for use on something like Sim-on-a-stick or to be taken to another grid. (Note: you don’t need to be a former region owner or land owner; if you had content in an InWorldz region somewhere at grid shutdown time in July then we can try to recover it. See my earlier blog post for more.)

I have about 20 of 150 or so known requests done but I might need to redo them. it’s all been on hold a couple of days while I investigated a problem with assets not being recognized on OpenSim. While there is an OpenSim problem in how OpenSim just dumps bad data (see my follow-up blog post for more), I’ve also tracked down what the data is and where it was coming from in Halcyon OAR exports.

The root of the trouble is that when Tranq implemented a new binary format for object storage (project Thoosa) in InWorldz, he did not complete that work everywhere it needed to be. There are three problems:

Problem 1: the code that scans a region for the IDs of assets to save in an OAR file did not support his new format, so it did not include those assets in an OAR backup. This will result in missing assets when the OAR is reloaded outside InWorldz. The problem wasn’t detected in InWorldz since the assets were already there and did not need to come from the OARs. Region backups and rollbacks did not save the assets since they were already available grid-wide. This is a problem for all of our OAR exports, but not all objects (more infor below).

Problem 2: The second area that didn’t get updated for Thoosa was the OAR storage code to store nested objects in Contents of other rezzed objects. The assets for those nested objects, if modified after Thoosa was introduced a few years ago, were stored in the OARs in the newer Thoosa binary format, not non-binary XML format needed by OpenSim. (This part is understandable since OpenSim couldn’t read Halcyon OARs until recently.) This means some objects in Contents were only readable by Halcyon servers, and for those objects, any assets referenced by them were not stored at all, due to the first problem above. (Non-object Contents items are not affected.)

Problem 3: OpenSim seems to have a problem where if it cannot find an asset referenced by an object, it doesn’t rez the object at all. It tells you there’s a problem and then refuses to rez the object. This means you cannot just rez a partial object and clean it up in an OpenSim region.

I’ve fixed the first two problems, which should fix future OAR exports and also take care of the “invalid XML” messages on OpenSim and the region hangs problem on OpenSim that can result from the dumping of invalid data to the console. However, this will not restore any assets that weren’t saved in July due to problem 1 above.

However, the process for OAR exports is for me to reload each unfiltered OAR with a whitelist of avatar accounts to include (often just one account, sometimes partners or teams), then save a new “export” OAR for use in other grids. The fact that these are reloaded under the latest Halcyon code and then resaved as a new OAR means that what does get saved should be usable in OpenSim after these changes. It will fix the Thoosa-specific nested objects to be exported in XML format, and provide at least the nested object itself.

I’ve also tracked down a couple of other issues with XML object formats not being compliant with OpenSim’s needs, which is a bit quirky, and worked around those in the Halcyon code to make them more compliant with OpenSim.

I’m still investigating the effects of these issues, now that I have a clearer understanding, to determine if I need to redo the OARs I’ve already completed or not, and if the code is now complete enough after these new changes that I can proceed again with the remaining OARs.

I should be able to proceed again in another day or so.

Update: That work was completed a couple of days ago, all looks good now. I’ve redone a couple of the region OARs already exported, but I will be redoing ALL of the regions already exported (about 20) over the next few days. Then I’ll be resuming my work through the list I have so far. If you haven’t emailed a request for your content yet, please be sure to email me according to my earlier post.

The coding is complete and we’ve produced a small number of OAR files for export from InWorldz. In some cases the intended destination is a Sim-on-a-stick personally installation or a grid running OpenSim. While the new OAR files have been updated to be recognized and supported by OpenSim, there is some data within the OAR file that will only be useful to Halcyon regions. These are assets which may not be in an XML format.

Problem: Back in July of 2015, Kitely’s technical lead (Oren) submitted some changes to OpenSim to actually dump any non-XML assets to the region console. This means that if you attempt to load a Halcyon OAR file under OpenSim, the console will stream thousands of lines of jibberish data. Because the data is known to not be valid XML in this case, it may include content that could cause the region to freeze or hang, but in the best case scenario it will simply spew thousands of lines of error messages for much of the oar load. These errors are false-positives (and will hide any legitimate errors).

Solution: You should ignore these errors and let it finish.

That’s all, just wait for it to finish. Then log in and visually verify that it has seemed to successfully load the OAR contents.

For those who build their own OpenSim executables:

For those who build their own, I suggest the following changes to mute the very noisy error handling:In SceneObjectSerializer.cs, change the first catch (in FromOriginalXmlFormat) to:

Both of these change the message to a warning, don’t bother reporting a stack traceback (since this can only happen from one place) and most importantly, completely eliminate the dumping of the invalid data to the console. This will still produce a noisy (3-lines per case) log, but OpenSim seems very verbose in logging, and this is considerably quieter than it is otherwise and won’t make you think something is horribly wrong when all is working according to plan.

I took a quick look at submitting this as a patch to OpenSim but first I must make it clear that I am not going to make it my mission to fix OpenSim issues. I have submitted fixes in the past directly to Justin, and I considered submitting this one using their patch process, but realized that there are a variety of development philosophies and I don’t know OpenSim’s. e.g. whether OpenSim has OAR debugging levels that the XML dump could be conditional upon, or whether I should remove that commented out line (or leave it for developers debugging issues), etc. Bottom line, I don’t know what to submit as a patch. I do know how to change the code to avoid this issue so I have posted that above. Since the key fix is a simple commenting out of a line, I’ll leave it to others to decide how much or how little they want to include from these trivial code changes.