Control Panel

All current outstanding requests for InWorldz OAR exports are complete.

In August of 2018, when the news came that InWorldz would shut down, I made a personal promise to the residents: I would do everything I could to ensure that any users who had any content rezzed in any region at shutdown time would be able to request, and be provided, a filtered OAR export archive of their content.

Now in July of 2019, after 11 months of near-continuous part-time efforts, I have completed the lengthy list of outstanding requests for OAR exports. Because the content was being taken off-grid, in order to respect the promises made to creators when they joined InWorldz, we needed to provide filtered OARs — only the content owned and created by a given user, or a “whitelist” of alts and collaborators providing their consent for export. This first required that a lot of specialized code be developed for the OAR loading process, and enhancements made to support not only the filtering aspects, but “drilling into” the Contents of rezzed objects, and the Contents of any objects nested within those objects.

It wasn’t just a matter of filtering objects with a subset of those in the region; when you’re looking 5 levels deep into the Contents of an object and 2 out of 5 of the Contents items are permitted for export, it means a new object must be created, and the other 3 Contents items must be replaced with placeholders — and then that new object must be substituted for the one referenced by (inserted back into) the object, which in many cases will trigger that object in the Contents list to be substituted as well, with another new object that has one or more Contents items replaced… all the way back up the parentage of objects to the one rezzed in the region. Also, most asset references needed to be checked and potentially scrubbed according to the whitelist of users. The code that scanned objects for asset IDs had no support for the newer InWorldz “Thoosa” asset storage, just the old OpenSim asset formats, and almost none of the code had support for Thoosa assets.

Just the coding itself was a tremendous amount of work, and if I had been aware of the effort required there is no way in hell I would have attempted it. I would have considered it far beyond that promise of “everything I could do” (reasonably). But I didn’t know that, dug in, and eventually completed the work anyway.

Then began the work of producing the actual filtered OAR export files for users.

Actually producing the OAR files was the easiest part. Trying to get information from users, such as a definitive lists of avatars to include, with authorizations from the email addresses that was used to register the accounts, and a list of specific region names, well that required a lot of user interaction. And interacting people slow down work tremendously. To date, it has required just under 900 email messages, often detailed and lengthy.

There was also the matter of user privacy. If “Joe User” collaborated with “Sally Avatar”, I couldn’t just provide the email addresses used to register those accounts to each of them and ask them to work it out. In many cases former project collaborators had no way to communicate, and I’d need to do that communication myself on behalf of the user requesting the OAR export, in order to protect those email addresses entrusted to InWorldz.

Even worse, I’d occasionally get an email with a confirmation of export authorization from what appeared to be the correct user, but not from the email address we had on file for that avatar account. So I could not accept such consent, and I could inform the user that it was the wrong email address, but for privacy reasons I could not provide the correct email even to that user.

Near the end I began sending a request for confirmation to the officially-registered email address in the hopes that it would reach the user and they could simply reply. But for those who moved on with email accounts, it was a battle between getting it done and privacy, and privacy was always paramount, especially since email addresses often revealed real-life identities behind the avatars.

If you would still like to receive an OAR export file:

I know some former InWorldz users are learning of all this process very late, so I’m still willing to provide OAR files for those who have missed out so far.

If you are still interested in a filtered OAR export of a former InWorldz region, please send a request to jim AT gridmail DOT org. The request should include:

the avatar name(s) of your account(s),

the avatar names of any creative collaborators who will confirm via email that I have their consent to include their content in an OAR export to you, and

the region name(s) of any regions where you had content rezzed when InWorldz was shut down in July.

Setting your OAR file expectations:

Please be aware that if you were not much of a creator of content, your OAR file will be heavily filtered. This means a lot of plywood boxes for things you purchased from other creators for use in InWorldz. That said, if there are third-party objects where you have your own content, for example a texture-changing photo frame where you added your own snapshots, the photo frame may be replaced by a box, but your snapshots would survive in the Contents of that box.

Also, an OAR export file is not really meant for end users. Second life has no knowledge of them, nor do any viewers (including Firestorm). OAR files are intended to be provided to grid owners for loading into an existing (empty) region on that grid. The only real exception to that, which isn’t really an exception, is if you are running your own Halcyon or OpenSim server (including Sim-on-a-Stick), you can do it yourself with the “load oar filename.oar” command. But there are many subtleties, including content Owner and Creator substitution that are best left to a knowledgeable grid owner. Also note: as far as I am aware, only TagGrid has a specialized process for attempting to preserve Creator info, other than the OAR owner, when loading an OAR file.

Back in July of last summer, there was a fast frenzy to save the contents of every running region as a new OAR file containing its own assets. There wasn’t time, or server code, to properly support filtering. But there was a limited amount of time, a few days, where there was an opportunity to save the contents of rezzed regions.

When it comes to exporting the content from InWorldz for former residents, I’ve been thinking of three milestones:

OAR-related server enhancements (COMPLETE): Complete the software changes to allow deep filtering of content based on the Creator and Owner of each prim, each Contents item, and that of any nested objects. This is the “whitelist”, a list of avatar accounts to include. The changes applied went much farther than I could have predicted, and as such it is much more advanced than other OAR filtering code, allowing everything that could possibly be saved to be included, even if the parent prim or embedded Contents item was not created by a whitelisted user. This work was completed months ago, although small improvements continued to be made each week.

Region OAR exports(COMPLETE) : Complete the processing of the list of requests for specific regions, for specific users. I am very (very) pleased to report that, with a couple of special exceptions, we reached this second milestone last night. A total of 166 OARs have been produced, so far, with about 15 more outstanding where I await the user. (The remaining cases are regions where the user requests more objects be included and we await confirmation of authorization to include that content from the creators in question.)

Sandboxes/other (BEGINNING NOW): All the other miscellaneous region requests where there are many many requests for smaller quantities of objects from the same region(s). This generally (and almost totally) refers to InWorldz Sandbox 1-6 regions, although we have some requests for owned content from the InWorldz Events regions and other sponsored ones.

As of last night, the first two are complete. If you have a request for a specific region (other than the Sandbox, and sponsored regions of the third point above), please let me know (by sending an email to ‘jim’ AT ‘gridmail.org’) because it means I am not aware of it.

This work (both of those first two points) were major efforts representing months of work in my spare time. But it was important to me to not just leave everyone without anything; as I said before, the house was on fire but we were able to grab a few things as we ran out, and it was time to give those region contents back to the owners of those wonderful creations. And I could not just leave Elenia on her own to try to get all of this done. We lost a lot. I hope we also managed to save a lot.

Milestone 3 remains. That is, processing of any requests for Sandbox regions and other sponsored regions with shared content. I’ll post another update when it’s complete.

I will begin work very soon on providing an OAR file for each Sandbox region where a user has contents, but only for everyone who requests one. There are far too many of them to try to send them out unsolicited, not knowing who to include in that specific users whitelist, or if their registered email address is even still valid.

I will automatically include any alts registered under the same email address.

I do not have very many requests for content from Sandboxes. Unless there is a sudden burst of requests, it should only take a few days to complete the Sandbox work and reach Milestone 3.

Going forward: Once the Sandboxes and sponsored regions are processed, I will continue to take requests, either for OAR exports from the Sandbox regions, or any other region, and process those on-demand. Should be about a one-week turnaround on getting an OAR export, with the backlog cleared.

OAR file deletion: I’m not sure how long I will keep the OAR files available (it costs me money each month), but I promised everyone I’d keep them available for at least a month, so what I can say is that I will ensure they are available at least until July 1. After that, they could disappear at any time. So download them before then if you may want to access them beyond the end of June.

Sandbox regions: Below is the list of the owners of every. single. prim on all of the Sandbox region OAR backups taken that week in July. (Note: Sandbox 3 seems problematic, there is only one owner, in one OAR backup.)

If you see “(REQUESTED)” after a name, it’s a request that I’m aware of for content from the Sandbox regions. If your name is listed and it doesn’t have that mark after it, and you want an OAR made of the content on the regions (even if it’s only one object), send me the email to the address above request it. And indicate if there is permission obtained or coming from other avatars to include. As I said, I’ll automatically include your alts on the same email address.

The two numbers at the beginning are the number of linked objects, and the total number of prims (LI) for those objects:

A quick update on the “export OARs”. These are the OAR files I have been accepting requests for, intended to be personal OAR exports for use on something like Sim-on-a-stick or to be taken to another grid. (Note: you don’t need to be a former region owner or land owner; if you had content in an InWorldz region somewhere at grid shutdown time in July then we can try to recover it. See my earlier blog post for more.)

I have about 20 of 150 or so known requests done but I might need to redo them. it’s all been on hold a couple of days while I investigated a problem with assets not being recognized on OpenSim. While there is an OpenSim problem in how OpenSim just dumps bad data (see my follow-up blog post for more), I’ve also tracked down what the data is and where it was coming from in Halcyon OAR exports.

The root of the trouble is that when Tranq implemented a new binary format for object storage (project Thoosa) in InWorldz, he did not complete that work everywhere it needed to be. There are three problems:

Problem 1: the code that scans a region for the IDs of assets to save in an OAR file did not support his new format, so it did not include those assets in an OAR backup. This will result in missing assets when the OAR is reloaded outside InWorldz. The problem wasn’t detected in InWorldz since the assets were already there and did not need to come from the OARs. Region backups and rollbacks did not save the assets since they were already available grid-wide. This is a problem for all of our OAR exports, but not all objects (more infor below).

Problem 2: The second area that didn’t get updated for Thoosa was the OAR storage code to store nested objects in Contents of other rezzed objects. The assets for those nested objects, if modified after Thoosa was introduced a few years ago, were stored in the OARs in the newer Thoosa binary format, not non-binary XML format needed by OpenSim. (This part is understandable since OpenSim couldn’t read Halcyon OARs until recently.) This means some objects in Contents were only readable by Halcyon servers, and for those objects, any assets referenced by them were not stored at all, due to the first problem above. (Non-object Contents items are not affected.)

Problem 3: OpenSim seems to have a problem where if it cannot find an asset referenced by an object, it doesn’t rez the object at all. It tells you there’s a problem and then refuses to rez the object. This means you cannot just rez a partial object and clean it up in an OpenSim region.

I’ve fixed the first two problems, which should fix future OAR exports and also take care of the “invalid XML” messages on OpenSim and the region hangs problem on OpenSim that can result from the dumping of invalid data to the console. However, this will not restore any assets that weren’t saved in July due to problem 1 above.

However, the process for OAR exports is for me to reload each unfiltered OAR with a whitelist of avatar accounts to include (often just one account, sometimes partners or teams), then save a new “export” OAR for use in other grids. The fact that these are reloaded under the latest Halcyon code and then resaved as a new OAR means that what does get saved should be usable in OpenSim after these changes. It will fix the Thoosa-specific nested objects to be exported in XML format, and provide at least the nested object itself.

I’ve also tracked down a couple of other issues with XML object formats not being compliant with OpenSim’s needs, which is a bit quirky, and worked around those in the Halcyon code to make them more compliant with OpenSim.

I’m still investigating the effects of these issues, now that I have a clearer understanding, to determine if I need to redo the OARs I’ve already completed or not, and if the code is now complete enough after these new changes that I can proceed again with the remaining OARs.

I should be able to proceed again in another day or so.

Update: That work was completed a couple of days ago, all looks good now. I’ve redone a couple of the region OARs already exported, but I will be redoing ALL of the regions already exported (about 20) over the next few days. Then I’ll be resuming my work through the list I have so far. If you haven’t emailed a request for your content yet, please be sure to email me according to my earlier post.

The coding is complete and we’ve produced a small number of OAR files for export from InWorldz. In some cases the intended destination is a Sim-on-a-stick personally installation or a grid running OpenSim. While the new OAR files have been updated to be recognized and supported by OpenSim, there is some data within the OAR file that will only be useful to Halcyon regions. These are assets which may not be in an XML format.

Problem: Back in July of 2015, Kitely’s technical lead (Oren) submitted some changes to OpenSim to actually dump any non-XML assets to the region console. This means that if you attempt to load a Halcyon OAR file under OpenSim, the console will stream thousands of lines of jibberish data. Because the data is known to not be valid XML in this case, it may include content that could cause the region to freeze or hang, but in the best case scenario it will simply spew thousands of lines of error messages for much of the oar load. These errors are false-positives (and will hide any legitimate errors).

Solution: You should ignore these errors and let it finish.

That’s all, just wait for it to finish. Then log in and visually verify that it has seemed to successfully load the OAR contents.

For those who build their own OpenSim executables:

For those who build their own, I suggest the following changes to mute the very noisy error handling:In SceneObjectSerializer.cs, change the first catch (in FromOriginalXmlFormat) to:

Both of these change the message to a warning, don’t bother reporting a stack traceback (since this can only happen from one place) and most importantly, completely eliminate the dumping of the invalid data to the console. This will still produce a noisy (3-lines per case) log, but OpenSim seems very verbose in logging, and this is considerably quieter than it is otherwise and won’t make you think something is horribly wrong when all is working according to plan.

I took a quick look at submitting this as a patch to OpenSim but first I must make it clear that I am not going to make it my mission to fix OpenSim issues. I have submitted fixes in the past directly to Justin, and I considered submitting this one using their patch process, but realized that there are a variety of development philosophies and I don’t know OpenSim’s. e.g. whether OpenSim has OAR debugging levels that the XML dump could be conditional upon, or whether I should remove that commented out line (or leave it for developers debugging issues), etc. Bottom line, I don’t know what to submit as a patch. I do know how to change the code to avoid this issue so I have posted that above. Since the key fix is a simple commenting out of a line, I’ll leave it to others to decide how much or how little they want to include from these trivial code changes.

This is a long post. For the specific instructions on requesting your own per-user exportable archive for a given region, skip ahead to the Instructions section below.

Background

You may have read my previous post of my Personal Changes going forward, and effective departure from virtual worlds. This is unrelated to the closure of either InWorldz or Islandz, but rather due to the combination of several factors.

Before I go, I’m trying to take care of former InWorldz users to the best of my ability. This is now specifically exportable per-user OAR archives of the regions taken at the time of the InWorldz shutdown back in July.

It was not possible to save all of the grids assets in the short time available, and without the assets inventories were of no value (they are really just directories of links to assets). However, we could save any objects rezzed in regions, including the assets referenced by those objects, so we let as many people know as possible to rez whatever they could in the few days available, and began saving OAR files of the regions.

These OAR files include many objects, items and assets protected by permissions and even full-perm items that were not licensed for export to, or use on, other grids. That said, there were many objects that users created themselves, or in small teams, and they held the Intellectual Property (IP) rights (copyright) on those items. Thus, these can be exported for their use.

InWorldz received very clear instructions from creators to not provide their creations in these per-user exportable OAR files. However, there wasn’t time to do thousands of individual archive files for each user who had content on a region, nor is it even known who would be interested in these. So a single OAR backup was done, with all referenced assets, for each region running on the grid.

A lot (weeks) of work was done coding load-time filtering to allow creator wishes to be respected and yet restore content from InWorldz, with a specific opt-in requirement for each creator (if desired). Also, because some asset references (such as textures or sounds on prims) do not have any known creator information, a full grid-wide scan of all Contents items in all prims in all regions was also performed, to try to identify the creators of each of these assets and allow them to be identified as being created by the creator of the prim, or by a user who had chosen to opt-in. In spite of this, the creators of many of the textures on prims could not be identified, and a substitute plywood texture was used. This was probably one of the larger contributing factors to Islandz not reaching the critical size needed to survive January.

Soon, with a bit more coding, that technology will be used to try to maximize the content saved in exportable OAR files. This is important to try to provide as much user content as possible while still respecting the wishes of other creators, and protecting their content from permissions-free regions elsewhere.

Islandz is Dead — Long Live InWorldz (OARs)

Without the huge workload of attempting to continue with Islandz, focus returned to the per-user exportable OAR archives from InWorldz. Since none of this content was being automatically uploaded by administrators on a new grid, but rather provided to the owners of the content for their use according to whatever license agreements they have with creators, the decision was made that textures on prims could be included in the filtered exportable OARs if the creator could not be identified. This is relatively safe because in order for the owner of a prim to place the texture, the creator of the texture would have already provided a full-perm (exportable) copy of the texture with the trust of the new user, and with or without an agreement for use.

Users who built their own objects and then texture them would have also been the original uploader of that texture in many cases, and filtering these would needlessly harm the continued use of these creations by the creator of the prims. Also, since the creator was already trusting the user to respect their license wishes, those textures can be included in an export of prims created by that second user. This means the InWorldz exportable OARs will be considerably more complete (much less plywood) than the objects that could be automatically restored by the grid administrators on the new of Islandz. However, they will still be filtered based on the recipient user’s status as owner and creator of each prim in an object, and for the creators of the Contents within each prim.

Let me try to make this point clear: again, while Islandz needed strict enforcement of explicit opt-in, exportable OARs to users can be slightly different in that, in order to put a texture on a prim, the creator had to give a copy of the texture to the new prim creator. That second user could have saved the texture outside the grid, the creator is trusting the second user to respect their license. If the export OAR filter assumes that any texture with an unknown creator cannot go out, then InWorldz is taking control of that agreement with the creator and assuming the worst, also enforcing something that the actual texture creator did not want enforced on the grid. Further, it also prevents people from keeping their own textures on their prims. It is something no other export does and filtering by creator and owner at maximum Contents depth is already very good enforcement (and likely better than ever done before anywhere else including SL). Basically, there was already an agreement in place with the texture creator and it will be respected and not overridden by the grid. It’s very subtle I know, but it’s different when you are automatically uploading content to a new grid on behalf of a user, rather than letting them choose to do so manually.

You can forget about Islandz opt-in completely; that is not relevant to the OAR filtering being done now. That was only important when regions were intended to be restored on Islandz. A “Part 2” of the original plan was to produce OAR files from InWorldz for export to other grids. That is pretty much the only part of the plan going forward. Since there will be no Islandz, these OARs are those exported versions from InWorldz, not related to Islandz in any way. They were captured back in July and it has always been said that they will be your stuff only, because they are leaving the grid.

*** Instructions for OAR Exports ***

This is not a small undertaking, and is being done for free in my spare time as part of my commitment to InWorldz users to do what I can to help them recover as much content as possible, while also trying to respect the copyright / IP of creators. If you do not follow these instructions as closely as possible, or provide false information, it is possible that you may not receive an OAR file.

You will need to provide:1. list of regions where you had content, 2. your avatar name, and 3. any other avatars from whom you have specifically obtained permission to take their content to another grid.

Note that point 3 above is unrelated to Islandz opt-in. In fact, this entire process is unrelated to Islandz in any way. You will only be receiving the content for the InWorldz avatars in that list, but you will receive their objects as well as your own. Thus this is not something for creators to give permission to but rather for teams and partners to allow a combined archive of major projects, such as the Memory Alpha region for Star Trek role play, or projects like The Land of Lar, or for romantic partners to receive a combined region with their combined creations.

Do not specify an avatar in the list in point 3 unless you have contacted them about this and have explicit permission. That avatar will need to email the addresses below to confirm that they wish to have their objects and creations on that region included in the OAR file you will receive.

If you are a content creator of any kind and you want your content (including objects owned by you) to be included in someone else’s export OAR archive, send an email to the email addressed below indicating:1. You give your permission to have the content in a specific region or list of regions given to another user.2. The name of the recipient avatar to whom you are allowing to have your content from that region.3. Whether you want things included that you have created but they own, or whether it should also include objects where you are the owner.

These OARs won’t be going to Islandz so they won’t be as heavily filtered (less plywood) but you need to provide the list of avatars who have agreed to let you take content to new grids. It is probably very short, usually members of a team, a partner, and/or a list of alts you may have built things with. It’s the best we can do to try to get your content from those regions in July to users.

Send the info above an email to: elenia.llewellyn AT gmail.com and to jim AT gridmail.org and I’ll do my best to get an exportable OAR posted and provide a download link to the avatar requesting.

I’m sorry if this is confusing (it was even more confusing to try to come up with a workable plan here)! I’m doing my best. I hope it helps some InWorldz residents at least. I don’t really have date of completion for the remaining coding, however I hope to complete it in the next week or so get started on OAR files right away after that.