I have thought about this topic again, although people have mostly voiced against the common data files so far, for good reasons.

However, I often come across the fact that certain things that are basically supposed to be the same, exist in completely different fashion across the projects. Things like amber, iron ore, silver ore and others exist in TR, SHOTN and P:C concurrently with differing values and effects. We have a bunch of redguard objects at SHOTN that would be useful for P:C and the other way around. A bunch of ingredients and container plants could be exchanged yet, like the potato, corn (these are much better at SHOTN) and lavender (an asset at P:C that would be relevant for Skyrim).

I think it would be well-possible to devote an update to making these minor things consistent between the projects and exchanging some more stuff. But there are still other things. Just think about books. All of the three projects write their own books, but somehow only the vanilla books and books from other TES sequels manage to exist across the province borders.

There are a bunch of other assets that would be extremely useful as purely situational objects, like samples of Cyrodillic wine or silk cloth in Skyrim or Morrowind. Adding these as constant installations into Skyrim-Data, for example, would possibly seduce into using them more often than it should be and they would otherwise only clutter the object list.

These are some aspects that speak for a common data files. Actually what I have in mind, wouldn't only involve Skyrim-Data and PC_Data but include TR_Data as dependency aswell. There are those gems TR uses. I have selectively added a few of those into our data files, but most of the others I have excluded would reasonably come from outside of Morrowind. I don't think TR will add deposits for all of them, at least. With TR_Data as dependency we could also replace the TR duplicates we have in our weapon- and armor-list with the originals. What about dunmeri export goods? Why should they only encompass goods from Vvardenfell? This is less much a concern for Skyrim, but Cyrodiil being a province of trade can only benefit from exotic goods.

A common data files would allow using the assets of the other projects anytime and it would always be a consciuos act of scrolling to the objects of the other projects, which would ensure that the projects mainly still stick with their own stuff. There are a bunch of useful things i have made for SHOTN like a special waterfall and many things i keep forgetting about, that could be used at the other projects without any concern. Stuff like the broken bottles and the water meshes that currently exist as duplicates in all data files could be replaced with the TR versions for which there have been made scripts even to simulate water behaviour. These are only some examples (some of the clothing could be shared aswell).

However the very important drawback is that everytime a project adds new stuff to their data files, the complete Tamriel_Data is updated. Also, every project will have to deal with an exceedingly long list of statics and containers all the time.
If the assets themselves remain in separate bsas that are downloaded separate from the esm-file the issue of the big download at each little update can be circumvented. But it is less straightforward, for sure.

Then there is the factor of Tamriel Rebuilt. I don't know, how much of that movement would take the other way around, with stuff from Tamriel_Data used at TR. They could really use our updated cyrodillic wines, for example. TR generally seems very open for resource exchange and cross-project coordination, but I have the impression that there are some worries, still, especially when stuff is extremely highpoly or highres (an overly photographical look of a texture is probably more problematic than the amount of pixels it contains, though).

I wonder, if we should step forth and speak to them about a common set of design directions, least common denominators and stuff like that, things to ensure that our work in the end does not utterly contradict them, as some of what they do certainly bear implications for the other provinces. I'm mostly dreaming about an ideal, connected world where things are somewhat plausible and consistent, with all economic goods having their sources somewhere in the gameworld for example and the imperials in Skyrim and Morrowind referring to the political circumstances as elaborated at P:C.

If you think that it's not realistic or takes away too much attention for something people will eventually not even notice, let me know. I'm not sure myself, really. I don't think we go particularly bad, as is, but there is still space for improvement, for sure.

arrianas wrote:I really like that idea, and I have wondered why it hasn't happened yet.

It hasn't happened yet for reasons Muspila partly mentioned. Imagine you're working on an interior claim for ShotN that doesn't involve any items coming from Cyrodiil, you'd still have to load every single item used in P:C. Not a problem per se, but the CS still is very slow when it comes to loading assets and I can see how it can become an annoyance.

Next up is the fact that not everybody works on every project. If a person is working on ShotN claims exclusively, why would they want to download an additional 400MB or more, and that's for practically nothing? As noted, items coming across borders should be sparsely used and only to show a distinct consistency within the gameworld, so most of what you download and load is basically redundant for your claims.

As far as I have gathered, the issue we've had so far is the same issue we've come across when we considered merging all projects into a single Project-Tamriel-Mod (not to be confused with the forum merge, which is merely a matter of convenience for all parties). It just so happens that there are modders who don't want to participate in other projects, and for good reasons. The fact that many of us participate in multiple projects on this board doesn't mean we should be selfish and disregard these other modders focusing on one province.

Muspila, I'm not sure I have wholly understood what you want exactly. Is it a single file containing all the data from P:C, ShotN, TR and others, or is it a single file containing P:C and ShotN data and a dependency on TR added to it? I fear it is not quite clear from what you've written. Overall I'd like to see some of the duplicates eliminated. And quite often it has happened that I was desperately looking for specific items which I thought were in the respective data files, only to find out that they hadn't been included in other data files even though they could have been. This would be a most welcome change indeed.

Muspila, I'm not sure I have wholly understood what you want exactly. Is it a single file containing all the data from P:C, ShotN, TR and others, or is it a single file containing P:C and ShotN data and a dependency on TR added to it? I fear it is not quite clear from what you've written. Overall I'd like to see some of the duplicates eliminated. And quite often it has happened that I was desperately looking for specific items which I thought were in the respective data files, only to find out that they hadn't been included in other data files even though they could have been. This would be a most welcome change indeed.

The idea was to make it dependant on TR_Data. Just merging the objects in will only cause trouble, whenever TR changes stats of their objects.

It would be possible to just keep going as we are and copy assets over that seem fit across the province borders. But it is a lot of fuss and duplication, if you want to make sure that they always remain consistent with each other. A possibility that has jumped to my mind, would be making a separate data which is dependant on TR_Data, which contains such stuff as ingredients, container plants and general purpose objects, that tendencially exist as duplicates across the data files and use these data files in addition to the projects data files. However, it would mean that each project uses 3 data.esms. 2 esms are more than enough dependency already. No, this is not an option.

I think that the genuinely best solution would be creating a completely new, common Data.esm from scratch and make sure that this Data pursues a good, mainly project-agnostic naming pattern which is foremostly oriented on purpose and place of the object in the game world. This would dissolve much of the independency of the two projects, but would be very effective for creating a seamless world. It would take a lot of effort to make this data and make the existing content use the latter, but in the end every economic good, every piece of literature and other things that are extremely handy would be readily available for everyone. It would be a very long statics list, but with a good naming pattern, you would be able to find your stuff relatively quickly and there is a lot of stuff that would be useful across the projects.

If we had something like this, we could also take the effort of actually merging in the TR objects. We would need a data.esp where our own objects exist in separately and only merge this file with TR_Data as the last step of the data update. Like this updates of TR_Data would be relatively painless, too.

worsas wrote:[...]It would take a lot of effort to make this data and make the existing content use the latter[...]

Right - whoever takes the job to create the new entries in this new data file would have to properly document which entries have been renamed, and later someone would do a global search & replace for all these objects in the very latest files.

worsas wrote:[...]It would be a very long statics list, but with a good naming pattern, you would be able to find your stuff relatively quickly[...]

My thoughts exactly. It happens quite rarely that you actually go through the statics list step by step, and even in these cases a solid naming pattern helps a ton.

Sounds good to me, really. Before anything happens we need a huge announcement and a poll. Let everyone vote and then we'll see if this receives any acclaim.

We should also look for a suitable moment then, assuming that this finds common approval. I don't know if it would be wise to tackle this before the coming releases, for example.

I also have some extended thoughts beyond this technical merge of the Data Files that I would like to write about later, but they are not completely trivial and come with many opposing points of view fighting over their rights in this development process. At the core of it is always the same question: What do we primarily want to achieve? Within this product we are working on, for ourselves and for the people that may eventually come in touch with it? What things deserve priority over others? Meta stuff like that.

I have started to get some more nerdy ideas while thinking about this data files stuff.

How about creating something like material sets? Let's say there is a certain kind of wood or a certain stone or metal. For each material we could promote a tilable pattern texture that specifies the general look of the material, maybe in different states like rusty, non-rusty for metals, withered and non-withered for woods, etc. In the vanilla game each metal has got a mostly consistent look. Silver has got a certain color and lightness that is everywhere the same. Wood in dunmeri structures tends to have an equally greyish color. Expensive imperial furniture uses cherry wood which has got a certain color, too. It is similar with glass, clay or other materials.

A set of reference materials would help to establish more visual continuity, especially for things made from wood, steel or iron. I'm also thinking about the current usage of unfitting wood textures at SHOTN. Why doesn't the lower class furniture and crates use cheap pine wood, which is available in abundance? Why does the wood used for docks look new and unaffected by this exposure? A more greyish/greenish color scheme for cheap, unpainted woods, especially when used for exterior places would be a much better fit than what we currently have.

Also, how does a plain silk-, or linen-canvas look like? Which pigments are available and what's their color impact? What is emperor red? This could include collecting the colors seen on vanilla- and TR- clothing. What's the color of leather?

I'm even thinking about stuff like climatic overlay colors for the various regions on the continent to account for differing light environments, as there is no color values for exterior cells to influence the appearance of exterior objects. We could also collect general-purpose overlays to add signs of exposure to materials in the exterior space, like moss-, lichen-, snow- or general grunge-overlays.

The colors in the vanilla game are not always realistic and rather stylized. Everything worn by people (clothing, armor, hair) tends to be more saturated. Materials in the environment opposedly tend to have a very low saturation. Maybe some of the color scheme on exterior objects doesn't account for realism that much, but rather wants to convey a certain mood. This is entirely reasonable, but it would be cool to have some kind of neutral median for each material nevertheless, maybe using vanilla.

I guess I'm starting to get overbearing with this all. But it's fun to think about it nevertheless.

---------------------------

Another thing: If we started to make a common Tamriel-wide data files, we would probably stop filtering additions by their likely usefulness in the particular province. So far I have hesitated to add heads and hair of other races to Skyrim_Data, for example, as it seemed wasteful to put much effort into the other races. This wouldn't apply anymore.

Starting from scratch would also offer an opportunity for being more selective and exclude old stuff that remains in the data files for pure compatibility sake or stuff that doesn't mesh well together. I'm thinking about the ship types at SHOTN here or some of the retexes that make no sense. If you carry over stuff piece by piece, you have the chance to look at each of these things again and exclude them or change them to make them more fitting.

Yesterday I announced I would speak about some meta-stuff. But it isn't quite as meta, as I thought it would be. There are maybe some issue with my big consistency effort around here: Consistency isn't necessarily the biggest good. Some of the game also simply defies realism. Take the imperial fortresses in the vanilla game as example: Where do they get their completely different-colored brick stones from? This is a question the game isn't concerned with. But as pointed out above, the game seems to do both, use some kind of semi-realism where it serves to increase the immersion or disregard it in places where probably nobody will think about it or where they want to do something new and cool.

At some point it is just a game and while continuity of materials is nice stuff, the good stories are told in other places. So this whole fuss isn't needed, unless we want it and will have fun at it.

Holy wall of text. Some nice creative ideas in here, I can see the access to a wider range of items having its benefits in a lot of places. The clean and bright look of the poor wood furniture and exterior woodwork is something that's bothered me for ages.

Sounds like a lot of work. It may be a concern, too, that if you have a lot of variations (or states, as you put it) of the same wood, modders are sort of drowned in a flood of choices and they wouldn't know what to use exactly. Not sure if this really is an issue.

We definitely should adjust some of these rather unnatural aspects in some of our existing objects that you speak about. Vanilla may have had a lack of realism in order to create a unique world with a certain color- and mood scheme, but certain statics like dock pieces did have a rather worn, drained look.

The idea of a materials set is very interesting. As a modeler, I often struggle to find proper textures - that's because I'm pretty bad at texturing, actually, but it would be useful to have a smart catalog of materials and colors on hand. it would improve immersion quite a bit, and actually save time since we don't have to start from scratch every time.

I don't think this would result in a flood of choices; much the reverse actually. We would have a clear guide of what is appropriate where and when, as opposed to the current situation where we have the entire web to pick through for materials and inspirations.

I am also fully in favor of a unified data library. If the content is listed in a logical and concise manner, and we can curate the current objects for doubles and bad quality, I can't actually see any drawbacks. The only thing would be the download size, but, as worsas said, objects can stay in their discrete .bsa's without issues.

Here is an overview of how the renaming could roughly look like. The main idea is that an object is primarily categorized by race/ethnicity or by province. Race-assigned assets may well show up outside of their province. Province-specific assets are more or less geographically bound to their province.

An object id would be structured as follows:
PT_[Category]_[Name of the set]_[suffix for static sets]_[name of the object]_[counter] --The words in the bracket placeholders MUST NOT incorporate additional underscores--

Sky_In_Barrow_02_Trans_Cave_01 ---> PT_Nor_Barrow02_I_TransCave_01
Sky_Ex_Altmer_Wall_01 ---> PT_He_Direnni_X_Wall_01
PC_Furn_Col_Barr_Brk4 ---> PT_Imp_Col_Barrow_F_JarBrk_04
PC_Furn_Rug_Big_01 ---> PT_Imp_RugBig_01 (F suffix not needed, because the rugs are not part of an overarching set)

Sometimes an object set is categorized by both a race and a province. In that case, the province category comes in front of the racial category, as you'll only want to use these in their respective province:
Sky_Ex_Impruin_Basement_01 ---> PT_Sky_Imp_Ruin_X_Basement_01
PC_In_Impruin_Room_Floor_01 ---> PT_Cyr_Imp_Ruin_I_RoomFloor_01
Sky_Ex_Imp_Tower_B_Top_02 ---> PT_Sky_Imp_Fort_X_TowerBTop_02
PC_Ex_Imp_Bridge_02 ---> PT_Cyr_Imp_Fort_X_Bridge_02

Sets of rocks, boulders and cliffs using the same rock texture will be grouped under a single terrain set:
Sky_Terrain_Rock_Grassy_04_09 ---> PT_Sky_Terr04_RockGrassy_09
Sky_Terrain_Cliff_S_02_K1 ---> PT_Sky_Terr02_CliffS_02K1
PC_Terrain_GC_rock_07 ---> PT_Cyr_TerrGC_Rock_07

Ingredients are not grouped in sets at all but may be assigned to a race:
Sky_Ingred_Garlic_01 ---> PT_Garlic_01
PC_Ingred_TigerLily_Nectar ---> PT_TigerLilyNectar_01
PC_Ingred_CCheddar ---> PT_Imp_Col_Cheddar_01
Sky_Ingred_Cheese_01 ---> PT_Nor_Cheese_01

-At the moment, my thought for barrels, crates and sacks is that they should be categorized by province rather than race. But this is open for debate.

Body parts use prefixes in front of the category, as body- and clothing- would get mixed with armor-parts in the list. The assigned part always acts as counter:
PC_A_Steel_Boots_A ---> PT_A_Imp_Col_Steel_Boots_A
PC_Col_C_Pants_W1_02_K ---> PT_C_Imp_Col_Common_Pants02_K
Sky_C_F_Dress_01 ---> PT_C_Nor_Common_Dress01_CF (CF standing for 'chest female')
Sky_B_N_Nord_M_Hair_02 ---> PT_B_Nor_Male_Hair02_H
PC_B_Col_F_Hair_03 ---> PT_B_Imp_Col_Female_Hair03_H

For weapons, clothing and armor the name of the enchantment can replace the counter. Also, all artifacts are grouped under the 'UNI'-set. Artifacts don't need a counter:
Sky_Companion_Greaves ---> PT_Nor_Companion_Greaves_01
Sky_Nord_Shield_Ironwood_02 ---> PT_Nor_Ironwood_Shield_02
PC_Newtscale_Pauldron_L ---> PT_Imp_Nib?_Newt_PauldronL_01

Books, Scrolls and the like are a bit more complicated, because we have differing book models in the two projects. Here we need to discuss the purpose of these. If each province retains its own book models sharing texts will require copypasting them to the other book model, relatively little effort, but potentially a lot of doubling. Opposedly assigning certain books to certain book models will require repositioning in existing interiors.

I would choose a suitable moment to merge Skyrim-Data, PC_Data and all content of both projects (reviewed and unreviewed) into a temporary file. The Ids can be edited within the Enchanted Editor that allows for ids up to 31 letters and adjust references of the objects at the same time. When all ids have been renamed, the file is split up into the pure data and the pure content part again. This would be the only halfway non-lethal way of implementing these name changes. The content file would need additional splitting up afterwards. But that would be a comparably minor issue.

There is no way you can still squash a 'furn' or 'ex' or 'in' into these ids. You wouldn't gain anything from using them anyway. The statics list would be primarily sorted by thematical sets (direnni, ayleid, barrow, etc.), which has got the big advantage that all objects of a thematical group are found in one place rather than spread across 'ex', 'in' and 'furn'. I think that is a much better way of sorting statics, at least.

Afaik, 'red' is used for Redoran in morrowind.esm. I thought about using dwrv but left it to dwr to not exceed three letters for the main categories.

worsas wrote:*Snip*I think that the genuinely best solution would be creating a completely new, common Data.esm from scratch and make sure that this Data pursues a good, mainly project-agnostic naming pattern which is foremostly oriented on purpose and place of the object in the game world. *Snip*

This! This sounds beautiful.

Its always concerned me, the amount of assets spread about between the projects. I genuinely think all of the projects have the same goals and ideals in mind. Having a single PT_Data to work from would be a much better solution. Even at this stage in development.

You wouldn't have to move all references to a single data file in one go either. And you wouldn't need to entirely remove dependency on that projects individual datafile. For example. Things that would 100% never exist in another province can stay in that provinces data files.

I really don't think it would be a problem at all if ShotN for example, relied on Skyrim_Data, PT_Data and TR_Data. All that these three contain should just be references. No land, cells or dialogue. So loading them along side the exterior, interior or whatever separate plugins, wont slow them down much at all.

I do think the two main projects datafiles do need a good clean up. Seriously, some of the naming in the Skyrim data.... its prefix hell!

To my surprise, a common Data with Tamriel Rebuilt seems to have moved into prospect. I spoke to a number of people on IRC, mainly the current lead developers at TR and they all seemed to like the idea of all province mods using the same data and generally working out a common version of Tamriel with us.

Though that would naturally require even more coordinative effort than a common data for only the projects around here, but the interest in doing that seems to be there, at least.
I personally think it would be the ideal way to achieve a seamless gameworld.

This is great news. Personally I am fully in favor of a unified common data library in the interest of a seamless gameworld.

Relatedly, I think I'll investigate TR's depiction of Legion and other Imperial factions. No reason we can't align the fiction as well. I'd be interested to see if they have any ideas on how to acknowledge the player's rank in related factions.

I'm not sure about joining all province mods into one master file, really. I can imagine it causing more cons than advantages, both for the player and for us, modders. They were already listed by Worsas and others and I'm really not sure they balance favourably. SG's concept, hovewer...

SGMonkey wrote:You wouldn't have to move all references to a single data file in one go either. And you wouldn't need to entirely remove dependency on that projects individual datafile. For example. Things that would 100% never exist in another province can stay in that provinces data files.

I really don't think it would be a problem at all if ShotN for example, relied on Skyrim_Data, PT_Data and TR_Data. All that these three contain should just be references. No land, cells or dialogue. So loading them along side the exterior, interior or whatever separate plugins, wont slow them down much at all.

I'm all for it. Maybe, if you guys are sure about eventually joining project esms into one common data, we could start with this approach? To get the job gradually done and as a practice run of some sort?

At any rate, I can help with cataloging and reviewing SHotN's data, I got rather familiar with it over these months.

worsas wrote:Also, every project will have to deal with an exceedingly long list of statics and containers all the time.

If we make province mods depend on that common data esm, not the other way around, it shouldn't be that bad? That way the modder for TR e.g. wouldn't have to load SHotN's esm as well, only the common data. And I wouldn't include that many statics in it - taking SHotN as an example, barrows assets, common/nord furniture and some flora might suffice. And maybe some old ruins, there was a talk on TR's boards recently about SHotN's Reman forts. And probably tons of other stuff I'm forgetting about, this data is quite massive. Books, creatures, container plants, ingredients, weapons, armours, cultural and common miscellania, flavour clothing and alchemical stuff would all go to this new esm, unless it's strictly region-related reference. Direnni ruins, Reachmen huts and Orsimer camps assets would all stay in SHotN's data for example, as they shouldn't be used by any other province mod (not until HR has a need for them).

The new naming convention is a great call. I agree that currently one has to jump back and forth from ex through furn to in static prefixes, all working on the same area.

Agreed the nord poor wood texture seems too clean. Good point about the enviromental colour variants too - it would be great to have some more general stuff, like nord fort assets, for example, available in normal, frosted and lichen-overgrown variants.

Edit: That common data could make a great resource for modders too. I can imagine plugins like "Nord decorations for Skyrim Mission in Ebonheart", "Jobasha's Rare Books: Tamriel", "Skyrim monsters for Solstheim" that are dependent on it. Cool. Some trans-province quest would be possible too, at least most simple ones, like "bring me X of Y from province Z". Really, it opens lots of possibilities up.

We shouldn't start to create these common data files just yet, but in the end, I'd like these data files to contain all stuff of all projects, even though it will result in very long object lists. Alternatively, we'd have a mess of objects from our original data, vanilla objects and objects with the new naming scheme of the common data files. Though, if I misunderstood your post and you had something different in mind, let me know.

At the moment, I feel a bit overtasked with this data files undertaking. Working towards our coming releases seems more important, right now. There are still some unanswered questions, too. Anonytroll has suggested creating a vanilla-style data files as default data and having our highres textures in an alternate bsa. What gives me a headache about that plan, while it probably would be the best thing to ensure that there is visual consistency, is that making textures look vanilla-friendly but not shitty at the same time, requires you to do some more beyond just downsampling them. The vanilla look is a topic in itself.

Then I'd like to establish material consistency, especially for metals and wood. Some of our objects, in particular the P:C ones, make use of the environmental effect, but it is still unclear whether and to what degree openmw will support it in future. Otherwise I'd thought about carrying over the effect to all other metal objects for the highres data, maybe even adding normal maps for all objects within the latter. But openmw in the osg branch doesn't even support the normal maps anymore, right now. There are still too many unknown variables at this point.

worsas wrote:We shouldn't start to create these common data files just yet, but in the end, I'd like these data files to contain all stuff of all projects, even though it will result in very long object lists. Alternatively, we'd have a mess of objects from our original data, vanilla objects and objects with the new naming scheme of the common data files. Though, if I misunderstood your post and you had something different in mind, let me know.

You got it right, that's what I meant. I don't really know, it's already quite annoying to search - for example - for one certain flora static between the loads of exes and furns. I don't mind quick jumps to flora_bc or _holly (in container section) though. I bet that solution would be equally convenient. Maybe it's just me, I can imagine one argumenting one way or another. It would be easier if OpenMW CS would include something like Ctrl + F function to stroll through IDs more efficiently.