Missing and unused data files

I have a script (source/tools/entity/checkrefs.pl) that searches for references between data files, and tries to identify references to non-existent files, and also files that aren't referenced (either directly or indirectly) from anything the game will use.

In particular, "anything the game will use" includes all entity template XML files (except "template_*.xml"), all scenario PMP/XML files, all actor XML files (since there's loads that are intended as editor eyecandy but aren't used yet), all GUI page_*.xml files, and all civ *.json files. This doesn't include files the engine uses directly (e.g. water textures), nor most of the texture files chosen dynamically by scripts (e.g. some icons), nor terrain textures that aren't currently used on any maps, nor anything used by RMS scripts; it's possible that various files in those categories are reported but are not safe to delete. Also the script might have some bugs and fail to notice some references between data files, but I've tried to be reasonably careful with it.

It would probably be nice (though not a particularly high priority) if some people could check through the lists and see if there's anything that's either a fixable bug, or is legitimately unused and obsolete and should be deleted, and then fix/delete as appropriate.

not used public/art/animation/biped/not used/backflip.psa
not used public/art/animation/biped/not used/fem_idle_a.psa
not used public/art/animation/biped/not used/fem_walk_b.psa
not used public/art/animation/biped/not used/idle.psa
not used public/art/animation/biped/not used/ijv_01.psa
not used public/art/animation/biped/not used/inf_2hspear_attack_a.psa
not used public/art/animation/biped/not used/inf_2hspear_attack_b.psa
not used public/art/animation/biped/not used/inf_2hspear_attack_c.psa
not used public/art/animation/biped/not used/inf_2hspear_attack_d.psa
not used public/art/animation/biped/not used/inf_2hspear_attack_e.psa
not used public/art/animation/biped/rider_archer_atk_a.psa
not used public/art/animation/cavalry/arch/attack/rider_archer_atk_a.psa
not used public/art/animation/cavalry/javelin/attack/rider_javelin_atk_a.psa
not used public/art/animation/infantry/general/forage_fem.psa
not used public/art/animation/infantry/general/orchard.psa
not used public/art/animation/infantry/support/idle/fem_01.psa
not used public/art/animation/infantry/support/move/walk/fem_01.psa
not used public/art/animation/mechanical/ship_idle.psa
not used public/art/animation/mechanical/ship_move.psa
not used public/art/animation/quadraped/elephant_idle.psa
not used public/art/animation/quadraped/elephant_walk.psa
not used public/art/animation/quadraped/gallop.psa
not used public/art/animation/quadraped/horseidle.psa
used public/art/animation/biped/cavalryidle.psa
used public/art/animation/biped/dudewalk_swordshield.psa
used public/art/animation/biped/hoe.psa
used public/art/animation/biped/inf_arch_atk_a.psa
used public/art/animation/biped/inf_hoplite_atk_a.psa
used public/art/animation/biped/inf_hoplite_idle_a.psa
used public/art/animation/biped/inf_hoplite_shield_run_a.psa
used public/art/animation/biped/inf_hoplite_walk.psa
used public/art/animation/biped/inf_phalanx_atk_a.psa
used public/art/animation/biped/inf_phalanx_c.psa
used public/art/animation/biped/inf_phalanx_walk_a.psa
used public/art/animation/biped/inf_salute_a.psa
used public/art/animation/biped/inf_salute_b.psa
used public/art/animation/biped/inf_salute_c.psa
used public/art/animation/biped/inf_salute_d.psa
used public/art/animation/biped/inf_salute_e.psa
used public/art/animation/biped/inf_sling_atk_a.psa
used public/art/animation/biped/inf_spear_shield_atk_a.psa
used public/art/animation/biped/inf_spear_shield_atk_b.psa
used public/art/animation/biped/not used/ijv_def_02.psa
used public/art/animation/biped/not used/ijv_def_03.psa
used public/art/animation/biped/not used/inf_2sword_attack_a.psa
used public/art/animation/biped/not used/inf_2sword_attack_b.psa
used public/art/animation/biped/not used/inf_2sword_attack_c.psa
used public/art/animation/biped/not used/inf_2sword_attack_d.psa
used public/art/animation/biped/not used/inf_2sword_attack_e.psa
used public/art/animation/biped/not used/inf_phalanx_a.psa
used public/art/animation/biped/not used/inf_phalanx_b.psa
used public/art/animation/biped/not used/inf_phalanx_d.psa
used public/art/animation/biped/rider_gallop.psa
used public/art/animation/biped/rider_javelin_atk_a.psa
used public/art/animation/biped/rider_spear_shield_atk_a.psa
used public/art/animation/biped/rider_sword_death_a.psa
used public/art/animation/biped/rider_sword_shield_atk_a.psa
used public/art/animation/biped/walk_spearshield.psa
used public/art/animation/cavalry/rider_gallop.psa
used public/art/animation/cavalry/spear/attack/rider_spear_shield_atk_a.psa
used public/art/animation/cavalry/sword/attack/rider_sword_shield_atk_a.psa
used public/art/animation/infantry/general/chop.psa
used public/art/animation/infantry/general/death/inf_01.psa
used public/art/animation/infantry/general/death/inf_02.psa
used public/art/animation/infantry/general/death/inf_03.psa
used public/art/animation/infantry/general/death/inf_04.psa
used public/art/animation/infantry/general/death/inf_05.psa
used public/art/animation/infantry/general/death/inf_06.psa
used public/art/animation/infantry/general/death/inf_07.psa
used public/art/animation/infantry/general/dude/dudeattack1.psa
used public/art/animation/infantry/general/dude/dudeattack.psa
used public/art/animation/infantry/general/dude/dudebuild.psa
used public/art/animation/infantry/general/dude/dudechop.psa
used public/art/animation/infantry/general/dude/dudedeath_sword.psa
used public/art/animation/infantry/general/dude/dudedecay_sword.psa
used public/art/animation/infantry/general/dude/dudeidle.psa
used public/art/animation/infantry/general/dude/dudewalk.psa
used public/art/animation/infantry/general/forage.psa
used public/art/animation/infantry/general/mine.psa
used public/art/animation/infantry/general/run.psa
used public/art/animation/infantry/javelin/attack/ijv_off_01.psa
used public/art/animation/infantry/spear/idle/isp_01.psa
used public/art/animation/infantry/sword/attack/isw_s_def_01.psa
used public/art/animation/infantry/sword/attack/isw_s_def_06.psa
used public/art/animation/infantry/sword/attack/isw_s_em_04.psa
used public/art/animation/infantry/sword/attack/isw_s_off_02.psa
used public/art/animation/infantry/sword/attack/isw_s_off_03.psa
used public/art/animation/infantry/sword/attack/isw_s_off_05.psa
used public/art/animation/infantry/sword/move/run/isw_s_def_02.psa
used public/art/animation/infantry/sword/move/run/isw_s_em_03.psa
used public/art/animation/infantry/sword/move/run/isw_s_off_01.psa
used public/art/animation/quadraped/elephant_rider_idle.psa
used public/art/animation/quadraped/horse_attack_a.psa
used public/art/animation/quadraped/horse_attack_b.psa
used public/art/animation/quadraped/horse_death.psa
used public/art/animation/quadraped/horse_gallop.psa
used public/art/animation/quadraped/horse_idle_a.psa
used public/art/animation/quadraped/horse_walk.psa
used public/art/animation/quadraped/mastiff_idle_a.psa
used public/art/animation/quadraped/mastiff_run.psa

As far as the meshes go, are these meshes that aren't associated with any actors? If so, I wouldn't mind going through them and taking a look at them in Atlas before deciding what to delete. It'll take a while, but I might find some diamonds in the rough.

Yes - any mesh referenced by any actor is counted as 'used'. (There might be other meshes referenced only by actors that are never used, but those actors *could* be used as eyecandy by map designers, so I'm not counting those meshes as unused yet.)

If you don't have an easy way to view the meshes, I think I could easily generate an (untextured) actor file per mesh so you can load them in Atlas easily.

If any of them turn out to be useful (unboned models that is), we should convert them to DAEs, and delete the ones that aren't useful. Then if someone wants to redo the animated PMDs in COLLADA *and* provide the source .blend or Max file for other artists to use, they would be my hero.

See attached file - no need to unzip, just stick it into mods/public/ and look for actors named "unusedmesh/...". (I put some random texture on them to hopefully make them slightly more visible in the actor viewer.)

I just committed a deleting of 250-ish useless or duplicate PMD files. I kept a few of the "unused" ones, like helmets and shields (I'll have to go through these with a fine toothed comb) and some others I found to be useful as variations of existing assets (like additional variations of swords that got overlooked previously). I also found a pretty sweet dude head with a wicked beard. Using that for Brennus now if you want to look at it. Not sure who else I'll give it to.

Nice job. For what it's worth, I'm working on the PMD-DAE converter script. It works with props now and is very close to working on bones, too. I just need to test a bit more and then we can run it on all the PMDs we want to keep. I'll take a look into a PSA-DAE converter too and see if that's feasible.

Is any art "source" being lost from this cleanup? certainly removing exported files from the distribution bundles is a no-brainer, but I'd be wary to delete something that could be used by a mod in the far future

Is any art "source" being lost from this cleanup? certainly removing exported files from the distribution bundles is a no-brainer, but I'd be wary to delete something that could be used by a mod in the far future

Svn keeps a full history, this just removes the files from the current checkout.

For the unused animations... is it possible some of the files are duplicated and used in another folder? Many of them look like they are unused because we are waiting for return of the run/stamina feature.

Animations are a precious commodity so I find it odd that any are not in use.

There's a slightly old list of duplicate files here. (Each section is listing two or more files which have precisely the same contents as each other, e.g. the first section says male.dds and plac_rome_e.dds are identical). Looks like only five animations in that list.

The source files for all the max animations are already in the Art SVN repository. Anyone with an active Max account should be able to reopen the .max files and export a .dae from it.

Does anyone on the team have 3DS Max? I for one do not At least some of the remaining skeletal PMDs (horse.pmd, mastiff.pmd) appear to have the original max files in the art SVN as you say. Anyone able/willing to convert them to DAEs?