Wednesday, May 24, 2006

ArchiveInvalidation Destroyed

Previous Thread

SOLVE YOUR ARCHIVEINVALIDATION PROBLEMS FOR GOOD!
Updated 7-1-2006 11:15 AM CDT

Timeslip's Oblivion Mod Manager (0.7.10 or later) and BSA Patcher provide an archive invalidation method that will rename textures in your BSA files so you never have to worry about ArchiveInvalidation problems again. This method is vastly superior to standard invalidation methods.

This BSA alteration method of invalidation has been tested by the ArchiveInvalidation Explained Research Team and adoption of this method in the community is rapidly growing. It has proven extremely stable. You should probably make a backup of your BSA files if you have the space (or a DVD-burner) just in case, but we haven't seen any corruption issues.

OBMM and BSA Patcher will keep track of which files have been renamed in your BSAs and provide a "Remove BSA edits" function to rollback any changes they have made to your BSA.

Assuming you have OBMM 0.7.10 or later already, and have already installed/activated some mods, the detailed steps for doing this in OBMM are as follows:

1. Start OBMM;
2. Click Utilities;
3. Select Archive invalidation;
4. Click Directly Edit BSAs;
5. Click the checkbox for autoupdate on exit and/or click Update Now;
6. Click the red X in the upper-left corner of the Archive invalidation popup to close it;
7. Click Launch Oblivion
8. Play!

If you didn't check the box for autoupdate on exit, then repeat this process after installing/activating each new mod.

The latest version of OBMM (0.7.11 or later) provides a lot of new options for the Directly Edit BSAs function. The default options may be okay, but you need to at least make sure to check the box for Generate archiveinvalidation entries on hash collision, since that setting is critical.

Here are the ideal OBMM BSA Settings (for 0.7.11 or later).

Using those settings will take care of everything for you.

As already noted, it's completely up to you whether you want to use autoupdate on exit or Update Now, or both. I find that when running with a lot of texture replacements installed, the autoupdate on exit option tends to cause an annoying delay when quitting OBMM or using the Launch Oblivion button. Since I've gotten into the habit of using the Launch Oblivion button instead of the normal launcher, and I don't install new mods nearly as often as I start Oblivion, I decided to avoid using autoupdate on exit and instead always use Update Now after I install a new mod.

If you later uninstall/deactivate some mods, you'll need to use the Remove BSA edits option. Follow these steps:

1. Start OBMM;
2. Click Utilities;
3. Select Archive invalidation;
4. Click Remove BSA edits;
5. Close OBMM;
6. Delete texture files that you want removed;
7. Start OBMM;
8. Click Utilities;
9. Select Archive invalidation;
10. Click Directly Edit BSAs;
11. You know the rest by now smile.gif

If you're using .omods, you should skip Steps 5-7 and just Deactivate the .omod instead.

New, Improved Version of BSA Patcher
BSA Patcher is now dramatically easier to use.

Download BSA Patcher

Unlike OBMM, BSA Patcher is a command-line application, although you can launch it by double-clicking the executable. BSA Patcher doesn't include as many configuration options as OBMM, but it accomplishes the same tasks and it only requires .NET 1.1, which means it will work for anyone who is unwilling or unable to run .NET 2.0. It will also work with mono, an open-source version of .NET 1.1.

The latest version of BSA Patcher (as of this writing) uses the BSA alteration code from OBMM 0.7.10 and, like OBMM, it will automatically generate an ArchiveInvalidation.txt file for you containing only the textures that could not be safely renamed in your BSA files. It includes support for several command-line options, but you can safely ignore them.

To use it, place BSAPatch.exe in your \Oblivion\Data\ folder and double-click it once to rename files in your BSA. Run it again to restore them to their original names.

NOTE: If you use either OBMM or BSA Patcher, you should not manually alter your ArchiveInvalidation.txt file. Let one of these programs do it for you. No other solutions are recommended.

Older Versions of BSA Patcher
If you have an older version of BSA Patcher, make sure you have at least the version based on code from OBMM 0.7.9. You should probably still make a backup of your BSA files before running it, just in case. Run it once to rename files in your BSA and run it again to restore them to their original names.

If you're not sure what all this is about, don't have any mods installed yet, or just need more information on how to use mods in Oblivion, please read the Oblivion Mods FAQ for more general instructions. There's also another version of the FAQ on the CS Wiki, which is a bit easier to read because it has a working table of contents.

For more information on ArchiveInvalidation, read my blogged version of ArchiveInvalidation Explained (or check to see if the original ArchiveInvalidation Explained thread still exists). The results of our testing are summarized neatly in the first post.

If you have questions about all this, post them here and members of our team will attempt to help you. You may also want to check the OBMM thread for updates.

DETAILS
Oblivion has major issues loading some replacement textures, even with an ArchiveInvalidation file. What OBMM and BSAPatch do is to sidestep this problem by making Oblivion think it never had a copy of the file you are replacing.

This feature edits the BSA archives so that oblivion cannot find the original version of files you have replaced, thus forcing it to search elsewhere for them.

Here's the summary from Timeslip's site: "OBMM will scan each of the BSA's, and check for the existence in the data directory of each file within the BSA. If a match is found, the hash of that file is incremented. Oblivion never reads file names, and relies solely on the hashes. As a result, it never notices the filename/hash mismatch, and never sees anything wrong. As far as the player is concerned, mods should suddenly start working correctly, entirely without the need for an archiveinvalidation file."

UPDATES
The 0.7.10 and later version of OBMM correct an issue from previous versions where several files with very similar names in the same directory (like castleintcolumndetail02.dds, castleintcolumndetail03.dds, and castleintcolumndetail04.dds) might get their textures mixed up if a replacer only replaces one of these files. OBMM now generates an ArchiveInvalidation file for only the textures that could not be renamed.

This feature is now also included in the standalone BSA Patcher, so you don't have to create your own ArchiveInvalidation.txt file for those problem textures by hand.

Go get the update from the links above! NOTE: If you previously used 0.7.8 to patch the BSAs, you must restore them with 0.7.8 before installing 0.7.10 (or later). If you forget to do this and then replace your messed up BSAs with backups, you have to delete the BSAEdits file in your Oblivion\obmm folder before it will work correctly again.

OBMM USAGE EXAMPLE
The following example assumes you are using OBMM 0.7.10 along with Parallax Cities 1.2 (all three parts) and LBM Weathered Direction Signs 1.3.

After clicking Directly Edit BSAs and then Update Now, OBMM reports changing 821 entries in 3 BSA files and creates an ArchiveInvalidation.txt file containing only three texture entries:

CODE
textures/architecture/castleinterior/castleintcolumndetail03.dds
textures/architecture/skingrad/skingradstone06.dds
textures/architecture/bravil/woodplanks02.dds


These three entries are the only textures that could not be safely renamed in the BSAs, so they are invalidated here instead.

I've moved the usage example for old versions of BSA Patcher into a later post because it is rather long and involved and hopefully nobody is still using that old version. If you need it for some reason, look here.

MODDER'S RESOURCE
I have released a modder's resource containing sample ArchiveInvalidation instructions that mod creators can cut and paste into their readme files. You can find it here.

CREDITS
This information would not exist without the help of a lot of people who contributed feedback, usage guidelines, suggestions, test results, and moral support in this thread, including:

* DoomedMarauder
* Malifrax
* vkaryl
* Zanderat
* DragoonWraith
* Skyfire76
* phoenixamon
* Fyrstig
* Fyndolyn
* MaGISTraTe
* Ismelda Lasombra
* ThePenquin
* and many more. Thank you all!

Previous Thread

Thursday, May 18, 2006

ArchiveInvalidation Explained

Preamble
This is from my ESF thread: ArchiveInvalidation Explained (2). My previous ESF threads on this subject were: ArchiveInvalidation Explained and ArchiveInvalidation testing thread.

Last Updated: 7-5-2006 12:15 PM CDT
=============
NEWS FLASH: DELETE YOUR ARCHIVEINVALIDATION.txt file!
Working closely with the contributors to this thread, Timeslip has created a new invalidation method that is is vastly superior to standard invalidation methods.

Timeslip's Oblivion Mod Manager (0.7.10 or later) and BSA Patcher provide an archive invalidation method that will rename textures in your BSA files so you never have to worry about ArchiveInvalidation problems again.

This BSA alteration method of invalidation has been fully tested by the ArchiveInvalidation Explained Research Team (see credits below) and adoption of this method in the community is rapidly growing. It has proven extremely stable.

I've created a separate thread to try and get the word out about this new invalidation method because so many people have struggled with ArchiveInvalidation problems caused by using standard invalidation methods. The thread is ArchiveInvalidation Revisited.

Please help us to spread the word about this new solution. It is a vast improvement over the extremely buggy "standard" ArchiveInvalidation mechanism included with Oblivion.

Now back to our regularly scheduled programming, assuming anyone still cares ...
=============

ArchiveInvalidation Explained

OVERVIEW
There is a terrible beast wreaking mass destruction and chaos in the mod community. It is called ArchiveInvalidation. The purpose of our mission is to slay this horrid creature and prevent it from respawning.

But from whence did this beast arise? What spurred it forth from its dank lair and into our peaceful community? The answers lie within this document and its related discussion threads.

The problems with ArchiveInvalidation described here most often appear right now for people using more than one armor texture replacer, but it could crop up anywhere Oblivion uses common filenames like "boots.dds" for more than one set of things. If you have not experienced any problems yet, it is probably because your configuration does not meet these criteria.

The extensive testing carried out by multiple users and reported in these threads makes it fairly clear that Bethesda probably never intended for anyone to have to use ArchiveInvalidation in the first place. The vast majority of textures seem to load correctly from the Data folders even without using an ArchiveInvalidation file, which explains why so many users get duped into believing that a simple three-line global/universal/generic ArchiveInvalidation has solved all their problems when in reality it has been proven to have no effect whatsoever (it's the same as having no AI file at all).

In fact, we are proposing that the only reason it's necessary to use ArchiveInvalidation at all is due to a bug in the game that causes some texture replacements to fail to load, while others load just fine. Based largely on Malifrax's contributions, we've isolated these problems to certain common directory names and perhaps even certain NIF files (meshes) that appear to have a different structure than ones which load textures correctly.

Note that you don't need to read all of this if you are using OBMM 0.7.8 or later and you select the option to "Directly Edit BSAs". Timeslip's workaround avoids the need for ArchiveInvalidation entirely in our tests.

If, on the other hand, you do intend to use ArchiveInvalidation, or just want to understand what led to such drastic measures as Timeslip's solution, then you do need to read this.

Hopefully, Bethesda will issue a patch for the textures bug at some point and all of this will become an odd historical footnote in the life of Oblivion. Until then, read on ...

ARCHIVEINVALIDATION TEST RESULTS
1) ArchiveInvalidation.txt (AI.txt) is only needed for replacement textures. Also, it is most often required for replacement textures that are specified from within a mesh (NIF) rather than specified in the CS (icons and landscape textures, for example, will usually load without being referenced in AI.txt).

2) Universal/global/generic ArchiveInvalidation methods do not work. Period. End of discussion.

3) Mesh replacers load regardless of whether they are specified in your ArchiveInvalidation.txt (AI.txt), so you don't need to list any .NIF files. If you do list them, try to only list the specific meshes that are giving you problems, and be aware of the ramifications, as described in #4 and #5 below.

4) A single entry of a common filename like "boots.dds" in AI.txt will invalidate all paths containing a "boots.dds" file that exist in both the BSA and your Data folder. It will do this regardless of whether those paths in your Data folder actually contain a "boots.dds" file and regardless of whether you have an AI.txt entry for that path. *

5) Listing a path in your AI.txt has the same effect as listing individual textures, in the sense that it causes all the items in that path to be invalidated. However, once you've listed one such folder, Oblivion will automatically invalidate all further instances of any common filenames it found in that first folder. *

If all this sounds really confusing, you're not alone. It's not easy to understand, even for computer experts, in part because it's such a crazy way of doing things that it seems in many ways like a bug in the invalidation code Bethesda implemented. Perhaps it is a bug, perhaps it is by design, we have no way of knowing.

* Credit for this discovery goes to Wonder Dog. See the ArchiveInvalidation testing thread for details.

HOW DO I MAKE ONE?
Instructions can be found in the Oblivion Mods FAQ. It's not yet completely up to date with all this info, but it is a good starting place.

Once you get it started, you can list either just individual filenames, such as "boots.dds", one per line, or list entire directory paths, one per line, like this:
CODE
textures/armor/Iron/ \s
textures/armor/glass/ \s

In the end, I think discussions about which approach is better - listing files with no path versus listing only paths - probably won't get us anywhere, since both methods are highly imprecise and cause similar errors. Pick one and stick with it or use a mixture of approaches. I don't have enough data yet to make a recommendation one way or the other.

WHAT NOW?
This is a terrible situation any way you look at it. You can extract your textures BSA into your Data folder. That will solve most of the problems, but really it's a terrible solution. If you do this, be careful not to overwrite all the mod files you've already installed. You've been warned.

Probably the only halfway reasonable approach to even begin dealing with this problem is to have a tool like ACV, OBMM, or Wrye Bash check to make sure all replaced common filenames exist in all replaced paths and warn you when this isn't the case.

I think the best we can hope for is a combination of these approaches.

1) Modders creating texture replacers for things with common names in the BSA need to package all the related files with their mod, as many are already doing; if they don't, then their mods would be subject to warnings or blocks from utilities like OBMM and ACV.

This only applies if you are making a texture replacer that affects files with common filenames like "boots.dds", "cuirass.dds", etc. In this case, you must package all of the original textures that you did not change in the directory where the textures reside, such as "textures\armor\iron", along with the ones you did change. Otherwise, the ones you did not change in that directory path will be broken on the user's system.

We've seen this over and over again with mod-makers releasing a 1.0 version of a replacer that only includes their changed files, then (after getting deluged with hate-mail) having to follow up with a 1.1 version that includes all the items in that set regardless of whether they changed the files.

Another solution would be for modders to avoid creating texture-only replacers whenever possible. You can achieve the same result in many cases by altering the texture references in an item's mesh (using NifSkope) so it points to a new directory structure with texture files that don't use the same common filenames. Since meshes are loaded automatically without any need for entries in AI.txt, this avoids the problem entirely.

2) Utilities like OBMM might warn users before they install a mod that it may cause problems, assuming the user takes advantage of the omod format; then the user can decide whether they want to ignore the warning or stop the installation process.

3) Utilities like ACV might warn users after they have installed a mod that they are likely to have problems with certain files (like it already does for meshes); the user can then decide whether they want to ignore the warning or go back and uninstall the mod.

4) Utilities like Wrye Bash might provide some combination of #2 and #3, perhaps with more advanced users in mind.

TECHNICAL DETAILS
The ArchiveInvalidation problems described here are related to the way Oblivion invalidates entries in its BSA hash tables. In the Oblivion.ini, these are named:

- DirectoryStringTable
- FilenameStringTable

A "hash table" is a very common method of improving performance when you've got a lot of things to keep track of. A directory structure full of files is a perfect candidate for this technology.

Unfortunately, Oblivion's invalidation routines don't seem to be very accurate about how they mark a file as needing to be replaced. It appears there is a "disconnect" in the intersection between these two tables that causes a file invalidated in one path to also get invalidated in all other paths where it exists, resulting in index "collisions" that cause Oblivion to mistakenly think a texture should not be loaded from the BSA.

Here is my first attempt at a simplified explanation of how this works. Unfortunately, I just couldn't figure out a way to make the explanation any simpler than this. (I also couldn't get any colors to work, so we'll have to settle for underscores and strikethroughs instead.) If you have an idea on how to make this explanation any simpler or clearer, please post it or send me a PM.

The FilenameStringTable and DirectoryStringTable are initialized with all the textures in the BSA file, so all the paths and names of existing default textures get indexed in the hashes.

A small sample of these two indexes might look like this:

- FilenameStringTable= shield.dds, cuirass.dds, greaves.dds, greaves.dds, helm.dds
- DirectoryStringTable = textures/armor/glass/, textures/armor/iron/

where the underscored items reside in the BSA.

If I install one texture replacer for, say, all the Glass Armor, with AI.txt entries, then the hash entries for those textures get marked as being replaced by files in the Data folders. This means these items will now be invalidated, i.e. marked as being in Data the folders rather than the BSA, so now the hash index will look something like this:

- FilenameStringTable = shield.dds, cuirass.dds, greaves.dds, greaves.dds, helm.dds
- DirectoryStringTable = textures/armor/glass/, textures/armor/iron/

where the strikethrough items have been invalidated, but the underscored items still reside in the BSA.

If you're following any of this so far, you can probably already start to see where problems will arise, since all the common filenames shared between the two armor sets have now been invalidated from the FilenameStringTable simply because we wanted to replace them for the Glass armor set.

Now, if I want to also replace the texture for just the Iron cuirass, I'm going to run into problems because the hash table entries for all the common filenames in the Iron armor folder, including cuirass.dds, have already been invalidated (i.e., marked as existing in the Data folder rather than the BSA) and now the hash index will look something like this:

- FilenameStringTable = shield.dds, cuirass.dds, greaves.dds, greaves.dds, helm.dds
- DirectoryStringTable = textures/armor/glass/, textures/armor/iron/

where everything in both sets of armor has been invalidated.

Because the path "textures/armor/iron/" is now marked as existing in the Data folder, and the hash already knows that path is linked to a set of files that includes shield.dds, cuirass.dds, greaves.dds, gauntlet.dds, and helm.dds, then the hash will now expect to find all those same names in the Data folders even though I only wanted to replace the Iron cuirass.

In fact, the problem is even worse than this because Oblivion immediately recognizes the new directory path textures\armor\iron\ as soon as you add it to your Data folder. It doesn't even need to be listed in your AI.txt because the Oblivion automatically scans your Data folders and recognizes that directory path as being one from the BSA that has common filenames which have already been invalidated.

TOOLS
Tools for generating ArchiveInvalidation.txt, with suggested usage where possible.

ElChE's Oblivion Automatic Content Validator 1.1.0
ACV
- Use the Smart Check feature (in the latest version it automatically ignores meshes; in the previous version, have it open your AI when done, and then delete meshes by hand).

Oblivion Mod Manager (OBMM)
- Haven't tested the latest version, but Timeslip says he has added new AI.txt options in 0.7.6: "It lets you choose exactly which types of files to include, and there's options for ignoring files not in BSA's, normal maps, and files with incorrect extensions. Following MadBat's suggestion, it also deletes empty subdirectories from the data folder. Default settings are to only include textures, and only ones which already appear in a BSA archive. Hopefully, that should keep most people happy." Heck yeah! (In previous versions, use the DDS textures option.)

Wrye Bash also now includes a feature to generate AI.txt, but I haven't tested it yet.

FURTHER READING
Oblivion Mods FAQ

Original Thread: ArchiveInvalidation method tests.

Glossary
- AI.txt is shorthand for ArchiveInvalidation.txt.

Credits
Man of the Year award goes to Timeslip for solving the whole AI mess by avoiding it entirely!

Aside from that, there's a very big list of very talented and somewhat crazy people who put a lot of hard work into figuring out the AI mess. The list includes:
- DoomedMarauder
- Auctionmule
- Wonder Dog
- Scanti
- Malifrax
- Jarol
- Buerban
- Midnight Voyager
- Zanderat
- Martigen
- MadBat
- rdjeke
- DragoonWraith
- Oblivionmasta
- and many, many others ... Thank you all for this herioc effort!