Wednesday, November 15, 2006

The Oblivion Quest List

Now for my next act: The Oblivion Quest List (TOQL), which is intended to be a comprehensive guide to good quest mods that make it fun to keep playing Oblivion. Once again, this is a community project, in the sense that all of the mods listed here have been recommended by members of the mod community. Good quest mods are critical to the future of Oblivion modding, because without them people quickly get bored with plundering random dungeons, no matter how great the "eye candy" is.

Now you have a reason to keep staring at all those great textures listed in the TOTO project!

The Oblivion Quest List

Thursday, October 12, 2006

The Oblivion Texture Overhaul

I've launched a new project called The Oblivion Texture Overhaul (TOTO), which is a guide to all the great mods out there that improve the appearance of Oblivion. I'm hoping it will grow into an ongoing community project for information about texture replacers, including comparison screenshots, installation instructions, download mirror listings, etc.

Check it out!

The Oblivion Texture Overhaul

Sunday, September 24, 2006

mod site finished

I finally finished working on my new web site. Yay!

This blog site isn't going away. The two sites compliment each other pretty well, I think.

dev/akm mods

Saturday, September 23, 2006

Mod Site Under Construction

I'm finally getting around to creating a web site dedicated to my Oblivion mods (now that I actually have enough of them to make it worth doing a separate site).

It's still under construction, but it's looking pretty good so far:

dev/akm mods

Enjoy!

Wednesday, September 20, 2006

Mod: Qarl's Normal Maps Reduced

My latest mod is a texture replacement that acts as an unofficial option pack for Qarl's Texture Pack 2 (QTP2). QTP2 is an amazing high-res texture replacer for Oblivion, but the textures are so large that it can cause some stuttering on 256Mb video cards. So, I have attempted to improve performance of QTP2 on mid-range video cards without sacrificing the quality of his great textures. I do this by replacing all of the high-res normal maps from QTP2 with dramatically smaller normal maps. None of the color maps are altered. The normal maps are reduced to one quarter of their original size, which can significantly help reduce stuttering on 256Mb video cards with almost no loss of quality that you can see in-game.

Here's some screenshots of the reduced normal maps compared to Qarl's originals:

before
after
before
after
before
after
before
after


The difference is only really noticeable when you are very close up to some objects (you can see it if you look very closely at the shot of the large rock outside Chorral), but as soon as you get more than a few feet away there's no difference at all. Even when you are very close, some objects seem virtually unchanged, at least to my eye.

Where there were previously 300Mb worth of normal maps in QTP2, they are reduced here to only 75Mb uncompressed. In my limited testing, this resulted in a significant reduction in stuttering and increased performance (around +3-5 FPS, maybe more in heavily bogged areas) with almost no noticeable loss of in-game quality in the images.

This mod was created using Qarl's original normal maps and MrWiseman's excellent texture reduction script. MrWiseman's python script chews through texture subdirectories and invokes Nvidia's command-line DDS tools to strip off the top layer of each DDS file it finds, leaving the first mipmap as the new top layer.

All the credit for this work goes to Qarl for the textures and MrWiseman for the scripts. My contribution here was extremely limited.

You can download the mod from TESSource.

Tuesday, August 15, 2006

FAQ posted to TESSource

Looks like my FAQ will be getting even more readers.

TESSource, which hosts by far the largest database of Oblivion Mods, has opened a new Tutorials section, and I've posted a copy of my Oblivion Mods FAQ there.

DarkOne, who manages the site, has promised to "pin" the tutorial once he gets finished creating his new "pinning" system. This will greatly increase the chances for people to find the FAQ, which is great news.

Thanks, DarkOne!

Friday, August 11, 2006

An Old-Timer's Review of Oblivion

Comparisons to Morrowind
What the Expansions Need to Improve

Revised and updated, but still a work in progress, so please offer feedback.

I'm an old-time PnP (pen-and-paper) gamer, but I love CRPGs and video games in general, too. I didn't play Daggerfall, but that's probably because I was too hooked on Ultima Online and Diablo. I loved video games like Windwaker, Halo, Prince of Persia, KOTOR, Burnout, Fable, Jade Empire, Half-Life/2, etc., etc.

Morrowind stands tall near the top of my list of favorite games, right up there with the original white box Dungeons & Dragons, perhaps even equal to Squad Leader, Melee/Wizards, AD&D, Car Wars, Hero, GURPS, and then computer games like the original Wolfenstien 3D, Doom, Command & Conquer, Quake, Warcraft, and Planescape: Torment, as well as the video games mentioned above.

Oblivion is the newest entry to my list of greats.

You see, all this fuss about preserving "true RPG" elements and such don't mean much to me. I like action, I like strategy, I like role-playing. CRPGs in general seem to fall into one of two main camps: they either force you into a certain series of "role" choices (ala KOTOR) or they try to stay out of your way and let you invent your own rational for what your character does. Morrowind mostly stuck with the latter approach, relying on you the player to put as little or as much thought into it as you like. In stark contrast to this, KOTOR constantly rams good-versus-evil choices down your throat. I enjoy both types of games, so I definitely don't have a problem with Oblivion's minimalist approach and I don't think it's very different from Morrowind in this respect.

The biggest difference between Oblivion and Morrowind (and the main thing most people seem to take issue with, although they will probably cite other more specific reasons) seems to be that Oblivion is more than just a CRPG. You can just go out and bash stuff and have fun doing it if you want. This wasn't true in Morrowind. For some reason, a lot of die-hard RPG players have a problem with this aspect of Oblivion, as if somehow all the emphasis on action detracts from their ability to role-play. While it is true that you're probably not going to take time to ponder your motivations during the heat of battle, there are plenty of opportunities to seek out quiet moments in a peaceful meadow or deep forest where you can ponder to your heart's content.

For me, Oblivion does exactly what I wanted it to do. It answers almost all of the problems Morrowind had and pushes the envelope for games even further. However, as with any ground-breaking effort, it has some rough spots that need to be polished.

Here are some specifics about the good and the bad, in no particular order:

* The combat rocks! It is vastly more fun, both spellcasting and bashing. I still miss all the time, but it's my fault when I do and not because of a dice roll. A bit like Halo with swords and spells. Combat is now a major part of what makes the game fun, which is a dramatic departure from Morrowind. Perfect!

* Animations. Awesome! Vast improvement over Morrowind here! Surprisingly clever idle animations that sneak up on you when you're not expecting it. Drinking, sleeping, praying, saluting, and more! The only major oversight here is the lack of any built-in way to make your character use these animations, too! There's also a problem with the jump animation not firing consistently, but that is addressed under the Mounts topic below.

* Graphics. Breathtaking. When you've been in a dungeon or an Oblivion gate for a while and you finally emerge into a sunset or sunrise, it will take your breath away. It is such a beautifull, vibrant land that you will develop an emotional attachment to it for this reason alone. If you have to run this game with minimum settings, you are really missing out! No elaboration necessary.

* The RAI is simply awesome. The NPCs seem dramatically more interesting when they go about their daily tasks and the possibilities are almost endless. Badguys will chase you down if they can! The only really major problems are that NPCs in combat don't use their healing potions and seem oblivious to the danger posed by molten lava and other environmental hazards (like traps).

* Fast travel is perfect. Morrowind had fast travel anyway, and anyone who tries to tell you differently is just full of it.

* The compass is great. No more endlessly wasting time hunting for things that my character should be able to find. It is overused, though. Once I find the cave where my target is, do I really still need a pointer guiding me to the exact spot in the cave? No, I can easily explore the cave on my own, so it's just silly in many cases and ruins the illusion of surprise. It should be used more sparingly in the expansions, or at least made optional for those who dislike it. I think it would be perfect if it vanished once you got within a certain radius of the target.

* Level-Scaling is great. I can find plenty of challenge even at a high level. Huge improvement over Morrowind. However, it needs refinement (ala Ken Rolston's statements). Not everything should scale infinitely. Some battles should get easier as you become more powerful. Common bandits shouldn't wear Daedric and Glass armor. Some things should be rare and hard to find. It should be possible to find some powerful things early on with luck and daring. Poor scaling of NPC bosses in comparison to creatures badly needs fixing, particularly in relation to the difficulty slider (if you crank up the difficulty enough to make NPC bosses challenging at a high level, then creatures become impossible to beat). NPC boss scaling in general needs to be addressed. See Oscuro's Oblivion Overhaul mod for a good atttempt at fixing this. What's really needed is the capability to set the scaling level offset for an NPC to a percentage above or below the player rather than a fixed number of levels above or below. Also, traps need much more aggressive scaling. I've expanded on this topic near the end of the review.

* Good-versus-Evil Choices. Huge improvement over Morrowind here. Being a thief really gets you in trouble, the Dark Brotherhood really is evil (not like those pansy Morag Tong in Morrowind), the Daedra are perverted, etc. Still some minor gameplay tuning needed here, though, especially with stolen items and houses owned by criminals you've killed honorably. The only thing really missing here compared to Morrowind is the option to complete a quest without violence (probably because combat is so much more important in Oblivion). It would be really nice to see more branching storylines based on your choices (like KOTOR).

* Levitation. I do miss it (how could you not!), but in practice Oblivion is more challenging without it, so this was a good call. Some sort of climbing system would be a huge improvement, though. Horses help to offset this loss.

* Spoken Dialog. It's awesome, but skip the celebrities next time and just hire more actors, or hire actors who can play multiple voices (for a good example, listen to the audio book "Eragon"). At a minimum, voice actors who don't have a lot of range shouldn't be used for more than one race. There was certainly a lot more dialog in Morrowind (since it didn't require voice acting), but the amount of unique dialog for each character was arguably almost as limited as it is in Oblivion. Also, in Morrowind, any unique dialog was usually buried under such a vast heap of repeated junk that it might as well have not existed at all. Oblivion takes exactly the opposite approach to dialog, which sometimes makes it feel like there isn't much of it. On the other hand, you won't have to waste any time trying to figure out whether an NPC has something to say or not. The spoken dialog in the Thieves Guild quest has major problems, though, which is really disturbing.

* Stealth. Awesome. Huge improvement over Morrowind. Needs some refinement in the way it interacts with RAI, but still awesome. The major problems seem to happen only at a very high stealth. NPCs you've attacked tend to just stand there getting slaughtered instead of running for help. Also, NPCs who manage to escape a surprise attack should be able to alert their allies to your presence and start hunting for you.

* Mounts. The horses are very, very cool. Mounted combat definitely needs to be added in an expansion, along with more options for controlling your horse. The only really annoying problem is the way the "jump delay" system works. If you set the jump delay down low enough to avoid problems where the player jump animation fails to fire, then your horse starts making really annoying jumps over small dips in the terrain. Charging down an incline may be extremely hazardous on horseback in real life, but in a fantasy game it seems completely out of place. Since Cyrodiil is mostly steep hills and mountains, you'd think that their horses would be a bit more surefooted!

* Hotkeys. Big mistake here. Each hotkey should cycle through a list of items you set rather than only having one thing per key (i.e., several swords on one key, several healing spells on one key, several destruction spells on another key, etc.). Also, you need to be able to hotkey potions without accidentally using them (especially on a gamepad), using the same select-and-then-fire system used for everything else. Since action-oriented combat is a huge part of what makes Oblivion great, this oversight is severe and must be fixed at the earliest opportunity.

* Mods. Dramatic improvements in the mod-handling system greatly reduce savegame corruption and mod-related crashes, but these new protections create some limitations that are hard to get used to. There are major issues that still need to be resolved with the Construction Set, but great mods are being released daily despite this hinderance. In many ways, these limitations have driven the mod community to release much better tools than were ever created for Morrowind. The sheer number of mods already released is staggering. Even if most of them are crap, there's still more than 4,000 mods already, and many of them are already very good. The patch 1.1 failed to correct a major texture-loading bug for replacers, but the mod community has managed to work around this issue. Bethesda created a really stupid PR nightmare with the mod community by failing to release a NIF exporter at the same time that it launched it's own efforts to charge money for official mods. As long as the exporter does get released and the CS bugs get fixed, everyone will forget these complaints in short order.

* Bugs in general. Arguably better than Morrowind, but maybe not by much.

* Houses. This point is less significant than the rest. The buyable houses in Oblivion are great, but they are a poor substitute for Morrowind's quests to build your stronghold. Sure you can accumulate a lot of very nice houses in Oblivion, but the sense of accomplishment is not the same, especially when there are only two NPCs who will join you in one of your houses (and one of those is the really annoying Devoted Fan). The stronghold theme from Morrowind should've been expanded on rather than removed, especially since RAI makes it possible to have NPCs actually doing useful stuff for your stronghold. Also, as mentioned earlier, it should be possible to acquire some houses through quests, such as the graverobber quest in Imperial City (which would also solve the problem of not having a buyable house there).

* Quests. The main quest in Oblivion is awe-inspiring. It's a really great story with all the same wit, subtlety, and sacreligious elements that made Morrowind (and the Tribunal expansion) great. The quality and depth of storytelling here puts almost every other CRPG to shame. The gameplay is intense and very challenging. Too bad it's so short. The Fighters Guild and Mages Guild quests are good, too, but disappointing in comparison -- the writing is good, but the gameplay suffers from significant balance issues and the NPC bosses you're up against are way too easy to kill once your character reaches a medium or high level. I can't comment on the Thieves Guild or Dark Brotherhood quests since I haven't gone very far into either of these yet. Some of the Miscellaneous Quests are quite good: the Vampire Hunters mini-guild stands out as a really good take on the typical Morrowind search/gather quest model.

* Havok. The physics in the game are a lot of fun and do add significantly to the overall experience. Be careful not to drool on yourself in shock and amazement the first time (heck, every time) you kill a Storm Atronach, especially if you're standing on a steep hill. The major complaint about Havok has to do with it making it so difficult to decorate your house(s) with the spoils of your adventures. This was a major element of fun in Morrowind, so I was really disappointed the first dozen times I tried to arrange my house in Oblivion and then acccidentally knocked everything back onto the floor, ruining all my hard work. After a while, though, I slowly started to enjoy it, making it into a bit of a game to see what I could get to work and what would just collapse. I've come to grudgingly accept the system and enjoy some aspects of it, although I still think it's way too jumpy. The other major disappointments are the lack of useful common objects (you can't use random objects to hit an NPC or cause a distraction to lure enemies away) and the fact that many of the traps fire way too soon and/or don't do much damage. I also find it really strange that Bethesda chose not to take more advantage of Havok than they did. It really would've made the game a lot better if Bethesda had spent some time adding physics-based puzzles like those in Half-Life/2.

* Quest Pop-Ups. Like the compass, this is an extremely controversial subject. The pop-ups can be a major spoiler for the game because they provide extensive hand-holding that ruins the challenge and sense of immersion for many players. Basically, it's like having a built-in walkthrough. I tried to ignore them as much as possible because they give away too much. The pop-ups definitely need to be made optional in the expansions so players who don't like them can turn them off.

* Load times. On a fast PC with a lot of RAM and some tweaking, the load times are not bad in general. The HUGE improvement over Morrowind is complete removal of reload delays after death. This improvement is such a big deal that it's almost worth the price of admission by itself.

Expanded Discussion of Scaling Issues

There are several distinct issues with scaling that Bethesda should have fixed, and probably would have fixed if they hadn't been in such a rush to release the game during the Xbox-360 "launch window".

Here's a bit more detail on the scaling issues.

* Loot scaling. Although loot scaling is generally a good idea, Bethesda took it too far in Oblivion, causing some major immersion problems as your character levels up. These problems are relatively easy to correct using the built-in leveled-lists mechanism, and Bethesda definitely needs to do so in the expansions. You should be able to find some powerful items even at a low level with some luck and daring. Common bandits shouldn't all suddenly start wearing glass armor when you get over level 20. Etcetera.

* Creature scaling. This is fairly well balanced in vanilla Oblivion because creatures do not rely on the strength of items they carry nearly as much as NPCs do (if at all). However, the fact that you stop seeing ANY weak creatures once you reach a high level and you never encounter strong creatures at a low level is a major problem. This means you never have to run from a fight and you never have an easy fight either, which tends to make the difficulty of any given encounter too predictable.

* NPC scaling. This has major problems in Oblivion, despite mods that attempt to correct it and despite being a huge improvement over Morrowind (where NPCs had no scaling at all). This problem is particularly evident with NPC bosses once the player reaches a high level. The problem simply boils down to the fact that "level offset", which controls how much weaker or stronger an NPC will be compared to the player, is a literal number of levels rather than a percentage. For example, when a level-5 player encounters a boss who is 5 levels higher than the player, the result will generally be a very hard fight (the boss is twice your level and probably has similar magic items). However, when a level-20 player encounters that same boss, the result will be a MUCH easier fight for the player (since the boss will only be level 25 and will have much weaker magic items). The only real way to correct this is for Bethesda to alter the Oblivion engine so that the level offset can be a percentage above or below the player rather than a fixed number of levels above or below.

* Scaled allies. This is a huge problem, especially in quests like "The Battle for Kvatch", where your NPC allies get trounced by the creatures they are up against no matter what you do. I suspect this problem is mostly a result of how imbalanced NPC scaling is versus Creature scaling, but it may also be caused by sloppy quest design. If NPC scaling were corrected, the problem would likely be much less severe.

Oscuro's Oblivion Overhaul does a good job of correcting the scaling issues, at least to the extent that it is possible to correct without a change to the game engine (i.e., NPC scaling offset).

If the expansions can tackle even some of these issues, especially the critical ones, Oblivion may yet surpass Morrowind on my list of favorite games of all time. And it will be even more successful than it is already.

For commentary, feedback, and points I failed to cover here, please see the original thread on the ESF boards and the second thread on the ESF boards.

Friday, August 04, 2006

Elder Scrolls Forum pins FAQ

The good news just keeps rolling in!

The moderators on the official Elder Scrolls Forums have pinned the ESF version of my Oblivion Mods FAQ. This means it will always stay at the top of the Oblivion Mods thread list and will continue to be available for new players to read. This should make it a LOT easier for new fans to get started with player-made mods for Oblivion and help to reduce the number of newbie questions posted in the official Oblivion Mods forum.

:)

Wednesday, August 02, 2006

PlanetElderScrolls Adds Links to Oblivion Mods FAQ

Good news update!

APY, over at PlanetElderScrolls, has added a link from his Installing Oblivion Mods intro to my FAQ. I think this will introduce a whole new group of readers to the Oblivion Mods FAQ.

Thanks, APY!

Wednesday, July 05, 2006

John Romero is hurting the game industry

John Romero, co-founder of id Software (Doom, Quake, etc.) is seriously out of touch. He has posted a series of blog entries claiming, and then retracting the claim, that game modders are hurting the industry. His comments were prompted by the ESRB's change in its rating for Oblivion from Teen to Mature.

This is old news, but it is worth mentioning even now because:

1. Romero's comments are wildly inaccurate. He claims the ESRB changed the rating for Oblivion because "some modder" added nudity to the game. The ESRB did mention this fact in its statement, but it is NOT the real reason they changed the rating, and Romero fails to address the main reason at all.

2. The press repeated Romero's statements, further spreading the misinformation and giving it a lot more validity than it deserves.

In fact, the nudity mod issue is a red herring.

It's just a convenient excuse for the ESRB to correct a mistake in the way they rated Oblivion to begin with.

I think the ESRB ratings are stupid for many reasons, but we live in a Puritan society, so it's to be expected I guess. In any case, ESRB ratings are fairly similar to the way movies are rated, so we should all be used to it by now.

The closest analogy would be a movie in which nude scenes were cut before release to gain a PG-13 rating. If some fan got ahold of the cut footage, spliced it back together and released this "unofficial" version of the movie, would the ratings board then go back and change the rating to R? No, they'd consider such a suggestion absurd, and yet that's almost exactly what we're seeing here, assuming the "hidden content/breasts" argument is valid at all.

In reality, Oblivion probably deserved an M rating for many other reasons, some obvious and some less obvious.

The obvious reason is the very large number of mutilated, bloody corpses in the Oblivion gates. Along with the various corpse mashers, etc., the game really is quite grotesque in places. I don't mind this at all and I think it is appropriate for the story, but if I'm in a spot like that in the game I do pause it if my daughter comes into the room because I think it would give her nightmares.

The more subtle reason is the storyline. It's a very mature story containing a lot of "sacreligious" themes, much like Morrowind did. To me, this is one of the best aspects of the game. However, most people would classify this sort of thing (assuming they're smart enough to recognize it at all) as a "mature theme".

The ESRB just used the nudity issue as a convenient cover story for the fact that they failed to rate Oblivion correctly, and Romero should've recognized this as the obvious subterfuge it is.

Romero should either put more thought into his commentary or stop posting such bad information.

Saturday, July 01, 2006

Current Release Threads

These are the curent release threads I'm actively supporting on the Elder Scrolls Forums.

Qarl's Texture Pack II
Basically I'm just handling support for Qarl's mod on the ESF forums and a few other places (at his request).

Skingrad Roof Textures
This is an unofficial add-on that I made for Qarl's Texture Pack II.

README ArchiveInvalidation Template
This is a modder's resource I created for people developing texture replacers.

Other threads where I can frequently be found lurking include:

Oblivion Mods FAQ
This is the ESF thread for my FAQ.

ArchiveInvalidation Destroyed
I frequently answer questions on almost anything related to ArchiveInvalidation problems.

I'm also active on the CanadianIce Oblivion forums.

Wednesday, May 24, 2006

ArchiveInvalidation Destroyed

Previous Thread

SOLVE YOUR ARCHIVEINVALIDATION PROBLEMS FOR GOOD!
Updated 7-1-2006 11:15 AM CDT

Timeslip's Oblivion Mod Manager (0.7.10 or later) and BSA Patcher provide an archive invalidation method that will rename textures in your BSA files so you never have to worry about ArchiveInvalidation problems again. This method is vastly superior to standard invalidation methods.

This BSA alteration method of invalidation has been tested by the ArchiveInvalidation Explained Research Team and adoption of this method in the community is rapidly growing. It has proven extremely stable. You should probably make a backup of your BSA files if you have the space (or a DVD-burner) just in case, but we haven't seen any corruption issues.

OBMM and BSA Patcher will keep track of which files have been renamed in your BSAs and provide a "Remove BSA edits" function to rollback any changes they have made to your BSA.

Assuming you have OBMM 0.7.10 or later already, and have already installed/activated some mods, the detailed steps for doing this in OBMM are as follows:

1. Start OBMM;
2. Click Utilities;
3. Select Archive invalidation;
4. Click Directly Edit BSAs;
5. Click the checkbox for autoupdate on exit and/or click Update Now;
6. Click the red X in the upper-left corner of the Archive invalidation popup to close it;
7. Click Launch Oblivion
8. Play!

If you didn't check the box for autoupdate on exit, then repeat this process after installing/activating each new mod.

The latest version of OBMM (0.7.11 or later) provides a lot of new options for the Directly Edit BSAs function. The default options may be okay, but you need to at least make sure to check the box for Generate archiveinvalidation entries on hash collision, since that setting is critical.

Here are the ideal OBMM BSA Settings (for 0.7.11 or later).

Using those settings will take care of everything for you.

As already noted, it's completely up to you whether you want to use autoupdate on exit or Update Now, or both. I find that when running with a lot of texture replacements installed, the autoupdate on exit option tends to cause an annoying delay when quitting OBMM or using the Launch Oblivion button. Since I've gotten into the habit of using the Launch Oblivion button instead of the normal launcher, and I don't install new mods nearly as often as I start Oblivion, I decided to avoid using autoupdate on exit and instead always use Update Now after I install a new mod.

If you later uninstall/deactivate some mods, you'll need to use the Remove BSA edits option. Follow these steps:

1. Start OBMM;
2. Click Utilities;
3. Select Archive invalidation;
4. Click Remove BSA edits;
5. Close OBMM;
6. Delete texture files that you want removed;
7. Start OBMM;
8. Click Utilities;
9. Select Archive invalidation;
10. Click Directly Edit BSAs;
11. You know the rest by now smile.gif

If you're using .omods, you should skip Steps 5-7 and just Deactivate the .omod instead.

New, Improved Version of BSA Patcher
BSA Patcher is now dramatically easier to use.

Download BSA Patcher

Unlike OBMM, BSA Patcher is a command-line application, although you can launch it by double-clicking the executable. BSA Patcher doesn't include as many configuration options as OBMM, but it accomplishes the same tasks and it only requires .NET 1.1, which means it will work for anyone who is unwilling or unable to run .NET 2.0. It will also work with mono, an open-source version of .NET 1.1.

The latest version of BSA Patcher (as of this writing) uses the BSA alteration code from OBMM 0.7.10 and, like OBMM, it will automatically generate an ArchiveInvalidation.txt file for you containing only the textures that could not be safely renamed in your BSA files. It includes support for several command-line options, but you can safely ignore them.

To use it, place BSAPatch.exe in your \Oblivion\Data\ folder and double-click it once to rename files in your BSA. Run it again to restore them to their original names.

NOTE: If you use either OBMM or BSA Patcher, you should not manually alter your ArchiveInvalidation.txt file. Let one of these programs do it for you. No other solutions are recommended.

Older Versions of BSA Patcher
If you have an older version of BSA Patcher, make sure you have at least the version based on code from OBMM 0.7.9. You should probably still make a backup of your BSA files before running it, just in case. Run it once to rename files in your BSA and run it again to restore them to their original names.

If you're not sure what all this is about, don't have any mods installed yet, or just need more information on how to use mods in Oblivion, please read the Oblivion Mods FAQ for more general instructions. There's also another version of the FAQ on the CS Wiki, which is a bit easier to read because it has a working table of contents.

For more information on ArchiveInvalidation, read my blogged version of ArchiveInvalidation Explained (or check to see if the original ArchiveInvalidation Explained thread still exists). The results of our testing are summarized neatly in the first post.

If you have questions about all this, post them here and members of our team will attempt to help you. You may also want to check the OBMM thread for updates.

DETAILS
Oblivion has major issues loading some replacement textures, even with an ArchiveInvalidation file. What OBMM and BSAPatch do is to sidestep this problem by making Oblivion think it never had a copy of the file you are replacing.

This feature edits the BSA archives so that oblivion cannot find the original version of files you have replaced, thus forcing it to search elsewhere for them.

Here's the summary from Timeslip's site: "OBMM will scan each of the BSA's, and check for the existence in the data directory of each file within the BSA. If a match is found, the hash of that file is incremented. Oblivion never reads file names, and relies solely on the hashes. As a result, it never notices the filename/hash mismatch, and never sees anything wrong. As far as the player is concerned, mods should suddenly start working correctly, entirely without the need for an archiveinvalidation file."

UPDATES
The 0.7.10 and later version of OBMM correct an issue from previous versions where several files with very similar names in the same directory (like castleintcolumndetail02.dds, castleintcolumndetail03.dds, and castleintcolumndetail04.dds) might get their textures mixed up if a replacer only replaces one of these files. OBMM now generates an ArchiveInvalidation file for only the textures that could not be renamed.

This feature is now also included in the standalone BSA Patcher, so you don't have to create your own ArchiveInvalidation.txt file for those problem textures by hand.

Go get the update from the links above! NOTE: If you previously used 0.7.8 to patch the BSAs, you must restore them with 0.7.8 before installing 0.7.10 (or later). If you forget to do this and then replace your messed up BSAs with backups, you have to delete the BSAEdits file in your Oblivion\obmm folder before it will work correctly again.

OBMM USAGE EXAMPLE
The following example assumes you are using OBMM 0.7.10 along with Parallax Cities 1.2 (all three parts) and LBM Weathered Direction Signs 1.3.

After clicking Directly Edit BSAs and then Update Now, OBMM reports changing 821 entries in 3 BSA files and creates an ArchiveInvalidation.txt file containing only three texture entries:

CODE
textures/architecture/castleinterior/castleintcolumndetail03.dds
textures/architecture/skingrad/skingradstone06.dds
textures/architecture/bravil/woodplanks02.dds


These three entries are the only textures that could not be safely renamed in the BSAs, so they are invalidated here instead.

I've moved the usage example for old versions of BSA Patcher into a later post because it is rather long and involved and hopefully nobody is still using that old version. If you need it for some reason, look here.

MODDER'S RESOURCE
I have released a modder's resource containing sample ArchiveInvalidation instructions that mod creators can cut and paste into their readme files. You can find it here.

CREDITS
This information would not exist without the help of a lot of people who contributed feedback, usage guidelines, suggestions, test results, and moral support in this thread, including:

* DoomedMarauder
* Malifrax
* vkaryl
* Zanderat
* DragoonWraith
* Skyfire76
* phoenixamon
* Fyrstig
* Fyndolyn
* MaGISTraTe
* Ismelda Lasombra
* ThePenquin
* and many more. Thank you all!

Previous Thread

Thursday, May 18, 2006

ArchiveInvalidation Explained

Preamble
This is from my ESF thread: ArchiveInvalidation Explained (2). My previous ESF threads on this subject were: ArchiveInvalidation Explained and ArchiveInvalidation testing thread.

Last Updated: 7-5-2006 12:15 PM CDT
=============
NEWS FLASH: DELETE YOUR ARCHIVEINVALIDATION.txt file!
Working closely with the contributors to this thread, Timeslip has created a new invalidation method that is is vastly superior to standard invalidation methods.

Timeslip's Oblivion Mod Manager (0.7.10 or later) and BSA Patcher provide an archive invalidation method that will rename textures in your BSA files so you never have to worry about ArchiveInvalidation problems again.

This BSA alteration method of invalidation has been fully tested by the ArchiveInvalidation Explained Research Team (see credits below) and adoption of this method in the community is rapidly growing. It has proven extremely stable.

I've created a separate thread to try and get the word out about this new invalidation method because so many people have struggled with ArchiveInvalidation problems caused by using standard invalidation methods. The thread is ArchiveInvalidation Revisited.

Please help us to spread the word about this new solution. It is a vast improvement over the extremely buggy "standard" ArchiveInvalidation mechanism included with Oblivion.

Now back to our regularly scheduled programming, assuming anyone still cares ...
=============

ArchiveInvalidation Explained

OVERVIEW
There is a terrible beast wreaking mass destruction and chaos in the mod community. It is called ArchiveInvalidation. The purpose of our mission is to slay this horrid creature and prevent it from respawning.

But from whence did this beast arise? What spurred it forth from its dank lair and into our peaceful community? The answers lie within this document and its related discussion threads.

The problems with ArchiveInvalidation described here most often appear right now for people using more than one armor texture replacer, but it could crop up anywhere Oblivion uses common filenames like "boots.dds" for more than one set of things. If you have not experienced any problems yet, it is probably because your configuration does not meet these criteria.

The extensive testing carried out by multiple users and reported in these threads makes it fairly clear that Bethesda probably never intended for anyone to have to use ArchiveInvalidation in the first place. The vast majority of textures seem to load correctly from the Data folders even without using an ArchiveInvalidation file, which explains why so many users get duped into believing that a simple three-line global/universal/generic ArchiveInvalidation has solved all their problems when in reality it has been proven to have no effect whatsoever (it's the same as having no AI file at all).

In fact, we are proposing that the only reason it's necessary to use ArchiveInvalidation at all is due to a bug in the game that causes some texture replacements to fail to load, while others load just fine. Based largely on Malifrax's contributions, we've isolated these problems to certain common directory names and perhaps even certain NIF files (meshes) that appear to have a different structure than ones which load textures correctly.

Note that you don't need to read all of this if you are using OBMM 0.7.8 or later and you select the option to "Directly Edit BSAs". Timeslip's workaround avoids the need for ArchiveInvalidation entirely in our tests.

If, on the other hand, you do intend to use ArchiveInvalidation, or just want to understand what led to such drastic measures as Timeslip's solution, then you do need to read this.

Hopefully, Bethesda will issue a patch for the textures bug at some point and all of this will become an odd historical footnote in the life of Oblivion. Until then, read on ...

ARCHIVEINVALIDATION TEST RESULTS
1) ArchiveInvalidation.txt (AI.txt) is only needed for replacement textures. Also, it is most often required for replacement textures that are specified from within a mesh (NIF) rather than specified in the CS (icons and landscape textures, for example, will usually load without being referenced in AI.txt).

2) Universal/global/generic ArchiveInvalidation methods do not work. Period. End of discussion.

3) Mesh replacers load regardless of whether they are specified in your ArchiveInvalidation.txt (AI.txt), so you don't need to list any .NIF files. If you do list them, try to only list the specific meshes that are giving you problems, and be aware of the ramifications, as described in #4 and #5 below.

4) A single entry of a common filename like "boots.dds" in AI.txt will invalidate all paths containing a "boots.dds" file that exist in both the BSA and your Data folder. It will do this regardless of whether those paths in your Data folder actually contain a "boots.dds" file and regardless of whether you have an AI.txt entry for that path. *

5) Listing a path in your AI.txt has the same effect as listing individual textures, in the sense that it causes all the items in that path to be invalidated. However, once you've listed one such folder, Oblivion will automatically invalidate all further instances of any common filenames it found in that first folder. *

If all this sounds really confusing, you're not alone. It's not easy to understand, even for computer experts, in part because it's such a crazy way of doing things that it seems in many ways like a bug in the invalidation code Bethesda implemented. Perhaps it is a bug, perhaps it is by design, we have no way of knowing.

* Credit for this discovery goes to Wonder Dog. See the ArchiveInvalidation testing thread for details.

HOW DO I MAKE ONE?
Instructions can be found in the Oblivion Mods FAQ. It's not yet completely up to date with all this info, but it is a good starting place.

Once you get it started, you can list either just individual filenames, such as "boots.dds", one per line, or list entire directory paths, one per line, like this:
CODE
textures/armor/Iron/ \s
textures/armor/glass/ \s

In the end, I think discussions about which approach is better - listing files with no path versus listing only paths - probably won't get us anywhere, since both methods are highly imprecise and cause similar errors. Pick one and stick with it or use a mixture of approaches. I don't have enough data yet to make a recommendation one way or the other.

WHAT NOW?
This is a terrible situation any way you look at it. You can extract your textures BSA into your Data folder. That will solve most of the problems, but really it's a terrible solution. If you do this, be careful not to overwrite all the mod files you've already installed. You've been warned.

Probably the only halfway reasonable approach to even begin dealing with this problem is to have a tool like ACV, OBMM, or Wrye Bash check to make sure all replaced common filenames exist in all replaced paths and warn you when this isn't the case.

I think the best we can hope for is a combination of these approaches.

1) Modders creating texture replacers for things with common names in the BSA need to package all the related files with their mod, as many are already doing; if they don't, then their mods would be subject to warnings or blocks from utilities like OBMM and ACV.

This only applies if you are making a texture replacer that affects files with common filenames like "boots.dds", "cuirass.dds", etc. In this case, you must package all of the original textures that you did not change in the directory where the textures reside, such as "textures\armor\iron", along with the ones you did change. Otherwise, the ones you did not change in that directory path will be broken on the user's system.

We've seen this over and over again with mod-makers releasing a 1.0 version of a replacer that only includes their changed files, then (after getting deluged with hate-mail) having to follow up with a 1.1 version that includes all the items in that set regardless of whether they changed the files.

Another solution would be for modders to avoid creating texture-only replacers whenever possible. You can achieve the same result in many cases by altering the texture references in an item's mesh (using NifSkope) so it points to a new directory structure with texture files that don't use the same common filenames. Since meshes are loaded automatically without any need for entries in AI.txt, this avoids the problem entirely.

2) Utilities like OBMM might warn users before they install a mod that it may cause problems, assuming the user takes advantage of the omod format; then the user can decide whether they want to ignore the warning or stop the installation process.

3) Utilities like ACV might warn users after they have installed a mod that they are likely to have problems with certain files (like it already does for meshes); the user can then decide whether they want to ignore the warning or go back and uninstall the mod.

4) Utilities like Wrye Bash might provide some combination of #2 and #3, perhaps with more advanced users in mind.

TECHNICAL DETAILS
The ArchiveInvalidation problems described here are related to the way Oblivion invalidates entries in its BSA hash tables. In the Oblivion.ini, these are named:

- DirectoryStringTable
- FilenameStringTable

A "hash table" is a very common method of improving performance when you've got a lot of things to keep track of. A directory structure full of files is a perfect candidate for this technology.

Unfortunately, Oblivion's invalidation routines don't seem to be very accurate about how they mark a file as needing to be replaced. It appears there is a "disconnect" in the intersection between these two tables that causes a file invalidated in one path to also get invalidated in all other paths where it exists, resulting in index "collisions" that cause Oblivion to mistakenly think a texture should not be loaded from the BSA.

Here is my first attempt at a simplified explanation of how this works. Unfortunately, I just couldn't figure out a way to make the explanation any simpler than this. (I also couldn't get any colors to work, so we'll have to settle for underscores and strikethroughs instead.) If you have an idea on how to make this explanation any simpler or clearer, please post it or send me a PM.

The FilenameStringTable and DirectoryStringTable are initialized with all the textures in the BSA file, so all the paths and names of existing default textures get indexed in the hashes.

A small sample of these two indexes might look like this:

- FilenameStringTable= shield.dds, cuirass.dds, greaves.dds, greaves.dds, helm.dds
- DirectoryStringTable = textures/armor/glass/, textures/armor/iron/

where the underscored items reside in the BSA.

If I install one texture replacer for, say, all the Glass Armor, with AI.txt entries, then the hash entries for those textures get marked as being replaced by files in the Data folders. This means these items will now be invalidated, i.e. marked as being in Data the folders rather than the BSA, so now the hash index will look something like this:

- FilenameStringTable = shield.dds, cuirass.dds, greaves.dds, greaves.dds, helm.dds
- DirectoryStringTable = textures/armor/glass/, textures/armor/iron/

where the strikethrough items have been invalidated, but the underscored items still reside in the BSA.

If you're following any of this so far, you can probably already start to see where problems will arise, since all the common filenames shared between the two armor sets have now been invalidated from the FilenameStringTable simply because we wanted to replace them for the Glass armor set.

Now, if I want to also replace the texture for just the Iron cuirass, I'm going to run into problems because the hash table entries for all the common filenames in the Iron armor folder, including cuirass.dds, have already been invalidated (i.e., marked as existing in the Data folder rather than the BSA) and now the hash index will look something like this:

- FilenameStringTable = shield.dds, cuirass.dds, greaves.dds, greaves.dds, helm.dds
- DirectoryStringTable = textures/armor/glass/, textures/armor/iron/

where everything in both sets of armor has been invalidated.

Because the path "textures/armor/iron/" is now marked as existing in the Data folder, and the hash already knows that path is linked to a set of files that includes shield.dds, cuirass.dds, greaves.dds, gauntlet.dds, and helm.dds, then the hash will now expect to find all those same names in the Data folders even though I only wanted to replace the Iron cuirass.

In fact, the problem is even worse than this because Oblivion immediately recognizes the new directory path textures\armor\iron\ as soon as you add it to your Data folder. It doesn't even need to be listed in your AI.txt because the Oblivion automatically scans your Data folders and recognizes that directory path as being one from the BSA that has common filenames which have already been invalidated.

TOOLS
Tools for generating ArchiveInvalidation.txt, with suggested usage where possible.

ElChE's Oblivion Automatic Content Validator 1.1.0
ACV
- Use the Smart Check feature (in the latest version it automatically ignores meshes; in the previous version, have it open your AI when done, and then delete meshes by hand).

Oblivion Mod Manager (OBMM)
- Haven't tested the latest version, but Timeslip says he has added new AI.txt options in 0.7.6: "It lets you choose exactly which types of files to include, and there's options for ignoring files not in BSA's, normal maps, and files with incorrect extensions. Following MadBat's suggestion, it also deletes empty subdirectories from the data folder. Default settings are to only include textures, and only ones which already appear in a BSA archive. Hopefully, that should keep most people happy." Heck yeah! (In previous versions, use the DDS textures option.)

Wrye Bash also now includes a feature to generate AI.txt, but I haven't tested it yet.

FURTHER READING
Oblivion Mods FAQ

Original Thread: ArchiveInvalidation method tests.

Glossary
- AI.txt is shorthand for ArchiveInvalidation.txt.

Credits
Man of the Year award goes to Timeslip for solving the whole AI mess by avoiding it entirely!

Aside from that, there's a very big list of very talented and somewhat crazy people who put a lot of hard work into figuring out the AI mess. The list includes:
- DoomedMarauder
- Auctionmule
- Wonder Dog
- Scanti
- Malifrax
- Jarol
- Buerban
- Midnight Voyager
- Zanderat
- Martigen
- MadBat
- rdjeke
- DragoonWraith
- Oblivionmasta
- and many, many others ... Thank you all for this herioc effort!