Mass Effect Legendary Edition arrives in less than a week, and I know many of the mod developers in my scene are excited to start working on it. With that said, I am writing this to ask you to have patience, as updating the tools we’ve spent more than half a decade building is going to take some time. All of these tools were built under the assumption that there would be no more than 3 games.
I’m here to outline what’s going to be occurring with each of the major tools in the scene.
ME3Explorer (UI ApplicatIon)
ME3Explorer is going to be put into maintenance mode. This means that we won’t have further updates beyond just some simple bugfixes – it will essentially remain as it is now, and will only work on the original 3 games. We’re hoping to issue one final stable build.
We have been working for the last couple of weeks to migrate the existing UI application’s code into a new project, named Legendary Explorer. This application will (eventually) support the Legendary Edition versions of the games, as well as also support the original versions. We are doing this for a couple of reasons:
This lets us do a refactor of the project, as the app has had some pretty messy project structure for years now
It will better convey that the tool works for the Legendary Edition of the games
We are upgrading the project to .NET 5, which is a pretty significant change from .NET Framework 4.8 (do not let the numbers confuse you, this is many years of change). This will require some significant changes to how we also deliver stable and nightly builds to you, and it will take some time before we can get back to automated builds for users. Using nighties will require installing the .NET 5 runtime
The latest nightlies of ME3Explorer have exe version checks, which will prevent adding LE versions of games for game paths, in the event they use the same names of executables as the current games.
ME3ExplorerCore (ME3C) is going to be put into maintenance mode. A new library, named LegendaryExplorerCore (LEC) (or just LegendaryCore, we haven’t decided), will essentially be a copy of ME3C, but with support for Legendary Edition games as well. This is being done for the following reasons:
Projects targeting ME3C will not be broken when we have large codebase changes. While this could be accomplished via an older branch, we wanted to make a whole new project to ensure consumers of this library don’t accidentally upgrade and break loads of stuff and waste their time. The original scope of ME3C did not include LE games, and a lot of code was written under that premise, which if we added LE support, would likely break a lot of stuff.
If you want Legendary Edition support, you switch to the new library, which will have a slightly different (but mostly the same) API. This will force you to update your application to handle the new game IDs and behavior. Apps using LEC can still work on OT games.
If a user of ME3C wants features from LEC, they can backport it to ME3C.
ME3TwEaks mod manager
I’m not 100% sure yet on the plans for M3. I’m thinking of making a ‘new’ Mod Manager named Legendary Mod Manager, but it would be just a rebrand of ME3Tweaks Mod Manager, with support for LE games as well. This will take some time – until we know what we’re dealing with in LE, it will be hard to make Mod Manager support, and I will be working on mod development tools to start with.
Essentially, M3 would be put on ‘maintenance’ mode for some time, before it upgrading it once to support an upgrade to Legendary Mod Manager, LMM, similar to the transition that occurred when moving from ME3CMM to M3.
This plan would be for the following reasons:
Users don’t seem to understand that ME3Tweaks is my organization’s name. Not the name of the program. I have lots of other software that I release under my organization.
It will take some time to figure things out for Legendary Explorer Core. M3 depends on ME3C, so until we have LEC up and running, LMM won’t really be able to depend on it. Things like opening packages, lots of helper directory classes, that sort of thing.
Mod Manager is far more complex under the hood than many users think and adding three more games to the codebase, even if similar, will require a pretty huge investment of time.
The server infrastructure that M3 currently uses pretty heavily will need some significant changes to accommodate more games in the databases, web pages, etc.
I want to split many core features of M3 into its own library that can be used in other projects, like how the GameTarget class is reused in ALOTInstallerCore, MassEffect2Randomizer, etc. It is a lot of effort to support these duplicates, but it is also hard to make a localizable core library, especially for programs that don’t use localizations.
ALOT Installer has some more features planned for the Original Trilogy (OT), but I simply haven’t had time to work on it. ALOT Installer will remain OT only for now.
Once Mass Effect Modder is updated for Legendary Edition (this will likely take some time) I would like to make a general purpose texture installer program that works for LE. I do not expect it to be cross-compatible with OT. A general texture installer would be nice, especially with the same UI as one of the most popular OT mods, however texture modding will not be a high priority for me to start.
Mass Effect Randomizer / Mass Effect 2 Randomizer
These will remain working for their original games. At some point I would like to revisit them for Legendary Edition but it will likely be months if not years. Work on these projects have significantly benefited ME3Explorer (+ME3C), as they stress test the library and help identify weaknesses, and also lead to new features, such as memory-safe extraction, import resolution, and new debugging tools.
ME3Tweaks ModMaker will remain as it is and only be for the OT version of Mass Effect 3. While I am very proud of the work I did for ModMaker, it is an absolutely huge project that has extremely high support requirements, and was built when I was a much more amateur programmer.
Since there is no multiplayer in LE, I don’t really see a need for something like ModMaker, which mostly was a gameplay modding tool. ModMaker worked well because multiplayer was very gameplay focused, so changes through ModMaker would be very noticeable, and could work in short spurts. This type of modding doesn’t really translate well to singleplayer which has a lot of things that are specific to each level and takes place over the course of the entire game, and also doesn’t work well due to how the game is compiled – singleplayer and multiplayer use very different strategies for asset loading.
It is a true shame there is no multiplayer in LE.
Gibbed SAve EDITORS
Gibbed has expressed interest in updating his save editors for LE. I don’t expect this will be too much work for ME2/ME3 since it doesn’t look like much has changed in those games. As I am not part of the development of these programs, I don’t have much more info on this.
I’m sure many developers want to be the first person to release a mod for LE. In the beginning there will be lots of teething, and there will likely be lots of buggy software. As someone who has to support mods through Mod Manager, there are some things that will be very helpful for later development in the scene:
Do NOT use exe installers. These are an absolute nightmare to deal with in the scene, and if some new modding method comes along, nothing can really work with exe mods. Mod Manager can be updated to account for new processes or installation corrections. Manual installation directions can be adjusted. Exe installer based mods cannot have their install behavior changed.Exe installers are also opaque, so it’s hard to know what it will actually install. It makes identifying incompatibilities difficult, especially if only a specific configuration installs a file, but the others don’t.
Do NOT leave mod pages with outdated main downloads paired with ‘updates’. These are a huge waste of end user’s time, and also waste tons of developer time because we end up having to support users who don’t know what they’re doing. Spend 20 minutes of your time integrating the update so you don’t waste collective thousands of hours of other people’s time.The exception to this is for mods like ALOT, which are huge, but have whole systems built for handling the update in an easy way.
If the new games use the DLC system, DO NOT release variants of your mod with the same folder name. These are not possible to distinguish without tons of investment of my time. Many early mods in the ME3 scene did this, and the only thing it does is confuse users! What version of a mod did I install? Who knows? They’re all named DLC_CON_FUNMOD.Mods installed through Mod Manager (that are designed for Mod Manager) don’t have this problem as I tag the options selected, so even if you have 3 ‘variants’, users can see what was installed.
Use DLC folders if they are supported – basegame file replacements SUCK unless they are absolutely required. They are hard to track, require lots of manual inventorying by me to identify where they came from, and are a pain to uninstall for users.
Be prepared for game updates. There will likely be a few months of them and they may break your mod. If files in your mod end up being updated by game updates, but your mod isn’t updated, I may blacklist the mod from installation in Mod Manager due to it leading to different ‘versions’ of games. All mods should be built from the same ‘base’ version of the game, which will eventually be the final game update, whenever that is.This may sound inconvenient to you, but it is much more inconvenient for us developers, when we have to code around poor decisions by mod developers. Make our work easier so we can focus on better tools rather than coding around your work. You may think this is a one time cost, but adding lots of edge cases just means every time we update something else we have to make sure it still works with edge case code, which has an ongoing burden.
There will be some significant pain at the start of modding these games.
Some formats have likely changed and will take time to reverse engineer.
There will be no reliable cross OT <-> LE modding for some time. Please note that when I say OT <-> LE, I mean porting individual pieces, like the properties of an object, not running a mod through a converter, this will simply not be possible due to them changing to DirectX 11.
There will be a lot less debugging tools – we will not have an asi loader, we may not have console access for a bit even. Pray that they leave some debugging tools in the game for us, because it’s going to suck BIG TIME if we are left with nothing. Why do you think there are so few ME2 modders 😉
Again, please be patient. There will be a lot of teething issues for setting up modding in these games. Remember that developers are people too 🙂
I’m Mgamerz, owner of ME3Tweaks.com, and the lead developer of the Mass Effect Trilogy editor toolset, ME3Explorer. I also write end user installation tools ME3Tweaks Mod Manager and ALOT Installer. If you’ve modded the series recently as an end user or a developer there is a high likelyhood you’ve used my software to do some or all of it.
Recently, BioWare announced Mass Effect Legendary Edition, which is a remastered version of the original three games. I’d like to share with you what I know about the new games, what they mean for modding, and what my plans are going forward, as I have had many people ask or tell me things that need to be cleared up.
Will existing mods work on Legendary Edition?
No. To be clear, when I say ‘existing mods’, I mean package file based mods, which is the majority of mods. Anything that ends in .pcc, .upk, .u, .sfm are package files, and are what comprise of the majority of each game.
Our files are tied to certain engine versions of BioWare’s fork of Unreal Engine 3. On top of this, each game in the trilogy is on a different build of Unreal Engine 3 with more and more BioWare changes on top, and that spans 5 years of engine development by Epic Games as well. Unless BioWare changes absolutely nothing in their game, things will not work.
Complicating matters is that while our modding tools have improved greatly recently, many mods were built with older, buggier tools, and it is more a bug that they actually work in the game, rather than them actually being properly supported. Many of my older mods even have issues, but the game somehow still works with them.
Essentially, you can think of it this way: Will vanilla package files from the trilogy games work on legendary edition? I’m extremely doubtful, there are too many dependencies on core files that are nearly guaranteed to be changed.
But BioWare says they’re working with the modding community?
BioWare may be working with some people, but I’m not sure who. I’m curious, but I also understand whoever they are, they are under NDA, and asking people things who are under NDA just makes their life more difficult.
However, I’m not under NDA, so I can tell you what I know. Nobody in the toolset development group has been contacted, which consists of three people: Myself (Mgamerz), SirCxyrtyx, and Kinkojiro. We are the ones who build and maintain the tools (ME3Explorer) that are used to make mods, and I personally am the one who also builds the tools that installs the majority of said mods (ME3Tweaks Mod Manager).
These tools are practically required to mod the trilogy (excluding texture-only mods), and all three of the developers know the engine internals in ways that many mod developers don’t – for example, the flags in package headers, various file formats, how property serialization works in our games, Unreal’s memory system, etc.
At this point in development of Legendary Edition, there is not a lot that BioWare can internally change to help with modding. You cannot bolt proper mod support onto a game, it must be done from the start, and nobody reasonable expected BioWare to do such a thing.
What are my concerns?
Ever since Mac Walters tweeted about working with the modding community, I have had dozens of people message me directly about how they’re excited to have existing mods work on legendary edition. Many mod authors are also excited to work on legendary edition.
I do feel that there is an expectation of modding Legendary Edition that is not actually based in reality, and it’s going to cause issues if the game turns out to not be so easy to work with. End users will blame both us, the tool developers, and BioWare, as they feel they were mislead.
They will also blame mod developers because BioWare’s statements made them think this was easy – we have users already come for assistance asking why these games are not easy to mod like Skyrim.
But can BioWare do anything to help?
BioWare for nearly a decade has seemingly taken a ‘do not acknowledge mods’ policy, so for them to say they’re working with the mod community out of the blue is a real whiplash for the scene. I would love to have a conversation with BioWare to discuss things that could help the modding scene that should not take a lot of effort to implement, but I’ll detail some of them below.
1. We need a way to debug the game
ME1 on PC included the application log functionality, which wrote a bunch of useful information to the Documents/BioWare/Mass Effect/Logs folder, including crash information (such as loading an unused mip).
This log functionality was removed in ME2, and all internal debug code also was stripped out, so debugging ME2 is nearly impossible.
ME3 included a blank stub for ‘appErrorF’, which we hooked to get a basic level of debugging, but it’s still pretty weak and often doesn’t have any useful information. Complicating things was the use of TOC files which for years added extra steps to modding that were not required for ME1/ME2.
If you want us to be able to more easily mod the game, we need a way to know what the game engine is doing, or is unhappy about. Can you imagine writing software where the only way you know something is wrong is when the application unexpectedly closes? Because that’s what modding is often like.
We may have advanced tools these days, but when it comes to debugging and fixing a mistake you made, it can be such a frustrating experience that you just give up. If Legendary Edition removes the logs feature and strips out debugging tools we can leverage, it’s going to seriously cripple the ability to mod these games. Even if logging functionality required another edit to enable (such as through the configuration files) it would go a LONG way in helping us.
2. WE NEED A WAY TO LOAD FILES IN AN OVERRIDE FASHION
In the current scene, we load files primarily through the use of the DLC loading mechanism for ME2 and ME3. We recently (about 2 years ago) uncovered some tricks in ME1 to also load DLC overrides in ME1, but it’s weaker and has some caveats, such as not working with movie files and the TLK files being embedded into a billion files (though I doubt this is an issue, as the PS3 version of ME1 did not use this system).
There are some core files that can’t load from DLC, such as SFXGame, which is a real problem because it has a lot of crucial edits. One system that I personally leverage is TESTPATCH in ME3, which can override classes, but we don’t understand how it works well enough to generate new entries.
If the DLC system is removed – which it very well could be, because if this is the ‘best’ version of the games, they won’t need to have post release content, and thus there’s no need for DLC loading – it’s going to greatly complicate modding. Replacing files in the main game is extremely easy to do wrong, leads to wide incompatibility, and is frustrating to support; I know this because for years I wrote multiplayer mods, which required file replacement.
DLC mod loading allowed us to override and remove mods fairly easily. There are plenty of downsides but it is a huge improvement to direct file replacement.
3. Moral support? (optional)
For nearly a decade, modders have built loads of mods – Mass Effect Happy Ending Mod, controller support, bugfix mods, new levels, same gender romances, texture overhauls, etc…
Personally I’ve built about a dozen mods, made many programs, even built an entire web application that can be used to build mods – but I can count on two fingers the amount of times I’ve ever had my work acknowledged by someone at BioWare in any sort of way that actually meant something, beyond essentially them saying ‘cool’.
This is a somewhat weird thing to bring up, but if you look at the creative scene for Mass Effect, there are many areas you look at – artwork, cosplay, streamers, etc, but the modding community is by and large completely ignored, and it is somewhat demoralizing.
We build these mods because we love these games, even if we at times are very critical of your work. This modding scene was actually created because of the critical reception to the end of Mass Effect 3, but it’s because those people loved the series, not because they hated it.
There are likely a lot of legal ramifications that bar you from coming out and endorsing mods, plus a lot of things like ‘is this a reputable person’, etc. that are not really present when you post on a picture on Twitter that someone made. I also know that by tweeting or coming out saying ‘What a nice mod this is!’, people would almost assuredly misinterpret it as you ‘supporting’ mods, rather than just ‘supporting the idea of them’.
That said, it would be nice if the work we do could at least have a way of being acknowledged. For years we have been mostly ignored, and while I’m sure some developers are fine with this, it would greatly improve modding morale if we at least knew someone at BioWare actually cared, or at least thought it was interesting, even off the record. I’m sure there are some people that make these games that do enjoy our work, but you would be hard pressed to find anyone who develops mods that knows that.
4. WE NEED A WAY TO IDENTIFY WHICH FILE BELONGS TO WHICH GAME
Yes, this is a real nitpick, but it’s actually come up when trying to parse files from PS3 and Xbox assets of ME games. If package file headers could have some way to identify their game (something unique, like package tag) it would greatly reduce complexity. It seems small but as someone who works on tools, knowing what game a file belongs to is VERY IMPORTANT.
This is of course assuming that you are unifying the engine versions, which I assume you are, since this was already done for the PS3 versions of the games.
I plan on working on legendary edition, but I want to make it clear to end users AND mod developers that things are going to take time. One of the things that greatly concerns me is how news sites are saying how great this game will be to mod based on just a few tweets and an interview that did not really talk about modding.
I want to make sure users and developers understand that modding will take time, will not work on day one, and likely will not work on month one, and will likely have serious teething issues possibly for years.
As someone who works at the top of the modding chain, a lot of the development efforts have fallen to me – a developer needs a feature in Mod Manager for their mod? I have to implement it. Need an easier way to edit game files? One of the toolset developers typically has to do it – and we do this for free on our own time.
The other developers I work with (SirCxyrtyx and Kinkojiro) are great people who have all done years of research and development, and together we’ve built lots of great tools and features, such as Pathfinding Editor, entry porting and relinking, dialogue editing tools, and more.
I personally spent YEARS developing and refining Mod Manager and have spent years completely reworking ME3Explorer into a much more professional grade tool. Our scene is much healthier than it was back in 2013-2015 when our tools were still immature, making and using mods is far easier these days. Some of this progress is going to be reset with the release of Legendary Edition.
WILL EXISTING TOOLS WORK ON LEGENDARY EDITION?
Short answer, no. We will have to wait and see. I do not have any plans to have ME3Tweaks Mod Manager work on LE – it will require me to build a new tool, as M3 is very tailored to the current games, tools and codebase. ALOT Installer will not be necessary (hopefully!), and ME3Explorer may be forked into a new project depending on how much has changed internally, but we don’t really know anything useful yet.
Personally, I am not a fan of trying to build a conversion tool between the trilogy games and legendary edition, as it would be a monumental amount of work, especially if we don’t understand changes to the internals of the games (including many things we currently don’t know about the existing games). The toolset already supports about 7 or 8 games across 4 different platforms, adding another 3 will be significantly complicate the codebase. But it really depends on how much is changed.
Yeah, the last thing I’d like to mention, is that a lot of this post is about the developer side of things. Myself, SirCxyrtyx, and Kinkojiro have all done absolutely masochistic things in the name of research and development (but especially Kinkojiro), but the flip side is that there are far more users who have little to no patience even installing mods.
If installing mods is difficult (such as a complicated system without a DLC loader), many users will not even attempt it, and this leads to diminished returns on the development side. I work on both sides of the coin, writing the tools for installing mods, as well as developing them, and over time I’ve adjusted my software to make it easier for users.
I’ve noticed that as it becomes easier to install mods, more developers have appeared, and part of it may have to do with decreased support time for mods. My end-user tools for example have a very detailed diagnostic system that can be used to identify a wide variety of issues, which means users don’t have to ask developers how to fix things.
I believe our modding tools for legendary edition could be greatly improved if we had someone we (the modding toolset developers) could talk to at BioWare about, as there are still many gaps in our knowledge of BioWare’s proprietary changes to the engine that significantly hinder our efforts. With the release of Legendary Edition, those knowledge gaps will likely significantly widen.
Want to have a talk about some other things the scene could use BioWare? Drop me a line at femshep AT me3tweaks.com, there are many small things that would greatly reduce the friction of modding that would really help set up the scene for success.
Hi all, it’s me again, your favorite modder who publishes a single research blog post a year. Welcome to my new blog, where I will also post maybe once a year! I got fed up with blogger’s endless unfixed bugs. I’m going to leave the content there though for historical sake.
I just finished a hardcore crunch to ship ALOT Installer V4, which is a complete rewrite of ALOT Installer. ALOT Installer is the Mass Effect modding scene’s main texture installation tool, built on top of aquadran’s MassEffectModder program, which can be used to install textures in a more advanced fashion. In V4 of ALOT Installer, I split the main ‘core’ features into a cross-platform .NET Core library so I can also write a frontend that works on Linux. But that’s not why I’m here today – I’m here to follow up on how I fixed Mass Effect on PC to not require elevation for good.
Mass Effect on PC: About what you’d be expect from a mid 2000’s console port
For those of you not in the know, Mass Effect came out on PC back in 2008, and was ported from the Xbox 360 by a studio named Demiurge, who also developed Pinnacle Station for Mass Effect. It’s… a really meh port that has not aged very well. It’s passable as a game but it has a lot of problems, even when it came out. Particle LODs not working properly, texture LODs being read backwards, ini settings being randomly reset to their defaults, the problems are pretty numerous, just to name a few. But nothing completely game breaking.
Well, kind of. There is one, but it’s not specifically due to Mass Effect. The big issue is that Mass Effect requires administrator rights to run, because Demiurge seems to have assumed everyone would run the game as administrator – which might have been OK if the game was only really developed when Windows XP existed, but Windows Vista had already been out for over a year by the time the game had released. Even back then though, Windows XP had a concept of LUA (Least User Access) with separated user accounts. For more information on this, you should check out the original post I wrote, Why Mass Effect on PC requires administrator. It describes a lot of backstory to this post.
Oh boy, PhysX, my favorite physics library!
Mass Effect for PC runs on a lightly modified version of Unreal Engine 3, which appears to be dated around late 2006. According to some former BioWare developers, this version of Unreal Engine was not very mature yet, to put it lightly. According to some stories from these developers, it was really difficult to work with because Epic Games was focused on Gears of War and not dedicating much time to their partners who were also using the engine.
Unreal Engine 3 uses PhysX for physics interactions, so Epic Games built a dll that interfaces PhysX to Unreal Engine data formats through a file named PhysXLoader.dll, which loads the PhysX libraries from both parties. PhysX is a physics simulation library that was acquired by AGEIA Technologies in the mid 2000s before AGEIA was sold to Nvidia in early 2008. If you remember Physics Processing Unit cards, or PPU, they were using PhysX before Nvidia promptly killed that idea.
All three Mass Effect games use PhysX, but Mass Effect 2 and Mass Effect 3 use the system’s install of PhysX, while Mass Effect uses the local game’s PhysX. Mass Effect 2 and Mass Effect 3 also use the ‘modern’ version of PhysX, rather than the legacy one that was shipped by AGEIA. Nvidia changed some paths under the hood when it took over, which separates Legacy out from it’s ‘modern’ versions.
But that doesn’t seem to stop Legacy PhysX’s uninstaller from deleting modern PhysX’s files/registry keys, so during the course of testing this fix, my other copies of Mass Effect 2/3 didn’t work, even after installing the ‘modern’ PhysX redistributable. It’s really annoying how BioWare couldn’t just ship a 8MB library with the game – they already shipped the installer for PhysX with the game, so it’s not like it saved space!
The issue with Epic Games’ PhysXLoader.dll is that it can load PhysXCore.dll locally, or from the system’s installed version
Err… wait, how is that an issue? Can’t you just load the local dll, and if that doesn’t exist, load the system one? How is that an issue exactly?
On boot, Mass Effect writes two values to the Windows HKEY_LOCAL_MACHINE registry:
*Mass Effect is a 32-bit program, so on 64-bit systems it goes into HKLM\SOFTWARE\WOW6432Node\AGEIA Technologies instead, if you’re looking for yourself.
Remember these registry values, they’re going to be important later!
These registry values are why Mass Effect requires administrative permissions. In my previous blog post linked above, we explored why these writings were enough to make Microsoft put Mass Effect into it’s compatibility database, which forces it to run as admin when matching on certain executable criteria, which we worked around by modifying the executable criteria to no longer match.
We have to modify the executable to enable Large Address Aware, so the game could load higher resolution textures without running out of memory, so there was no way to avoid breaking the signature. This in turn caused Origin to no longer run the game as it would not elevate games without a valid EA signature. But if the game cannot write these registry keys on boot, the game may crash…
So it’s already a big fun chain of problems, but we worked around Mass Effect needing administrative rights by simply giving the user account permissions to that specific AGEIA Technologies registry key. This would let the game process write the values it needed, and would we could go on our merry way. I assumed the game crashed because it was denied write permissions and Demiurge couldn’t be bothered to write a try/catch around the registry writing code.
You probably shouldn’t name your registry values as a hack if you want me to think this is a good idea
Our solution to this problem did not change Mass Effect’s behavior – the values it wanted to write to the registry were going to be written one way or another, so we were just letting it do the thing it’s always done, just without administrative rights. There wasn’t really any change in application behavior.
mirh, a moderator for PC Gaming Wiki, sounded the alarm for years that somehow we were breaking other games in ALOT Installer – even though our application didn’t actually change how Mass Effect was behaving writing these values, so there’s no way our change would break other games.
After many months, he wrote a fairly detailed reason why ALOT Installer (when, in reality, it was Mass Effect) is breaking other games: enableLocalPhysXCore being in the registry is used by other games using Epic Game’s PhysXLoader.dll. When I was writing V4 of ALOT Installer, I told mirh I would take a more serious look into his idea of a solution that would not break other games, even though at the time I did not really understand how a registry key with the system’s MAC address would break other games – or why it even used a MAC address to begin with.
mirh seems to have determined this enableLocalPhysXCore lets Mass Effect use the local directory’s PhysXCore.dll/NxCooking.dll, instead of loading the one from the installed PhysX redistributable. Mass Effect doesn’t install the PhysX redistributable, so it could not rely on it existing, so it needed to use the local libraries.
Hope you’re strapped in because this is where it gets really dumb:
The MAC address stored in in the registry by MassEffect.exe is read by PhysXLoader.dll and compared against your system’s MAC address to determine if it should load the local directory’s PhysX libraries or the system’s.
Which MAC address?
So the way Mass Effect works:
Very early in the boot process of MassEffect.exe, your MAC address is read and written to the registry as enableLocalPhysXCore (along with EpicLocalDllHack)
MassEffect.exe loads PhysXLoader.dll
PhysXLoader.dll reads the value of enableLocalPhysXCore and compares your system’s MAC address against it
If it matches, it uses the local folder’s PhysX, if not, it uses the system’s redistributable version of PhysX
Yes, you read that right.
It turns out that other games, such as Mirror’s Edge, have a PhysXLoader.dll that also reads these values (as they’re based on the same code), but they don’t include local PhysX libraries. So those games boot up, see enableLocalPhysXCore, and try to load the local library, which fails, and the game doesn’t start. This information is second hand from mirh – I have not tested other games broken by this registry value.
Normally that value wouldn’t exist, and it should use the system PhysX. This behavior can be tested in Mass Effect by denying it write permissions to the registry key, deleting the values, and having Legacy PhysX installed – it will use the system libraries instead. If system PhysX is not installed, the application will not boot – this is why we originally had to let Mass Effect write these keys, otherwise it could appear that the installer broke Mass Effect, when it actually was a terrible implementation by Epic Games.
If you’re interfacing with a library that has exports you can call to initialize/load the PhysX SDK… couldn’t you just, you know, pass a boolean to tell it to locally load? Why does it not locally look to begin with? And what’s up with the MAC address? Why is this in the registry, where it behaves LIKE A GLOBAL SETTING???
All of these seem like terrible design decisions – and after disassembling the PhysXLoader.dll, it seems like that’s all they were – terrible design decisions. Let’s take a deeper look at Mass Effect and walk through the process of fixing this, from start to finish.
Finding a starting point
WARNING: I’m a super novice reverse engineer. I’ve assembly modded Megaman Battle Network games (and wrote a pretty neat guide to how to design hooks), designed mods in ActionScript2 P-Code, and worked with UnrealScript bytecode, but never really got far into x86 assembly. I’ve opened IDA numerous times, and could kind of find what I was looking for, but never really could understand it. I’m sure this process is much easier for more experience reverse engineers.
Recently (as in the past 2 years), the National Security Agency of the USA (the NSA) released Ghidra, a free open-source reverse engineering toolkit that can reverse assembly back into somewhat-readable C code, which is infinitely easier to read than IDA assembly graphs. Both IDA and Ghidra have their strengths; IDA has a debugger that lets you step through the assembly and see which code paths are about to execute, and can find Unicode strings (which Mass Effect games use), while Ghidra can recompile assemblies from it’s decompiled C code (sometimes), has an assembly-to-C converter (sorry, I don’t know the name of this), is open source, and works on tons of platforms, binaries, you name it.
So the information I knew starting out was that Mass Effect was writing enableLocalPhysXCore and EpicLocalDllHack. Let’s start by looking at MassEffect.exe, finding these strings, and seeing what references them. Using a hex editor, I know these are unicode strings, so I’m going to look for them in IDA, since Ghidra doesn’t seem to support this.
Inside of IDA’s Strings window, searching for enableLocalPhysXCore shows the string. Double clicking it takes us to the data section of the executable it’s defined in:
Above we can see the definitions of the strings that all seem related to our quest. Above the text definition, we can see there’s a DATA XREF, which means something directly references this data – probably what’s writing it. Let’s double click the XREF and see where we go.
Looking at this, we can see it’s writing RegSetValueExW. I’m a very weak C developer, so after doing some googling, I can see that this is the stack being setup to invoke a method call from the Windows API in C, which you can somewhat see with the parameter name that IDA shows, such as lpData and dwType. We know that enableLocalPhysXCore’s value is set to your MAC address – lets see where that gets set. Let’s change to Graph View for a more logical look at this.
We can see in the third block that eax is pushed onto the stack for lpData, and it’s also pushed onto the stack for this mysterious ‘sub_10929393’ call. There are no other calls in this subroutine that don’t have defined names, so this is probably where the MAC is obtained. Let’s jump into it.
This seems to be some sort of wrapper subroutine, or perhaps something IDA does, but it just points to another subroutine. Let’s go there instead.
This subroutine has names that are pulled in from the Windows API that show us that this has to do with networking. We don’t really care about the MAC address, but this lets us define the name of this subroutine. We’ll call it GetMacAddress. We’ll go back to our original subroutine we were looking at, and rename that too – this seems to be something like SetupPhysXSDKLoad, so we’ll name it that.
This is a relatively small subroutine, and all it does is write the two aforementioned registry values, and that’s it. There are no other references to this subroutine beyond one early in the boot process – so at this point I’m fairly certain that Mass Effect’s executable never actually reads these values, and at this point I’m still not sure what EpicLocalDllHack does.
Cracking open PhysXLoader.dll
Now we know that Mass Effect’s executable never reads this key, it must be in one of it’s dll’s. While I won’t show it here, using ProcMon (a great tool for modding and doing things like this in general), I could see that the registry value was read right before the library was loaded in the MassEffect.exe process, and the local dll was loaded. I observed it reading the system one when I denied Mass Effect write permissions to this directory, and the game not loading at all when there was no system version of Legacy PhysX installed.
The first dll loaded is PhysXLoader, after which PhysXCore.dll loads, so that is our logical dll to analyze. Let’s crack this open in IDA and see where enableLocalPhysXCore gets used there. I’m also going to pop this dll open in Ghidra so I can get a better idea of what’s going on. Doing the same procedure as above for finding where enableLocalPhysXCore’s string is used, we find this subroutine:
This is not too hard of a subroutine to read, especially in graph view – we can see there’s a loop from the left box going to the box above it. Still, not the easiest thing for a novice to read, so let’s see what this looks like in Ghidra. I use the location of this subroutine to jump to it in Ghidra (0x10001640).
This gives us a bit more of an idea what’s going on – the subroutine calls, the loop, the return type of the subroutine. We can identify things in one tool and label them in both to help us figure out what we’re doing. Immediately we can see that there’s a subroutine call that passes a registry key path, a value name, as well as two variables. In the loop we can see it’s comparing 6 items by index to determine if they’re the same.
Just to make this a post a bit shorter, that registry lookup subroutine is indeed, a simple registry fetch wrapper. Its parameters after reverse engineering (which mostly was just mapping subroutine inputs to what they are for the windows api calls) are subkey, valuename, returned data pointer, returned data size.
You can see it has a loop that runs for 6 iterations, checking each value is identical. Using the information we’ve gained, we can rename some variables to get a better picture of what this subroutine is doing.
We know that Mass Effect wrote a 6 byte mac address to the registry, and that PhysXLoader.dll just read that value from the registry, and the subroutine is comparing something byte by byte 6 times. Logically, we can assume that in the above picture local_14 is the MAC address. Knowing this as well, we can guess that FUN_10001580 is fetching the MAC address and setting it, so we’ll rename some more items in this subroutine.
Now, this subroutine call seems to not do the actual loading – it is just checking for the key, and if the MAC address matches. Given the name and what we know about this key’s activity, we can give this subroutine an educated guess of a name of ‘ShouldUseLocalPhysX’. Comparing IDA and Ghidra’s decompilation of this subroutine however lead to slightly different results, with Ghidra’s seemingly being wrong:
After doing some googling for this part of the post, I learned that EAX is typically used as the return register for x86, and the ‘al’ register is the bottom 8 bits of EAX. I’m not experienced enough with Ghidra to know how to retype the signature for this kind of lower 8 bits being returned, it may be just something Ghidra does not support currently, or I am missing a setting I was supposed to use.
However, if we look at the references to this subroutine (there are two – likely one for each library) in IDA and Ghidra, we can see that ShouldUseLocalPhysX is being called, it’s checking if al is not zero. If it’s not zero, it loads the local PhysXCore.dll. If it is, it appears to look it up through the system PhysX installation, which is identified by another registry value in the AGEIA Technologies key named PhysXCore Path. We’re not really interested in that, since we’re looking to force PhysX to always load locally, regardless of the enableLocalPhysXCore being set or not.
Looking at the other cross reference, we can indeed see it’s loading the NxCooking library, using ShouldUseLocalPhysX in the same way:
Armed with this knowledge, there are a few ways we could fix this. I’ve done a lot of function modding in UnrealScript bytecode, which for a very long time could not be extended or shrunk in size, so figuring out ways to make logic checks fail or pass without changing the size of the script was a challenge.
For example, if I needed an if check to essentially be removed, I’d need to find a way to change the comparisons to always be true or false. One way I could make an if check fail would be to modify the object references and comparison bytecode tokens to produce a condition such as if (objectA != objectA), will always produce false (assuming they are not null). We need to find a way in ShouldUseLocalPhysX here to always produce a true result.
When I was writing a symbol table for Megaman Battle Network 3, I learned to comment everything I figured out about the disassembly. I would work on something for hours, totally forgetting what I did, but could come back to my comments and understand it again.
Sometimes my symbols/comments on certain lines, such as subroutine names or pointer tables I’d identified, would come up in other subroutines, giving me useful context that I would not have seen otherwise. I’ve commented and labeled several items below, which make reading this subroutine easier.
Patching out the world’s worst boolean check
We need to ensure the bottom left block is always reached – while also making sure we don’t touch the stack (which can EASILY nuke the whole program). Technically, we could just rewrite some of this subroutine, but I’d like to make the least amount of change possible here. We need to essentially make sure the jump to the bottom right block never happens.
Conveniently, x86 has a 1 byte instruction named ‘nop’, which does literally nothing but take up one byte. Also conveniently, the jump instruction to that block is 2 bytes, comprising of a 0x75 (jnz rel8) and a 0x19 (relative offset).
[FUN SIDE STORY] Seeing a 1-byte offset brings me back to my Megaman Battle Network modding days, where jump/branch instructions would make or break moddability for certain sections of the ROM. When writing a hook (which redirects the program counter to your own code), you had to find a jump or branch instruction, which you would modify it’s relative offset to point to your code. You would then push registers onto the stack, run your code, then put the stack back together so the subroutine exited like it should.
ARM (more specifically, THUMB) had limited branch instructions that used different sizes for their relative offsets, which were not able to always jump to any point in the ROM due to their location in the ROM. Since the game was written in assembly, finding free space at times could be difficult – sometimes you’d have to chain multiple hooks until you could get the program counter into to a free area to write new assembly code. This jnz uses opcode 0x75, which is jnz rel8, so it can only jump up to 128 bytes (or is it forward only, so 255 bytes?) away, which would be a real problem if I was doing assembly modding the way we used to back before we had fancy tools like IDA and Ghidra. [END OF FUN SIDE STORY]
Anyways, after we nop out that jnz to the not-matching block of assembly, our ShouldUseLocalPhysX subroutine now looks like this:
Now the not equal block cannot be reached. The ‘check’ still occurs, but it never can return false. The local PhysX core dll will always be used.
What are the downsides?
The PhysXLoader.dll file is signed by Epic Games, so this obviously breaks the signature since we modified the file. The game is not checking signatures on load, so that’s not an issue. Some antivirus programs may complain about broken signatures, but over time that typically improves. Outside of writing an in-memory patch (like we do with the asi mod loader our scene has), we would have to modify the binary of the library.
Using this patched dll, the game now works with or without the registry value, which means Mass Effect no longer needs administrative rights at all to run. Disassembling this was a real rant-fest because I could not get over the asinine way this check was implemented, not just a registry value but a match on MAC address too. While I was debugging and stepping through the instructions, I actually broke the game because I turned on my VPN and my MAC address it used changed.
This was a good learning experience, I’ve learned more about Ghidra and IDA, and about more problems in the PC version of Mass Effect. This patch is automatically applied as part of the install step of ALOT Installer, so users will never need to worry about having the enableLocalPhysXCore key set after it’s patched. We also modify the Mass Effect executable to write a value named enableLocalPhysXCor_, so our patched versions won’t write the value that breaks games. Vanilla Mass Effect executables will still break other games, but it is beyond the scope of our program to try and protect people’s software from poorly written PhysX loaders.
Essentially, this is how I felt afterwards:
Oh, and what about EpicLocalDllHack? Well… it does literally nothing. Absolutely worthless, it’s never read. The only thing I can think of that it could maybe exist for is to keep the registry key around if a PhysX redistributable is uninstalled, as it’s not part of the PhysX redistributable’s list of values – but that’s just a guess.
Was having a parameter ‘PreferLocalSDK’ for PhysXLoader.dll too much to ask for Epic Games?
This post originally appeared on my now deprecated ME3Tweaks Blog that was hosted on Blogger. It’s been ported here for consistency. This post was published originally on January 30, 2018 and has only been slightly modified to make it easier to read here. The contents remain unchanged.
Hi all. It’s been some time since I’ve posted. No I am not dead. I have been doing a lot of work behind the scenes. Been working on my own ME3Explorer fork which has many quality of life improvements and even some new tools. I’ve also been working on Mod Manager 5.1 which has some slick new importing of third party mods, but it’s been on the back burner since I’ve been working on the new ALOT Installer front end.
I have been working with CreeperLava and Aquadran to implement this, which should make life way easier for end users installing ALOT and its Addon (textures by others). One issue I have had is that Origin will not launch the game once ALOT is installed unless run as admin. And since you cannot have origin startup as admin on boot it is really annoying. This also affects MEUITM. So I looked into figuring out why. It’s actually a perfect storm of security, bad coding, and people trying to make others lives easier.
Lets break down how Mass Effect works with Origin in a non-modified state, on Windows 10.
You run MassEffect.exe. It immediately asks for elevation to administrator.
At the end of the MassEffect.exe image there is code to call Origin to run the game, as a form of DRM. It calls origin, and then exits.
Origin checks your entitlement to the game and will then run MassEffect.exe as specified by the registry (not the one you booted specifically) and attempts to run the executable.
Origin is unable to run the executable because it requires elevation. In order for the DRM to work it must be able to interface with the process, so it elevates one of its internal services so it can communicate with the game for DRM purposes.
MassEffect.exe is run as administrator. Origin communicates with MassEffect.exe and game execution continues like it does from the DVD version.
Now, this all works (though two UAC prompts) on an unmodified game. But install MEUITM or ALOT and you won’t be able to run the game anymore through Origin as a standard user. What gives?
Both MEUITM and ALOT modify the MassEffect.exe executable to use Large Address Aware. This allows the 32-bit Mass Effect process to use up to 4GB of ram instead of the usual 32-bit limit of 2GB. By modifying the LAA flag, the digital signature on MassEffect.exe is broken – the signature is used to verify the file is not modified. Once the file is modified, the signature is no longer valid.
Origin, when running an elevated process, checks to see if the EXE is signed by EA and is valid. If it is not signed by EA, it doesn’t elevate its DRM communication module. Mass Effect boots up, and then immediately closes as the DRM unlock doesn’t work because there is nothing to talk to it from Origin’s side as it refused to elevate.
So by modifying the EXE, origin will refuse to run an elevated game executable. But we need LAA, so we must work around this issue. Our only hope is to prevent MassEffect.exe from running as an administrator. We should figure out how it is set to run as administrator first.
Looking in the EXE’s manifest I can see it runs as the invoker – the user who is running the EXE. Which means this executable should not require administrator. I check my compatibility settings, nothing there either. Somehow, this is being elevated at boot but not through the exe itself nor my own doing. The culprit? The Microsoft Windows Compatibility Database.
mirh (I’ve seen him in some of the modding circles) did some sleuthing to figure out why Mass Effect is being forced to run as administrator. It meets the criteria in the database – it specifically has an entry for Mass Effect – which has its own always-forced compatibility settings. It makes sense – users shouldn’t have to configure the settings if MS already knows ones that (technically) work.
As you can see there are two entries for this game – MassEffect.exe (the game) and the launcher (which is not included in origin’s version sadly). The compatibility fix is RunAsHighest – which means administrator – and the criteria are:
EXE is named MassEffect.exe
Company name in manifest is BioWare
Product name in manifest is Mass Effect
Product version is equal to or less than 18.104.22.168.
This criteria matches all known versions of the game – I believe including pirated versions. So by matching all of these criteria, the exe is forced to run as an administrator. You can test this out easily by simply renaming MassEffect.exe to anything else and it will no longer require admin to run. (Origin won’t be happy).
So now we have an idea of how to fix this – but why is this entry here? Well, due to Demiurge/Bioware not following the idea of Least User Access (LUA), Mass Effect on its very first boot requires administrative privilege to write to the HKEY_LOCAL_MACHINE\SOFTWARE\AGEIA Technologies registry key. If this key does not exist, it will attempt to make it – without admin, it doesn’t have permission to, and the game simply crashes. This key seems to hold some sort of information about what is now known as PhysX. This could possibly have been done by the installer, but it seems they made it be done in the game instead.
This is why Microsoft forces the game to run as admin always – for this one single item. It makes sense – make it run as admin, user does not need to worry about compatibility settings. However due to this combo of all three – LAA breaking the signature, MS forcing it to run as admin, and Origin refusing to work with elevated processes that have a broken EA signature – Mass Effect does not run with Origin and LAA.
So what did we do to fix it? We simply modify in the EXE the product name from Mass Effect to Mass_Effect. Really, that’s it. It fails the criteria check and the game no longer needs admin and Origin is happy (except for that update nagging it always does). In both MEUITM and ALOT Installer we added code to make it create this registry key with write privileges for the current user, so if Mass Effect needs to make the keys (maybe its never been run somehow) it will be happy.