Don't do coremods - a dev diary
Table of contents
Introduction - a rant about the Forge forums
Unless you are very familiar with Minecraft modding (and especially with hacked cli- er, I mean, utility mods) you are unlikely to know what a "core mod" is. According to the first thread I found from googling, core mods are:
a special type of mod that have the ability to not only have their own code in it, but also modify every piece of code in vanilla minecraft however they please
In other words, they are mods that mess with Minecraft's source. The follow up egregiously summarises the Forum's official stance on the matter:
This is a bad thing, because if this modified code fails, it will look as if vanilla minecraft itself has produced this problem, often making it near impossible to trace it back to the originating mod. Moreover ensuring compatibility when mods just modify vanilla code is much harder.
Put simply, they are getting overwhelmed with stupid bug reports that they can't really solve because the patched code looks just like vanilla Minecraft. It must suck, but... what can you do? Well, I can tell you what they did: demonise core mods, and eventually ban all discussion of them.
Skim through a few threads: you might notice the ever-friendly forum folks equating more and more core mods to hacking. Even if that was the case (it's really not), what's the big deal? We "utility mod" folks are as legitimate as any other mod developer. If you want to blame someone, blame the idiots using our clients to ruin everyone's fun. Not us developers. Most of us don't make a buck out of it, we develop these for ourselves to mess around on our own servers for fun.
And, as I said, they have eventually dropped core mod support altogether, locking any thread daring to ask questions even vaguely related to the subject, often accompanied by passive-aggressive remarks by the moderator about how "they don't support hacking".
Yeah, you heard me right. They don't just discard bug reports that include core mods (as would be legitimate), they outright ban any discussion even about implementing them. That's dumb, especially if you consider that most support is given by peers, not moderators.
This "father knows best" approach, discouraging any inquire about anything but the most basic API, ensures that nobody looks further than their own noses. Unfortunately, this attitude is spread to a good chunk of the project, not just the forums. Take a look at the READMEs of any Forge repo if you don't believe me: they are cryptic at best, and openly gatekeeping (saying stuff like "we won't spoonfeed you", as if writing documentation was a bad thing) at worst.
I'm pretty sure that this attitude deterred a number of competent developers who otherwise might've contributed. Just saying, most people would probably rather start from scratch than contribute to a barely-documented project. Ah, I'm just bitter. That won't happen!
But honestly, what actually annoys me is the hacker boogeyman. Whenever a question is asked on there that deviates from API usage, they will insistently inquire about why would you need to do that, regardless of how relevant that is to the question asked. The purpose here - and you can't convince me otherwise - is to find out if any evil hackers are hiding among the honest, hard-working, God-fearing folks of the forum, and out them as such and lock their thread, regardless of whether the question was interesting or even related.
muh xy problem
I could write another blogpost about how that's a non-issue, and I might, but for now let's just say that's my opinion and leave it that. The thing is though, Minecraft has a very wide age target. This means that they will attract anyone between the 35-years-old professional developer and the 13-years-old newbie.
Logically, even assuming that the XY problem is legitimate, you can't expect everyone to ask intelligent and to-the-point questions in such an environment. Kids will be kids. This, however, must not prevent someone capable of understanding from asking questions. The solution to the XY problem is most definitely not to assume everyone but you is stupid.
But if core mods are off-limits, how do they suggest we modify the code? Because, obviously, it's unrealistic for all mods (which, I remind you, stands for "modifications") to work around modifying the game code.
The problems stems from them being somehow unable to recognize core mod errors (even though any crash log is flagged with "core mod detected"), so what's their better way? That would be Mixin, which is shipped as a part Forge 1.13+ and can be used as a library in some earlier versions.
I, frankly, fail to see the merits in using it over direct ASM patching, as a poorly written Mixin patch is (obviously) just as prone to conflicts. It's probably easier for the developer, especially since you write Java rather than bytecode instructions. Downside is that, for obvious reasons, Mixin is a pretty big - even bloated - library. And, depending on who you ask, having to write in Java is a downside of its own. Cherry on top, in 1.12.2 and prior, you even had to compile Mixin into your mod's JAR.
Of course, that's no longer necessary, and the Forums have taken note of that. Since now they ship together, any inquire on the forums remotely related to "the devil's mixins" - the barbarous practice formerly known as core modding - will be met with distrust. Since newbies aren't capable of doing it, suggesting that it's possible to modify Minecraft classes without passing through Mixin is tantamount to heresy, and as such it should be treated with the same approach used a long time ago by the Spanish Inquisition.
You can't ask about core modding in older versions, either. Because asking about "old versions" gets your thread locked even if you are asking the most basic, innocent thing ever. "It's no longer supported" the moderator cries "use modern Minecraft!". Considering that, as I said, most support is given by peers, I can't help but think that banning discussion of Forge for older versions is... a bit of an odd decision. They even went out of their way to hide the 1.12 official Forge documentation, don't ask me why.
Enough about the Forums, though. You are probably here for the code, so let's get into that part.
The problem
Now that we have established that there is nothing wrong with a modder patching the game's code directly, and that Mixin is not ideal either, let's give them a point. Core mods are, admittedly, an inefficient way to patch the game. Reason being that they are applied at runtime by Forge. Plus, with how much they dislike this practice, they may eventually drop support for it altogether. I, in fact, thought they had when I originally wrote this post. They actually just moved it elsewhere in 1.13+ and never explained how to do it in the new way anywhere.
Now, I've mentioned "1.13+" a few times, and that is because the Forge developers decided to do a major rewrite of a lot of their code in that version (which is apparently the reason it took years to even reach beta status). The point is that they made Forge less monolithic, separating it into its various components. Of course, this modularity has a price, and interal components that previously could only work with each other had to be changed into publicly-accessible APIs.
This is relevant to us, because now we can write a program that acts on the same level as Forge and Mixin. This will enable us to apply ASM patches without bothering with middlemen. This is supposed to be more efficient than core mods, while still enabling us to patch with actual ASM, just how we like it. Why? Why not!
This is my dev diary of how we achieved that, despite the documentation roughly amounting to "here be dragons", and of all the problems we ran into during development. The code feature here is by definition buggy and broken, and is here just to detail how it evolved. So, if you are just interested in the final result, scroll to the end, but I'll hate you forever if you do.
The story
So, as we got the first prototype of our alternate auth server working, we figured that we could use a client to test it on. After some debating, during which even talks of using Kotlin (shivers) were heard, we decided to start a brand new client. We retained the name theme, for those who can get it.
As I mentioned, we thought Forge had ditched core mods altogether. My stubborness compelled me to continue looking for a way to do direct ASM patching, despite my less-than-enthusiast co-developer, and we got into it.
We needed to dig around for ideas from others. One problem: as I mentioned, ASM patching is really rare nowadays. Many client developers, unfortunately, are script kiddies who barely know Java. The idea of messing with something as scary-looking as ASM frightens them. This is not to say that Mixin is only used by incompetent people: many great developers do, in fact, use it, for ease of use or other considerations. I'm just saying that, between those that choose it out of convenience and those who choose it because bytecode looks spooky, not many are left.
To my knowledge, only two recent, good clients using ASM patching exist: Seppuku (although it's 1.12, so it wasn't going to help us), and...
On our own (sort of)
...the client developed by the person who unknowingly taught me how to make a client, fr1kin. rigamortis personally taught me the basics, but I learned everything else from working on our ForgeHax fork.
And, to be sure, fr1kin did something related to what we were shooting for in ForgeHax 1.16. We eventually figured out that his method was not perfect: he applies - just like he used to do in 1.12 - the patches using runtime reflection. That can get really slow. ForgeHax 1.16 has, like, a dozen patches, so it's not really noticeable; but in our ForgeHax 1.12 fork, we had about fifty, and let me tell you, that got slow. There went my original plan of skidding the shit out of whatever he did.
At least, he gave us a nudge in the general direction: it was by looking at his code that we figured out how to use Java Services and that one of Forge's libraries, called ModLauncher, was our ticket to ASM patching. It was very clearly only meant for internal use, but we figured that wouldn't be an issue: it was obviously designed to make integrating whatever new services they came up with easy. In other words, they had no easy way to lock everyone else out.
Figuring out ModLauncher
It took some digging (and reading what little documentation was available) but we figured out what we had to do: an ILaunchPlugin
- called at the very beginning - would just do the trick.
So, apparently all it took to make a service was a META-INF
file to let it know that we were there too, and to implement a pretty simple interface. Neat! Unsure about where to beging with, we figured out that, as the only obligatory one to implement, handlesClass()
was definitely the one who would receive the classes. It expected us to return an EnumSet<Phase>
, so we weren't sure what that was all about, but we put together an initial skeleton that would simply print whatever classes it received.
;
;
;
;
;
;
;
;
Unfortunately, it didn't seem to load us. Why? Well, we figured it out soon enough: we are a Forge mod, so our JAR is loaded way after this takes place. I had an idea: we edited the version JSON and added ourselves in as a library. It worked! We resolved to split the client into two: a patch framework, and the actual client. But, for now, we'd keep it together, for testing.
We then went to sleep, content with what we achieved. We were hopeful, and thought it might just work, for a change.
Minimum Viable Product (MVP)
The next day I went nuclear on it. I figured out that the next step was processClassWithFlags()
. The person working with me had told me that, according to what he'd read, at this stage we'd receive unobfuscted names. As we'll see, this claim came straight from his anus, but we can't fault the man for misreading. The documentation was not exactly clear.
Anyway, this is where the Loader got really big. Ready?
For starters, a quick dive into Mixin's code taught me that there are two different handlesClass()
methods in the interface for retrocompatibility. We want to work with whatever Forge brings in, so I did the same as them:
private static final HashMultimap patchClasses ;
private static final EnumSet YAY ;
private static final EnumSet NAY ;
public EnumSet
public EnumSet
The YAY
/NAY
thing is copied from Forge's Access transformer: apparently you give it a Phase
to mark that your mod is interested in the class, or an empty EnumSet
to say that you don't care about it. To this day I've got no clue about what the exact difference between BEFORE
and AFTER
is, but whatever. If it works, it works.
HashMultimap
is a guava thing, it allows multiple values for each key. I'm using it because, potentially, multiple patches may want to work on the same class. And also for future-proofing: while writing this, I was daydreaming about the possibility of multiple clients using this.
hasAnnotation()
should be self-explanatory, but since it's custom, I'll briefly explain how it works:
private boolean throws IOException
This checks whether a certain class (passed through the fully qualified name) has the @Patch
annotation, which is how we originally recognized patches. The AnnotationChecker is a very simple ClassVisitor I wrote myself:
Without going into much detail - read the ASM documentation if you want to go in depth about how ClassVisitors work - all it does is "visit" a certain class, and set a boolean if it has a certain annotation.
Next, I wrote the actual processClassWithFlags()
thing. Fair warning: as I said, I learned my Minecraft stuff by reading fr1kin's code, so I really really like using Streams.
public int
Explaining what this code does in detail would probably take more than reading it. I'm about to explain the methods, so read those and then go back and read this thing by yourself.
ComputeFlags
is a ModLauncher thing: you return a different value of it depending on what you changed, for efficiency. Honestly, I wasn't really sure at this stage, so I only used the two extremes: COMPUTE_FRAMES
tells it that it needs to rewrite everything, and NO_REWRITE
that I did no change.
Of course, many of the methods called there were made by yours truly.
private Set
This does exactly what the name suggests. It uses the already explained isAnnotationPresent()
method to figure out which of the Class methods have the @Inject
annotation and returns them as a Set
. Simple enough.
public static String
Simply builds a "method descriptor" for a certain method, given its return type and parameters. A descriptor looks something like [[Ljava/lang/Object;
, but if you really want to understand what it is, read the ASM guide I linked before.
private boolean
This is where the magic happens: it processes the given Class, finds the target method, and applies the patch.
And... that's it (or so I thought)! Time to write a simple test patch:
;
;
;
;
;
;
;
;
This stupid patch crashes Minecraft on its first tick. It's very handy to figure out whether it's working: crashes due to NegativeArraySizeExceptions aren't common. And... didn't work. It crashed, but not because of it: it was a ClassNotFoundException.
I did somethings very stupid which I noticed immediately. Can you spot it? It's here in the patch. I'll give you three seconds. Three, two, one... I can't do Minecraft.class
because we are patching before that class is put into the classpath. Grumbling, I minorly altered the code so that it would use the a String
containing the fully qualified name.
Once again: close, but no cigar. It didn't crash, but didn't work either. It took me a while to figure this other one out, but eventually I did. And it wasn't good news: our entire approach was deeply flawed. I learned that handlesClass()
and processClassWithFlags()
are called one after the other for each patch. So, by the time net.minecraft.client.Minecraft
was loaded, our patch was not yet in the HashMultimap
containing them!
By the way, I'd been keeping the other guy documented about my progress. He kept saying "MVP"... at first I thought he was cheering me on, but eventually I realised he meant "Minimum Viable Product", not "Most Valuable Player". He was pushing me to get it working ASAP. Well, this was a big issue, but there was a quick and dirty way to fix it.
I wrote a static method called in the constructor of our Loader - so that it would come before it starts receiving patches:
private static void
Aaaand... nope. This issue drove me crazy, but eventually I figured it out: in Guava 21, the version bundled with forge, the ClassPath
object is marked with @Beta
. I must've missed the convention at which they changed the definition of "Beta" from "experimental" to "doesn't really work at all". I "solved" by pasting the newer, fixed version of ClassPath into our JAR. I really wanted to see if all the stuff I wrote worked properly.
And... it didn't. Rather, it kind of did. It didn't work when I built it, but it did when I tested it with ForgeGradle's runClient
task. Why? runClient doesn't reobfuscate. Apparently, methods are passed with their Searge names. I dwelved deep into Mixin code, failed to use their stuff, and then turned to Forge. However, ObfuscationReflectionHelper.remapName()
didn't work either.
I called it a day, but first updated the co-developer about my progress. I told him that probably Forge does bundle the mapping csv in its JAR, so we could realistically receive it with addResources()
, another method of the ILaunchPlugin
interface.
The rewrite
The next day I didn't do much, and the other guy took over. He spent the whole day researching it, and eventually figured it out: I was right in saying that addResources()
was our way out of this mess, but not in the way I thought. Apparently, that thing also passes us all the Forge JARs: therefore, we could use it to receive our patches. My quick-and-dirty fix was no longer needed.
The obfuscation problem remained unsolved, however, so we opted - for now - to write descriptors and obf name by hand. He deleted most of what I wrote, and rewrote it in a simpler, slimmer way. He likes fors more than Streams, unfortunately, but I suppose that makes the code more readable for those that don't know Java. I don't really see what people that don't know Java could gain from reading our Java plugin for a Java loader in a Java game, but whatever.
I digress. His new system, though slightly uncomfortable for the developer, worked decently enough. Every patch is now a service of its own: you must register the patch in the META-INF as a service implementing our IInjector
interface. We plan to mitigate the unfortunate side effects of this approach with a Gradle plugin or something like that, but we haven't come around to it yet. Meet Lillero-processor, capable of mitigating the considerable amount of boilerplate code needed to run this.
All that was left now was to split the loader from the mod. We called our new creature Lillero - lll
for short - keeping in theme with our other names. Don't worry: you're not ignorant if you don't get it, you're just not an Italian born around the turn of the millennium. You can take a look at its code lillero-loader here.
The more perceptive among you may have noticed that something's missing. And you are correct: we split it into three, not two. After some debating, we figured that we should distribute the "library part" - at the time only the IInjector
- separately from the plugin and client, so that both may implement it as a dependency without bringing in extra stuff. Admire our library - stripped of Javadocs for space constraints:
;
;
;
The Helpers
Patching in itself can, at times, be slightly uncomfortable: with such a low-level language, you often find yourself working with parts that can and should be automated. Working on our ForgeHax fork, we'd been cuddled by the gigantic suite of helpers that fr1kin created (and that I expanded). We need helpers here, so I got to work on writing them.
I suppose this is as good a time as ever to credit fr1kin. I really did write these helpers from scratch, but they ended up looking very similar to an upgraded version of ForgeHax's own. Force of habit, I guess: I'm used to developing on ForgeHax, and I've improved its helpers before. Speaking of that, might do a PR fixing fr1kin's "reverse lookup" todo once I'm done with this.
Anyway, enough premises. Let's get in the code. Rather than pasting it here, I'll link the classes in the repo from now on.
At the time I'm writing this, the helper counts three exceptions - which I'm not going to cover, they do nothing special - and four helper classes. The crown jewel is the PatternMatcher: a class, featuring a fancy builder, capable of finding specific sequences of opcodes. Or also not-specific ones. It's very versatile. And it does reverse lookups - from the end of the method, rather than from the beginning.
What does it return? It returns an InsnSequence. It's an extension of InsnList that does a bunch of fancy tricks. Not sure how useful they are going to be to us or others, but they were pretty fun to code.
Then come the StackTools, a collection of static methods that can do stuff like instantiating objects and creating new local variables in the ASM code. This is perhaps the part most reminiscing of fr1kin's code - but to be fair, there are only so many ways to do these simple tasks, and as I said, I too like Streams.
Finally, the DescriptorBuilder, like the name suggests, tries to spare us the hassle of writing descriptors by hand.
Surprisingly, I got these right on my first try: we tested (some of) them after finishing this, and they worked just fine.
And... that's it, for now. I'll probably make a follow-up if/when we come around to solving the obfuscation problem. But until then, I hope this was helpful to you.
Conclusions
Although highly experimental, our ASM patching API is now in a somewhat usable state. You can find detailed instructions by checking the README of lillero and of lillero-loader. Do note that, while we will keep the old JARs on our maven repo, until they hit version 1.0.0 they are going to be subject to frequent, and probably unannounced, changes.
As soon as it's a bit more stable, we'll put it on GitHub too, so you all can do some pull requests to help us improve. Well, I know that's unrealistic, but a man can dream...
They're now on GitHub: lillero and lillero-loader.