It takes a lot of effort to do this right. We want to preserve the game correctly - no one should ever have to do this again.
If you aren’t familiar with our approach or programming in general it may seem super straightforward to go from May to June 2004. In some ways it is, but it is a long checklist of items whenever there are engine updates involved.
We unpack and pack everything from source using our data tools. This has a nice side effect of allowing anyone to jump in to create your own custom content easily too. Most importantly it allows everyone to understand how the game worked and teach the generations after us.
Let me explain what 244’s process involved so you understand better
Archive, preserve, restore! Users need to check their old PCs for as much data as possible. We are missing files for many versions, lots of lost history. Before we can work on a version we need the client archived and enough of the cache archived+restored. Caches weren’t 100% preloaded back then, but identical versions from different PC backups can be combined to get closer to 100%. Everything from RSPS has been sourced already!
Decompile, deobfuscate, rename, and refactor the Java client. I take the original applet and turn it into human readable Java code again. This process can take days for the bulk of the work and weeks to finish polishing it (avoiding burnout)
Write tools to unpack the cache into a human-editable format. Transitioning from 225 to 244, there were some internal changes to learn, but the hardest part here is retaining our existing work. Unpacking has to result in a seamlessly updated cache since 225 is complete and we don’t want to lose anything - remove what was removed in 244, update what was updated in 244, and add what was added in 244.
Write tools to pack the cache back into the original formats 1:1. That 1:1 part is what takes the longest to get right, I have to triple check everything and understand why every byte in a file exists.
Update the server with any protocol changes observed in the client. Each version scrambles the network protocol (Jagex’s anti-bot) and in 244’s case, a new “OnDemand” cache delivery system was added.
Update the webclient by porting the refactored Java code to TypeScript.
Write the content and engine changes! This takes time to research and confirm exact behavior. Lots of time finding and watching videos, finding screenshots, reading forum posts, and playing live RS where applicable.
The tools are time consuming upfront but work for a range of versions, they only require a little time investment over time as new features get added to the clients.
Revisions 185-225 (Dec 2003 - May 2004) use the same data tools.
Revisions 226-377 (June 2004 - May 2006) use the same data tools.
Revisions 400+ use the same data tools.
We only have to focus on minor updates to the data tools for a while now
tl;dr Reverse engineering the original applet and cache gets us a runnable client and compatible server. Next we unpack the cache into usable formats and pack it back into the original to use it again. Finally we recreate content and engine behaviors based on what we can observe and measure.
P.S. no AI gibberish here! don’t let prompts take em-dash away from us humans.
First, much love for both making and hosting this server
Second, will there be an easy way for new contributors to pick up tasks once the tools for 226+ get released? I.e. if I want to write Mage Training Arena, will there be an easy way for me to contribute that content even if the main branch is only at 317 by this point?
I plan on having a couple more branches available to encourage contributors to jump in lots of people want to jump in to contribute but working fully incrementally means we don’t have enough tasks to give out!
There will be the “main” branch that’s incrementally preservation oriented, and the live server will mirror it after each completed version. We can have a 2005+2006 revision branch (ie 317 and 377) no problem, and my role as a maintainer means I’ll handle synchronizing work between versions as necessary.
You won’t have to wait for live to reach Jan 2006 to get started on MTA, basically. Just wait until the 2006 branch is up and be sure to chat with us to learn the right process to get your PRs accepted.
Read this when it dropped but needed to digest for a while so I could distill my thoughts. Here’s what I came up with that I wanted to know the most:
Is there any definitive tutorial-style documentation on the more fundamental parts of creating the data tools? I’d imagine this involves understanding both the client and the cache at a pretty low level.
I think a big issue with the RSPS scene is the built-up cruft around documentation. Lost City has the final-product software part done, and LC-specific setup/bootstrap documentation is solid, but “how” things are built is still opaque to me. Is anything historical on Rune-Server relevant for this?
For example, steps 2 and 3 in your OP – you know you have a 244 cache and you know you need to be able to pack and unpack it. How do you go from there all the way to a working tool? What parts of that process are reasonable to explain in-depth for someone with little-to-moderate domain-specific knowledge?
My RS knowledge is built mostly from extrapolation and guess-and-check. Time and motivation can make things more efficient but it’d be nice if there was a definitive reference so I knew I could progress through understanding A->B->C with code samples and runnable examples along the way.
I’m thinking longform written tutorial articles where caveats are notated, error cases covered, and alternative routes explained. I’ve got enough oomph to do some of this for RuneScript and some of the high-level stuff, but other areas are a mystery to me.
anyways, what you’re thinking is too short sighted. there basically needs to be what people call a “white paper” on RuneScape and how it works. this can be done in many different ways. i have all of it in my head but it all needs to come out in something comprehensive that covers all aspects of the game. it sounds easy but it’s a lot of work to write all the shit down.
anyways someone at the minimum just needs to document every single thing about the code on the engine, atleast thats a start.
Haven’t been playing lately as I’ve been busy irl and went too hard after the release and sort of burnt out on the limited content, but just want to say thank you for preserving history, especially for free. I’ve sought this for many years and it’s hard to believe that someone has actually done what you’ve done, and to not have given up on it either.
don’t let prompts take em-dash away from us humans.