Are you trying to connect to the live game? The live client won’t work on vintage operating systems, it relies on WebSockets and modern encryption that are likely unsupported (both by your OS and by the browser backports).
The webclient should still work on older systems if you are connecting it to a local server, or a server accessible over TCP. For example here’s a community server, you can probably connect to this with a browser running on Windows XP: 2004Scape "Leagues" Community Server
The Java client will also work if you’ve got a JDK to compile it and a JRE to execute it. Where to find these, I’m not sure, presumably there is a dump of old Win32 Java binaries somewhere on the Internet…