Modders are using AI to upscale pre-rendered PS1 backgrounds with phenomenal results

PC modders have a history of fixing and improving ports of Japanese games, and the latest case is one of the coolest, most exciting applications of technology to old games I've ever seen. We recently wrote about how some people are using neural networks to upscale the backgrounds of Final Fantasy 7, and in just a few weeks these projects have made enormous gains in quality and complexity, and for more than just Final Fantasy 7. Anyone can use these tools, and as a result we're at the start of what feels like a renaissance for PlayStation-era 2D art.

A little background: When 3D was in its infancy, PlayStation game developers compensated for the hardware's shortcomings by pairing 3D character models with pre-rendered 2D backgrounds. Many of them, from series like Final Fantasy, are still stunning today: Detailed, stylized, beautiful works of art you could walk through as you went on an adventure. But the art files on those discs were naturally low resolution, and sloppy archival meant that when Square Enix finally ported games like Final Fantasy 8 and 9 to PC in "HD" form, it didn't have the original art files to use for modern high-res displays. Square simply upscaled the low-res files from the original games to 1080p and called it a day.

The results weren't great. The backgrounds can be blurry or oversharpened—just like if you opened a jpeg in Photoshop and tried to blow it up to four times the original size. But what if ~the magic of AI~ could analyze the image and more intelligently resize it? That's what ESRGAN, or Enhanced Super Resolution Generative Adversarial Networks, are all about. OK, that name is a mouthful, and I'm not about to pretend I fully understand how it works. But there's a whole field of study around the concept of perceptual super resolution, and competing AI implementations for how to upscale images with the best results. ESRGAN is one of the most successful. 

Game fans have latched onto this technology and are dialing in how to best use it for low-resolution background images. Here's a blog that walks you through how to install ESRGAN on Windows and shows off a few results. This forum thread compiles even more examples of the AI applied to different game backgrounds and textures.

The most exciting project to me right now is for Final Fantasy 9, one of my favorite games and one with phenomenal art that doesn't look too hot upscaled to 1080p. Modder Ze PilOt is using this technique to upscale FF9's backgrounds and other textures, which is much more involved than upscaling a jpeg: 

So, this nice little tool allows to extract background from the Unity asset files & re-inject them. https://t.co/uBeBQry8Fg. Inside the engine, a background looks like this. pic.twitter.com/5KQLnKt8pJDecember 30, 2018

As he tweeted in December, "every frame of the animation is on a different layer. Lighting is mixed with the background. I need to separate all these frames into single images."

But he's made huge progress already, and is rapidly iterating on a mod implementing these new backgrounds and making other touches, like throwing in the old PS1 font to replace the PC version's bland new one. You can grab version 4 of the mod, Moguri Memoria, here, and follow its development in this forum thread.

Here's a great example of an original texture next to an AI-upscaled one: 

And now, ingame assets & battle fields... pic.twitter.com/yL3kQQxXiFJanuary 25, 2019

These upscales will, of course, never be as good as having the original art assets in high resolution. They'll be blurrier in places and lose detail in others. Purists can argue they distort the original artistic intent, and a poor implementation definitely can. But from what I've seen, these AI-driven upscales are vastly better than the pixelated versions we've seen in some official releases, and could be the best way to modernize games with pre-rendered backgrounds for modern displays.

For now, the easiest targets for these upscales will be the PC ports of older games, but expect to see fans also upscaling backgrounds that can then be injected into games via emulators like Dolphin for the GameCube/Wii and PCSX2 for the PS2. For a long time, emulators have been able to greatly enhance 3D graphics but rendering them in higher resolutions—we're just beginning to see how technology can help us do the same with 2D graphics without destroying the style of the original artwork.