Skip to content

Running Unpaint on the Xbox Series consoles

Jump to bottom
Péter Major edited this page Jun 10, 2023 · 1 revision

Introduction

Generative AI is envisioned to play a key role in the future of game development. It is already used for content generation to a degree, but further changes will come when games will run generative AI in real-time while the game is running, unlocking many new possibilities.

Currently, this can be done on cloud-hosted servers. Unfortunately the involved cost of running such large models could be prohibitive without subscriptions or other recurring revenue streams.

To address this it is also possible to run such systems locally on existing gaming hardware. This avoids recurring costs for the game developers.

In this example, it will be demonstrated that current console hardware can already run complex image generation AI. While this solution is too heavy to run alongside a high-eng game, it could still be used for many things, such as:

  • generating character portraits in classic CRPG style games
  • applying AI filters in photo mode, which make the scene look like a live-action movie or stylized
  • unique custom patterns generated for cars in a racing game, or for character items in a multiplayer game
  • generating infinite maps, with non-repeating content

For more technical details, please refer to this page.

If you would like to see the solution in action, but would avoid installing locally, you can view this video on youtube.

Notes

Before you proceed please note that this is a technical demonstration. It is for game and machine learning engineers to evaluate technical possibilities. It is not suitable for productivity work or for general use. Dev Mode unlock costs money. If you have a good PC you can get far better experience for free.

The safety features of the app cannot be disabled. Please use the app responsibly. This app is not suitable for children.

Requirements

  • This demonstration works on Xbox Series X and S only, older consoles will not work

  • You will need a PC or a smartphone with web browser on the same network as your Xbox

  • During the setup you will need connection to the Internet

  • You will need to enable Dev Mode on your Xbox, you can follow this guide to do it.

    Keep in mind that dev mode is for development. When I did not use mine for many years, it had been suspended.

    I had to contact Microsoft to reenable it, and while they quickly reactivated it without any hassle, I was asked to demonstrate some presence in the Microsoft Store or it might be suspended again.

  • While dev mode is active you will not have access to your retail apps and games, do not worry, they are not lost or anything, but you will need to switch back to retail mode to play your games again, switching requires a reboot.

Features and limitations

  • You can use text-to-image, image-to-image and in-painting pipelines, all desktop features are included
  • Image generation is currently limited to 512 x 512 pixels, higher resolutions will fail - further optimizations will be required
  • The inference on Series S is noticeably slower than on Series X, but it seems to work the same

    I think this is since UWP apps on Xbox are not allowed to use all the available memory. So essentially both consoles will be able to do the same things, but X will be faster.

  • I have changed the app so it will start with suitable settings.

    You may change these, but that might cause the app to fail. If this happens you might need to restart the app to be able to generate images again.

  • UWP apps on Xbox do NOT support the mouse for input, so will need to settle with a controller and keyboard (on the other hand you can refine your controller aiming skills with in-painting)
  • While running the inference the console UI will lag somewhat.

Installation

Getting the installers

  • Download the latest Unpaint installer (the MSIX file) here.
  • Download the Microsoft.UI.Xaml package - this contains the WinUI GUI framework used by Unpaint.

Preparing your Xbox

  • Switch to dev mode

  • Once you are in dev mode, ensure that your Xbox is connected to the Internet

  • If you see the regular Xbox home screen, open Dev home, if not your Xbox probably started into Dev home already

    Dev Home usually appears on your Xbox home screen, where the ads are in retail mode

    If not, open up Settings, select the developer tab, and click developer home

  • In dev home go to the bottom right corner and select remote access settings and open it

  • On the authentication section set a username and password (or select that you do not require authentication)

  • Go back to dev home

  • Note down the URL in the bottom right corner, e.g. https://192.168.1.73:11443/

    note it has to have the port number, and has to be https

Installing the Unpaint

  • Open the browser on your PC / smartphone

  • Enter the URL you noted down above

  • You will get certificate warning, select to ignore and go forward

    You need to click advanced, and then Continue to <your xbox IP> (unsafe)

    This is not tied to this particular demo, it is just how dev mode works. It is completely safe.

  • On the screen you will see the Home tab, with the My games & apps section, click the Add button below that

  • Now get the unpaint installer (msix file), and provide it to the app installation wizard, press next

    Drop it onto the window or click browse and select it

  • Now we will have to add the dependencies, of which there is only one: the Microsoft.UI.Xaml appx you downloaded above

  • Click next and the app will be installed

Launching the app

  • Go to your home screen and select Unpaint
  • If you have not done so, you will be asked to sign in to your Xbox Live account
  • Now Unpaint will start, use the positive and negative prompts and the play (generate) button to generate images
Clone this wiki locally