How vtubers inspired me to create a moddable game

Runtime loading, streaming, caching, and mod tooling
← Back

Intro

On New Year's Eve of 2023, I was watching Fuwawa and Mococo, thinking about how lovely it would be if I could create a game where they and other vtubers could apply their creativity, while having fun and entertaining viewers. Suika was at its peak of popularity, so I've decided to recreate it, but with a twist: it would allow anyone to change how the game looks and share it with others.

Unfortunately, when I finished the game, the popularity of Suika genre had already plummeted, and also I didn't have enough courage to write emails to vtubers that I loved watching at the time.

I had no idea how to implement it, other than some basics from modding other games. And when you don't know, this is where the fun part begins!

Mod loading concepts and planning

In the case of games like Skyrim, the default game content is handled like a mod itself that comes bundled with the game. This allows you to change the default game and makes it easier to support the code base, because there's only one point of failure — the modding framework itself.

The first thing you would need is to create a game and a plan for its modding; otherwise you may spend much more time adding modding support than developing an actual game. Trust me, I did exactly that. :grin:

So, I made a game; it works but doesn't support modding. I started wondering what players can change with a mod; for this game it was dozens of things: mod icon, player sprite, container sprite, backgrounds, suika skins, suika icons, suika spawn audio, suika merge audio, songs, and some in-game parameters. Even though the game concept is simple and straightforward, I was able to capture every modding aspect with the exception of runtime code modification.

Runtime sounds and music streaming

Let's start with audio, it's easy, right? Well, it depends...

To start off, there are plethora of audio formats; the game supports only mp3, wav, ogg. Fortunately, Unity handles the aforementioned formats for us, and all we need to do is specify the format based on the file extension.

It's better to read the file format header instead of relying on the file extension alone! 

var audioData = _configLoader.CurrentConfig.MergeSoundsAudios[index];
var extension = Path.GetExtension(audioData.Path);

switch (extension)
{
     case ".mp3":
         await DownloadAndPlayMergeSound(audioData.Path, audioData.Volume, AudioType.MPEG, this.GetCancellationTokenOnDestroy());
         break;
     case ".ogg":
         await DownloadAndPlayMergeSound(audioData.Path, audioData.Volume, AudioType.OGGVORBIS, this.GetCancellationTokenOnDestroy());
         break;
     case ".wav":
         await DownloadAndPlayMergeSound(audioData.Path, audioData.Volume, AudioType.WAV, this.GetCancellationTokenOnDestroy());
         break;
}
private async UniTask DownloadAndPlayMergeSound(string path, float volume, AudioType audioType, CancellationToken cancellationToken)
{
    var webRequest = new UnityWebRequest(path, "GET", new DownloadHandlerAudioClip(path, audioType), null);
    await webRequest.SendWebRequest().WithCancellation(cancellationToken);
    ((DownloadHandlerAudioClip)webRequest.downloadHandler).streamAudio = true;
    var song = DownloadHandlerAudioClip.GetContent(webRequest);
    musicSource.clip = song;
    musicSource.clip.name = path;
    musicSource.volume = volume * _audioSettingsLoader.AudioData.MusicVolume;
    musicSource.Play();
    OnSongChanged?.Invoke();
    webRequest.Dispose();
}

Loading the whole music clip from a file is a time-consuming operation and can only be executed on the main thread, which results in freezes/stutters. Luckily, there's a streamAudio parameter that loads audio in chunks and plays them.

Runtime sprite loading and caching

Audio is done; sprites are on the way. Again, an image can't be loaded as is — it must be compressed; otherwise, it would cause issues like lag spikes and out-of-memory crashes. Also, the way Unity handles sprite creation with texture allocation and reading from a file disturbs the GC. They must be created only once, cached, and reused. 

public static async UniTask<Sprite> CreateSprite(string relativePath, bool compress = true, bool highQuality = false)
{
    var imageBytes = await File.ReadAllBytesAsync(relativePath);
    var texture = new Texture2D(1, 1);
    texture.LoadImage(imageBytes);

    if (compress)
    {
         texture.Compress(highQuality);
    }

    return Sprite.Create(texture, new Rect(0, 0, texture.width, texture.height), new Vector2(0.5f, 0.5f), 256);
}

Cache sprites. Use a helper to create them once, compress when needed, and reuse them instead of rebuilding textures every time.

Runtime collider generation

Great, we have sprites now, but the colliders are still intact. Since shapes of the suikas can differ, let's generate them at runtime too!

Huge kudos to aniketrajnish for the polygon collider optimizer. 

private void AddPolygonCollider(Suika suika)
{
    //Causing lag spikes
    //It's not a noticeable lag, so ignoring it for now
    var collider = suika.SpriteRenderer.gameObject.AddComponent<PolygonCollider2D>();
    var optimizer = suika.PolygonColliderOptimizer;
    optimizer.GetInitPaths(collider);
    optimizer.OptimizePolygonCollider(.01f);
}

Tooling

That's all pretty cool, but how would another player create a mod for the game? Would the player have to read through all of this, the default mod, and manually input all of the JSON values? That would be lame!

Don't worry, I have it covered too! I've created a website that allows you to upload images and audio files, change in-game parameters, and that's all running locally in the browser using vanilla JS!

It does a couple of things:

Notes

Despite Fuwawa and Mococo inspiring me, they are not included in this version of the game, since I didn't make sprites myself :cry:

Due to how the default mod is handled, there are two methods of loading everything: from streaming assets and from local path.

You can play it here.