Skip to content

Home

Jump to bottom
mos9527 edited this page Dec 6, 2024 · 46 revisions

Preface

DISCLAIMER: This tool is not to be used for commercial purposes, nor for redistributing or exposing the original game’s protected assets in any form that violates intellectual property laws.

Ensure any content created using this tool complies with applicable copyright laws in your region.

Properly credit the original creators for any derivative work. This tool and its creator assume no responsibility for misuse or infringement.

And as always, update your installation and read the latest help text (sssekai -h) before moving on.

In Game Assets

The game loads its assets on-demand.

However, you can use abcache to download selected assets offline, instead of extracting it from your own device.

Notes

  • ATTENTION: Only the latest version information are guaranteed to enable access the game's asset repository. Older ones may work, but is not offically supported.
  • You'll need the game's version number (i.e. 4.1.0), region (i.e. jp) and its app hash (i.e. for jp, 4.1.0 it's cbda2f12-c804-4163-5e11-3148631f9ab0) to download the assets. See the table below for details.
  • The files may take up a lot of storage space (e.g. it's ~70GB for 3.8.1). Make sure enough is reserved since there's no free space checks in the script

Game Versions, Regions And Their Hashes

  • ATTENTION: These values could be out-of-date. It's advised that you'd pull these values yourself.

    • This is explained in the next section.

  • The platform (i.e. ios or android) is omitted since the values for them are the same.

  • As stated earlier, only the latest (as the wiki is updated) values are stored here.

region version appHash
jp 4.1.1 41fd71f2-f715-bc10-5852-0a9d8542f760
en 3.1.0 a892dc93-798e-4007-8d07-54cb13c9500a
tw 3.4.0 a3015fe8-785f-27e1-fb8b-546a23c82c1f
kr 2.8.0 4d9acca8-553f-c3f4-398b-a678e32e7f85
cn 3.4.0 a3015fe8-785f-27e1-fb8b-546a23c82c1f

Bonus: How to extract appHash from the Android releases

With sssekai apphash (i.e. the apphash subcommand) you can automatically determine the game's version and appHash from an Android distribution of the game's binary, from any region. (i.e. JP,EN,TW,KR,CN) The output (if successful) would be a tuplet containing [game version] [game region] [game appHash], seperated by space.

  • With a pre-exisiting game package (APK/XAPK supported)
$ python -m sssekai apphash -s en_310.xapk
3.1.0 en a892dc93-798e-4007-8d07-54cb13c9500a
  • Download the latest package (for JP only) online w/o writing it to disk
$ python -m sssekai apphash
4.1.1 jp 41fd71f2-f715-bc10-5852-0a9d8542f760

Downloading the assets

NOTE: Please read the latest sssekai abcache -h help text before moving on.

Updating the metadata cache

  • ATTENTION: It's advised to perform the cache index update when sssekai is updated itself. DO NOT use others' cache index, ALWAYS prepare it on your own machine as this could otherwise lead to arbitrary code execution.
  • AbCache by itself doesn't store (nor check) the downloaded bundles themselves. Only the asset index is stored.
  • This is also needed to download and resolve asset dependencies on demand, which is not unlike how the game handles this itself.
  • Example: Updating and saving the metadata cache to /.sssekai/abcache.db 
sssekai abcache --db ~/.sssekai/abcache.db  --app-region ... --app-version ... --app-appHash ...
  • Note that --app-region, --app-version and --app-appHash are required when (and only when) updating the cache index. (i.e. not loading from an existing one)
  • Loading from a existing cache index (which contains these values) supercedes these arguments

Using AbCache to download bundle files

The metadata cache may be updated everytime. In practice, however, it's advised to load the previously cached metadata to avoid extra data/time cost.

  • To load the previously cachced data, you can specify --no-update to skip the update process.
  • You can specify where to store the downloaded data with --download-dir
  • Example: Downloading all available game assets, using a pre-cached metadata set
sssekai abcache --db ~/.sssekai/abcache.db --no-update --download-dir ~/.sssekai/bundles/

In most cases you probably only want a subset of the game's files. You can filter which bundles to download with --download-filter, which takes a regex as its input

  • In this case, the bundles themselves may require other bundles (i.e. dependencies) as well. To automatically resolve this issue, use --download-ensure-deps, which would download them alongside the selected ones.
  • Example: Download all v2 PV models for character 31 (25-ji Miku), including the dependencies
sssekai abcache --no-update --db ~/.sssekai/abcache.db --download-filter .*characterv2/.*/31.* --download-ensure-deps --download-dir ~/.sssekai/bundles

The downloaded bundles are obfuscation-free, meaning you can load them (e.g. by folder) directly into Unity bundle loaders

Exploring the Cache

abserve offers both a simple Web UI and FUSE backend implementation for exploring the assets hierarchy - and downloading them from the respective interfaces directly.

  • To use the Web UI

    sssekai abserve --db <your abcache database file> --host <ip address to listen on> --port <port number to listen on>

    The Web interface borrowed that from the de-facto python -m http.server (i.e. pure HTML and basically not much to look at). Files can be downloaded directly whilst being decrypted on the fly - without caching them to disk first.

  • To use the FUSE interface

    ATTENTION: You'll need a system that supports libfuse and fusepy natively (e.g. with Linux & macOS). Windows support is not yet implemented by upstream.

    sssekai abserve --db <your abcache database file> --fuse <your AbCache mount point>

    The mount point will then serve as a regular read-only filesystem. Again, the game files will be decrypted on the fly.

    /mnt/abcache » ls -altr
total 4
drwxr-xr-x 8 root    root       4096 Nov 21 20:46 ..
drwxrwxrwx 0 mos9527 mos9527       0 Nov 21 21:31 .
drwxrwxrwx 0 mos9527 mos9527       0 Nov 21 21:31 actionset
drwxrwxrwx 0 mos9527 mos9527       0 Nov 21 21:31 area
drwxrwxrwx 0 mos9527 mos9527       0 Nov 21 21:31 area_image
drwxrwxrwx 0 mos9527 mos9527       0 Nov 21 21:31 area_sd
drwxrwxrwx 0 mos9527 mos9527       0 Nov 21 21:31 areaitem
drwxrwxrwx 0 mos9527 mos9527       0 Nov 21 21:31 bonds_honor
drwxrwxrwx 0 mos9527 mos9527       0 Nov 21 21:31 campaign
drwxrwxrwx 0 mos9527 mos9527       0 Nov 21 21:31 character
drwxrwxrwx 0 mos9527 mos9527       0 Nov 21 21:31 character_archive
drwxrwxrwx 0 mos9527 mos9527       0 Nov 21 21:31 comic
drwxrwxrwx 0 mos9527 mos9527       0 Nov 21 21:31 crystal_shop
drwxrwxrwx 0 mos9527 mos9527       0 Nov 21 21:31 custom_profile
drwxrwxrwx 0 mos9527 mos9527       0 Nov 21 21:31 effect_asset
drwxrwxrwx 0 mos9527 mos9527       0 Nov 21 21:31 event
drwxrwxrwx 0 mos9527 mos9527       0 Nov 21 21:31 event_story
drwxrwxrwx 0 mos9527 mos9527       0 Nov 21 21:31 exchange
drwxrwxrwx 0 mos9527 mos9527       0 Nov 21 21:31 fix_prefab
drwxrwxrwx 0 mos9527 mos9527       0 Nov 21 21:31 font
drwxrwxrwx 0 mos9527 mos9527       0 Nov 21 21:31 gacha
drwxrwxrwx 0 mos9527 mos9527       0 Nov 21 21:31 home
drwxrwxrwx 0 mos9527 mos9527       0 Nov 21 21:31 honor
drwxrwxrwx 0 mos9527 mos9527       0 Nov 21 21:31 honor_frame
drwxrwxrwx 0 mos9527 mos9527       0 Nov 21 21:31 live
drwxrwxrwx 0 mos9527 mos9527       0 Nov 21 21:31 live2d
drwxrwxrwx 0 mos9527 mos9527       0 Nov 21 21:31 live_pv
drwxrwxrwx 0 mos9527 mos9527       0 Nov 21 21:31 livetalk
drwxrwxrwx 0 mos9527 mos9527       0 Nov 21 21:31 loginbonus
drwxrwxrwx 0 mos9527 mos9527       0 Nov 21 21:31 lottery_game
drwxrwxrwx 0 mos9527 mos9527       0 Nov 21 21:31 mission
drwxrwxrwx 0 mos9527 mos9527       0 Nov 21 21:31 model3d
drwxrwxrwx 0 mos9527 mos9527       0 Nov 21 21:31 movie
drwxrwxrwx 0 mos9527 mos9527       0 Nov 21 21:31 multi_room
drwxrwxrwx 0 mos9527 mos9527       0 Nov 21 21:31 music
drwxrwxrwx 0 mos9527 mos9527       0 Nov 21 21:31 paid_virtual_live
drwxrwxrwx 0 mos9527 mos9527       0 Nov 21 21:31 rank_live
drwxrwxrwx 0 mos9527 mos9527       0 Nov 21 21:31 rank_match
drwxrwxrwx 0 mos9527 mos9527       0 Nov 21 21:31 scenario
drwxrwxrwx 0 mos9527 mos9527       0 Nov 21 21:31 shader
drwxrwxrwx 0 mos9527 mos9527       0 Nov 21 21:31 share
drwxrwxrwx 0 mos9527 mos9527       0 Nov 21 21:31 sound
drwxrwxrwx 0 mos9527 mos9527       0 Nov 21 21:31 stamp
-rwxrwxrwx 1 mos9527 mos9527  148411 Nov 21 21:31 stamp_balloon
drwxrwxrwx 0 mos9527 mos9527       0 Nov 21 21:31 stamp_mission
-rwxrwxrwx 1 mos9527 mos9527   55118 Nov 21 21:31 stamp_mission_campaign_home_button_icons
drwxrwxrwx 0 mos9527 mos9527       0 Nov 21 21:31 story
drwxrwxrwx 0 mos9527 mos9527       0 Nov 21 21:31 streaming_live
drwxrwxrwx 0 mos9527 mos9527       0 Nov 21 21:31 temp
drwxrwxrwx 0 mos9527 mos9527       0 Nov 21 21:31 thumbnail
drwxrwxrwx 0 mos9527 mos9527       0 Nov 21 21:31 title_screen
drwxrwxrwx 0 mos9527 mos9527       0 Nov 21 21:31 tutorial
drwxrwxrwx 0 mos9527 mos9527       0 Nov 21 21:31 ui
drwxrwxrwx 0 mos9527 mos9527       0 Nov 21 21:31 unit
-rwxrwxrwx 1 mos9527 mos9527 4360943 Nov 21 21:31 unit_background
drwxrwxrwx 0 mos9527 mos9527       0 Nov 21 21:31 unit_story
drwxrwxrwx 0 mos9527 mos9527       0 Nov 21 21:31 virtual_live
drwxrwxrwx 0 mos9527 mos9527       0 Nov 21 21:31 virtual_live_booth_shop
drwxrwxrwx 0 mos9527 mos9527       0 Nov 21 21:31 worldmap

Note on Unity Version

  • You should specify the Unity version to use since it's stripped from the asset bundles - if you'd load them into other Unity bundle loaders.
Game Version (Region,Platform agnostic) Unity Version
<3.6.0 2020.3.21f1
>=3.6.0 2022.3.21f1
  • Another heuristic for determining Unity version is whether the game has received the new UI update or not. If it has - try 2022.3.21f1
  • In sssekai, you can specify the Unity version to use with sssekai --unity-version ... <rest of your command>
    • This is defaulted to 2022.3.21f1 otherwise.

Using the assets

ATTENTION: Ensure the correct Unity version is selected for your bundle files (see section above).

Streaming Live (RLA) segments

  • sssekai rla2json converts what are colloquially called "RLA,RTVL" segments into JSON.

  • These segments can be found in streamling_live bundles, plus the "Virtual Live" events (with streamed performance) the JP server hosts from time to time.

  • The parser also accepts ZIP archives containing sekai.rlh sekai_xx_xxxxxx.rla... files - this is useful if you'd captured SSE packets for the said Virtual Live event and want to parse it with the tool.

Addendum: A short snippet demonstrating how such ZIP archives may be created

import os, sys
from io import BytesIO
from base64 import b64encode
from sssekai.fmt.rla import (
    decode_buffer_base64,
    decode_buffer_payload,
    decode_streaming_data,
)
from sssekai.fmt.rla import SSEDataLengthOutOfRangeException

VALIDATE = False
RLA_VERSION = (1, 5)
if len(sys.argv) != 3:
    print("Usage: python make_streaming_archive.py <directory> <out zipfile name>")
    print(
        "Packages loose SSE packets into an archival format (zip), for use with sssekai and  sssekai_blender_io ONLY"
    )
    print(
        "       * the packets should be in its base64-esque form (or accidentally decoded. this script will try the best effort to recover the otherwise corrupted data in this case)"
    )
    print(
        "       * note that the directory should contain only the sse data packets. the hierarchy of the directory is not important."
    )
    print(
        "         - however, the files should be named as <unix timestamp>-<data>.bin"
    )
    print(
        "       * the output file will be a zip file containing the rla files and a rlh file, mimicking the structure of the game's streaming archive"
    )
    print("       * and, no, the files won't work with the game itself^^")
    sys.exit(1)

ROOT = sys.argv[1]
OUTFILE = sys.argv[2]
MAGNITUDE = 1e7

paths = dict()
for path, directories, files in os.walk(ROOT):
    for file in files:
        decoded_data = file.split("-")
        if len(decoded_data) == 2:
            ts, decoded_data = decoded_data
            paths[int(ts)] = os.path.join(path, file)
paths = sorted(paths.items(), key=lambda x: x[0])
ts_start, ts_end = paths[0][0], paths[-1][0]
print("* effective duration %.3fs" % ((ts_end - ts_start) / 1000))

base_tick = 1e18
STREAM_SEGMENT_SIZE = 100

rla_segments = []
rla_stream = BytesIO()
rla_stream_current_size = 0

write_int = lambda value, nbytes: rla_stream.write(value.to_bytes(nbytes, "little"))
write_buffer = lambda value: rla_stream.write(value)

num_motion_data = 0
for idx, (ts, path) in enumerate(paths):
    data = open(path, "rb").read()
    if not data[:4] == b"RTVL":
        data = b64encode(data)
    print("* reading %s %08d/%08d" % ("/|-"[idx % 3], idx, len(paths)), end="\r")
    try:
        header_signature, decoded_data = decode_buffer_base64(data)
        if VALIDATE:
            decoder_signature, decoded_data = decode_buffer_payload(decoded_data)
            assert (
                header_signature == decoder_signature
            ), "mismatching signature (header/decoder). packet may be corrupt"
            payload = decode_streaming_data(
                RLA_VERSION, decoder_signature, decoded_data, True
            )
            pass
    except SSEDataLengthOutOfRangeException as e:
        if e.needed > e.current:
            data += b"A" * (e.needed - e.current)
        else:
            data = data[: -(e.current - e.needed)]
        header_signature, decoded_data = decode_buffer_base64(data)
    # We now have a valid data field that can be decoded by ((our)) decoder
    if header_signature == 1:  # MotionCaptureData
        if not num_motion_data:
            # With the first 'motion data' we can extract its starting tick
            decoder_signature, decoded_data = decode_buffer_payload(decoded_data)
            payload = decode_streaming_data(
                RLA_VERSION, decoder_signature, decoded_data, strict=False
            )
            base_tick = payload["data"][0]["timestamp"]
            print("* base tick %d" % base_tick)
            num_motion_data += 1
    tick = int(base_tick + (ts - ts_start) / 1000 * MAGNITUDE)
    if not data[:4] == b"RTVL":
        print("* invalid packet %s" % path)
        continue
    write_int(tick, 8)
    write_int(len(data), 4)
    write_buffer(data)
    rla_stream_current_size += 1
    if rla_stream_current_size == STREAM_SEGMENT_SIZE:
        rla_segments.append(rla_stream.getvalue())
        rla_stream = BytesIO()
        rla_stream_current_size = 0

if rla_stream_current_size > 0:
    rla_segments.append(rla_stream.getvalue())
rlh_header = {
    "baseTicks": base_tick,
    "version": ".".join(map(str, RLA_VERSION)),
    "splitSeconds": 0,
    "splitFileIds": list(range(len(rla_segments))),
}

import zipfile, json

with zipfile.ZipFile(OUTFILE, "w") as z:
    z.writestr("sekai.rlh", json.dumps(rlh_header))
    for idx, segment in enumerate(rla_segments):
        print("* packing %s %08d/%08d" % ("/|-"[idx % 3], idx, len(segment)), end="\r")
        z.writestr("sekai_%02d_%08d.rla" % (0, idx), segment)

Usage:

  • Convert all RLA packets into separate JSON files
sssekai rla2json streaming_live/archive/1st_live_vbs-1 streaming_live/archive/1st_live_vbs-1_src
  • Dump audio data (in CriWare HCA segments/frames) into separate frames in binary.
sssekai rla2json streaming_live/archive/1st_live_vbs-1 streaming_live/archive/1st_live_vbs-1_hca

NOTE: The resultant binaries may be further decoded with sssekai_streaming_hca_decoder into WAV format. Once you have it installed:

sssekai_streaming_hca_decoder streaming_live/archive/1st_live_vbs-1_src streaming_live/archive/1st_live_vbs-1.wav

You will find a single, large WAV file (44100Hz, 1ch, 128kbps) located at streaming_live/archive/1st_live_vbs-1.wav

Path Purpose (Assumed)
streaming_live/archive Archival bundles for past Virtual Live shows. Contains chat log, character animations (skeletal/blendshape),streamed audio, etc. See 3D Assets for more info.

Spine2D Related Files

NOTE: You'll need Spine2D SDK 4.x to load these.

Path Purpose (Assumed)
area_sd Spine2D model files Can be extracted with sssekai spineextract

CriWare Related Files

NOTE: USM/HCA files, mainly. For the videos usmdemux can handle it by itself. For audio tracks try vgmstream

Path Purpose (Assumed)
live/2dmode/original_mv Video, Original PV/MVs for selected tracks. Can be extracted with sssekai usmdemux. Filenames are the MV IDs which can be queried by mvdata.
live/2dmode/sekai_mv Video, PV/MVs tailor made for proseka, for selected tracks. Can be extracted with sssekai usmdemux Filenames are the MV IDs which can be queried by mvdata.
music/short Audio, Preview track for the songs.
music/long Audio, Songs that plays in rhythm game or PV viewer.
music/jacket Images, The cover art for the songs.

SUS (Chart) Files

You can find them in music/music_score in SUS Format

OSS Viewers exist for this format. For example:

Live2D Related Files

NOTE: Cubism's Live2D edtior comes with a viewer which you can use to view these things

Path Purpose (Assumed)
live2d/model Cubism Live2D model files. Can be extracted with sssekai live2dextract
live2d/motion Animation files for the models. Again, Can be extracted with sssekai live2dextract

3D Assets

NOTE: Assets flagged with * can be imported by sssekai_blender_io into Blender

Path Purpose (Assumed)
live_pv/model 3D Assets for real-time rendered PVs
live_pv/model/character *Character models
live_pv/model/characterv2 *Also character models. (w/ slightly more modern NPR techniques e.g. SDF Face shadows, eye specular and so on)
live_pv/model/stage *Stage models.
live_pv/model/stage_decoration *Stage decoration models (of course). Per-PV. Can be queried by mvdata
live_pv/model/music_item *PV Props. Bone weights are missing for some reason.
model3d/model/stage_object *PV Props, again. With the same aforementioned issues.
live_pv/timeline/... *Animation data for character, stage objects, camera, and more.
streaming_live/archive *Streaming Live (Virtual Live) recording archives, contains character motion data & shapekey expressions, and more.

Game Master Data

Master data (hereby referred to as DB) is the game's monolithic database for states,events,etc.

  • You can download the DB with the following command:
sssekai abcache --app-region ... --app-version ... --app-appHash ... --dump-master-data <directory name>

The files (in JSON format) will be stored in the specified directory, in a structure similar to https://github.com/Sekai-World/sekai-master-db-diff

Clone this wiki locally