GkmasObjectManager GUI v1.4

A Python OOP interface with the asset database in Gakuen Idolm@ster.

Changelog

• Backend
  • [Feature] Implement rich download progress bar
    • _reporter is now a member of GkmasResource
  • [Fix/Media] Concatenate subsongs in segmented .awb BGM
  • [Fix/Media] Tolerate alpha extrema = (254, 255) in RGBA -> RGB optimization
  • [Fix/Media] Use -movflags faststart in .mp4 video conversion
  • [Style] Clean up getters with 'property' and 'staticmethod' class method decorators
  • [Style/Media] Switch to native subprocess call, remove dependency on 'ffmpeg-python'
  • [Docs] Add type annotation for all class attributes
• Frontend
  • [Feature/View] Forward download progress to upstream queue, expose through SSE
    • Handled by utils.progressedMediaDriver(), which requires a rigid class structure in media container
  • [Feature] Display caption in .acb archive through caption_map API
    • Also factor out command cleanup logic in sovits_dataset to utils.make_caption_map()
  • [Feature] Switch to "blinking character" loading animation
  • [Fix] Switch CSS unit from vh to rem for smooth zooming
  • [Fix] Unify top margin (past navbar) of page elements
  • [Fix/Search] Scroll to top at page/sortState change
  • [Style] Merge 'assetbundle' and 'resource' route handlers in Flask server

Development Roadmap

• Backend
  • [Manifest] Export as database
  • [View] Convert models to .glb
  • [View] Spine animation renderer
  • [Adv] Story context parser
  • [Docs] Deploy automated test
  • [Docs] img file structure in README
  • Adaptation to CGSS, MLTD, SGSC solutions
• Frontend
  • [Home] "Manifest Management" panel
  • [View] Embedded Unity model preview (powered by mari-mari-dance and glTFast)
  • [View] Display adventure digest
  • [Adv] GkmasAdventurer storyline editor (WebGAL adaptation)

GkmasObjectManager GUI v1.3

A Python OOP interface with Gakuen Idolm@ster's object database ("manifest").

Changelog

• Backend
  • [Refactor/Object] Include modification time in get_media()
  • [Refactor/Media] Accept lazy downloader from Object, triggered by _get_raw()
  • [Feature/Manifest] In object search, add "sort by" and ascending/descending options
  • [Feature/Media] Check for existing file with media-specific extension
    • Additionally check for existing .zip for potential .acb archives (unsure without raw bytes)
  • [Feature/Media] Automatically remove redundant alpha channels in images
  • [Fix/Manifest] Cache instantiated objects in ObjectList
  • [Fix/Media] Fallback to "fixed default" instead of "last" converted format when unspecified
  • [Fix/Media] Cache and check for image_resize to avoid skipped conversion in same format
  • [Fix/Media] Fallback to PNG instead of converting to RGB for no-alpha image formats
  • [Fix/Media] Remove inapplicable ?n.wav in vgmstream conversion for .awb audio
  • [Fix/Media] Remove empty_moov movflag in .usm conversion to allow WMP seek
  • [Style/Manifest] Remove hardcoded manifest urlFormat and exact resource info fields
    • Driven by IDOLY PRIDE adaptation
  • [Style/Media] Remove deprecated get_embed_url()
  • [Style] In sovits_dataset, set ffmpeg loglevel to fatal, suppress config dump
  • [Style] Use Pylance to ensure type coverage in function signatures
  • [Style] Fix PyLint warnings
    • const - Deprecated IMAGE_RESIZE_ARGTYPE
    • manifest - Catch JSONDecodeError, (protobuf) DecodeError
    • manifest/manifest - Catch (JSON) TypeError, (protobuf) ParseError; specify file encoding
    • manifest/listing - Set default this_repr; use url_format in sub/add reinstantiation
    • object/resource - Use response.raise_for_status()
    • media/image - Use isinstance() for type check
  • [Format] Reorder imports with isort (Black formatted compatible mode)
  • [Docs] Write missing docstrings, fix long lines
• Frontend
  • [Fix] Fine-tune flex layout properties; unify title margin
  • [Style] Switch to native Bootstrap classes wherever possible

Development Roadmap

• Backend
  • [Manifest] Export as database
  • [Adv] Story context/alias parser
  • [Docs] Deploy automated test [NEW]
  • [Docs] img/sud file structure in README
  • Adaptation to CGSS, MLTD, SGSC solutions [NEW]
• Frontend
  • [Home] 'Manifest Load/Export' buttons
  • [Search] Fixed size loading placeholders [NEW]
  • [Search] Filter search by assetbundle-/resource- only
  • [View] Embedded Unity model preview (powered by mari-mari-dance)
  • [View] Progress bars for downloading, MD5 checksum, and ffmpeg conversion
  • [View] Display adventure digest; display voice caption [NEW]
  • [Adv] GkmasAdventurer storyline editor (WebGAL adaptation)
  • Deploy on GCP (may involve high-concurrency middleware)

GkmasObjectManager GUI v1.2

A Python OOP interface with Gakuen Idolm@ster's object database ("manifest").

Running the Flask GUI

$ pip install -r requirements.txt
$ python server.py

Using the Python Module

import GkmasObjectManager as gom

m = gom.fetch(pc=True)  # live loading from server
m.export("manifest.json")  # supports ProtoDB/JSON/CSV

m_old = gom.load("octocacheevai")  # loading from local 'octocacheevai'
m_diff = gom.fetch(m_old.revision._get_canon_repr())  # fetch update
m_new = m_old + m_diff  # apply update
assert m_new == gom.fetch()

m.download_preset("presets/namecard_kit.yml")  # implies export dir

m.download(
    "sud_vo_adv_unit.*",
    path="mainstory_voicelines",
    convert_audio=True,  # default, may omit
    audio_format="mp3",  # applies to all subsongs, may take a while
    unpack_subsongs=True,  # unpack ZIP to output directory
)
m.download("adv_unit.*", path="mainstory_scripts")  # auto converts to JSON

Using the SoVITS Dataset Maker

$ python sovits_dataset.py -h
usage: sovits_dataset.py [-h] [-o OUTPUT] [-f FORMAT] [-b BITRATE] [-c] [-m] [-g] [-d CACHE_DIR] [-p] character

Creates a dataset for training a voice cloning model.

positional arguments:
  character                 Character name

options:
  --output OUTPUT           Output filename, default to 'sovits_dataset_v{REVISION}_{CHARACTER}[_greedy][_captioned].{FORMAT}'
  --format FORMAT           Output format, default to .wav
  --bitrate BITRATE         Output bitrate, default to 128kbps
  --caption                 Include captions
  --merge                   Merge dataset into one audio file (otherwise export as ZIP)
  --greedy                  Search *beyond* adventures with character abbrev in filename (e.g., appearance in others' stories)
  --cache-dir CACHE_DIR     Cache directory, default to ./.sovits-cache
  --purge-cache             Clean up cache after use

$ python sovits_dataset.py fktn -d D:/tmp/gkmas/.sovits-cache -g -c -m -f mp3
- Character is Fujita KoToNe
- [G]reedily cache all voice packs
- Include adventure [c]aption
- [M]erge samples into one file
- Output [f]ormat is .mp3

Character Abbreviations

Hanami SaKI	Tsukimura TeMaRi	Fujita KoToNe	Arimura MAO	Katsuragi LiLJa	Kuramoto ChiNA
Shiun SuMiKa	Shinosawa HiRO	Himesaki RiNaMi	Hanami UME	Hataya MiSuZu	Juo SeNA
Amaya TsuBaMe	Juo KuNiO	Neo ASaRi	VOcal TRainer	DAnce TRainer	VIsual TRainer
Kayo RiNHa	Aoi NaDeshiKo	Shirakusa ShiON	Shirakusa GekKA	Kuroi TaKaO

Changelog

• Backend
  • [Feature/Manifest] Live manifest fetch of PC version by DMM GAMES
  • [Feature/Manifest] Manifest add (for applying diff patches)
  • [Feature/Manifest] YAML downloading scheme presets
  • [Feature/Object] Record 'Last-Modified' at asset request; reflect in mtime of exported files and frontend
  • [Feature/Media] Extract multiple subsongs from voiceline ACB, export as ZIP, and optionally unpack
  • [Feature/Adv] Adventure script parser (courtesy of Hatsuboshi_Teleprompter, Gakumas-Commu-Parser)
  • [Feature] SoVITS- and diff-svc-compatible dataset maker script with cache & caption (transcript) support
  • [Fix/Manifest] Accelerate manifest search by concatenating regexes & vectorization with filter
  • [Fix/Media] Remove deprecated GPT image captioning engine
  • [Style] Patch security loopholes according to CodeQL
  • [Style] Replace insecure/dispreferred standard libraries (advised by Julian Berman)
    • os.path -> pathlib.Path
    • hashlib, pycryptodome -> cryptography
    • multiprocessing, futures -> asyncio ("nworker" param is therefore deprecated)
• Frontend
  • [Feature/Search] Add "entries per page" option in search result page (valid range is 12~96, with step size of 12)
  • [Feature/View] Display multiple subsongs for voiceline ACBs
  • [Fix] Add Neo Asari to the cohort (due to an April Fools' event)
  • [Fix] Revise accent color scheme (courtesy of @vertesan and their site)

Development Roadmap

• Backend
  • [Manifest] Export as database
  • [Media] 'File exists' check with conversion
  • [Adv] Story context/alias parser
  • [Docs] Check class docstrings and PyLint warnings
  • [Docs] img/sud file structure in README
• Frontend
  • [Home] 'Manifest Load/Export' buttons
  • [Search] Filter search by assetbundle-/resource- only
  • [View] Embedded Unity model preview (powered by mari-mari-dance)
  • [View] Progress bars for downloading, MD5 checksum, and ffmpeg conversion
  • [Adv] Storyline editor (the codename would be GkmasAdventurer)
  • Deploy on GCP (may involve high-concurrency middleware)

GkmasObjectManager GUI v1.1

A Python interface with Gakuen Idolm@ster's object database ("manifest")

Running the GUI

pip install -r requirements.txt
python server.py

Then access the Flask service at http://127.0.0.1:5001/.

Using the Module on Python CLI

import GkmasObjectManager as gom

m = gom.fetch()  # Fetching manifest from server
m.export("manifest.json")  # Supports exporting as ProtoDB/JSON/CSV

m_old = gom.load("octocacheevai")  # Loading local manifest
m_diff = m - m_old  # Taking the difference
m_diff.export("manifest_diff.json")

m.download(  # Downloading with media conversion
    "img.*cidol.*full.*", "img.*csprt.*full.*",  # (character & support cards)
    image_format="JPEG", image_resize="16:9"
)
m.download("sud.*inst.*.awb", audio_format="WAV")  # (instrumental tracks)
m.download("mov.*cidol.*loop.usm", video_format="MP4")  # (animated character cards)

m.search('mdl.*ttmr.*')  # Searching by regex (all Unity models of Tsukimura Temari)
m['mdl_chr_ttmr-cstm-0076_body'].download(path='', categorize=False)  # Indexing a single object
m['img_general_cidol-kllj-3-007_1-thumb-landscape-large'].get_caption()  # Generating caption with GPT 4o
# (requires setting a valid 'openai_api_key' in credentials.yml)

Comparing Versions

	Tiny Build	Full Build
Branch	offline-demo	main
CRUD Support	CRU	R
# Entries	111 Every featured song	23,033 (as of manifest revision 202) All assetbundles and resources
Manifest	JSON load JSON export Differentiate Search by regex	Online fetch & decrypt ProtoDB, JSON load ProtoDB, JSON, CSV export Differentiate Search by regex Multiprocess download
Object	Resource indexing	Resource download AssetBundle deobfuscate, download
Multimedia	PNG image MP3 audio	USM video Texture2D, PNG image AudioClip, AWB/ACB, MP3 audio GPT image captioning

Changelog

• [Feature] Implement plugin-based conversion for Unity image/audio, AWB/ACB audio, and USM video
  • Set JPG as image conversion target with image_format="JPEG", due to BytesIO's restrictions
  • Only supports extracting a single audio stream from an ACB archive, due to vgmstream's restrictions
  • video_format="MKV" or "Matroska" produces an error, due to compatibility issues with ffmpeg
• [Feature] Implement Flask-based GUI
  • Home page: 12 accent color schemes, search bar with template dropdown, wallpaper "latest releases"
  • Search page: card-based layout, image preview, keyword highlighting, sort by name/ID, pagination
  • View page: embedded multimedia (image/audio/video), real-time captioning, blob download links
• [Docs] Get the "manifest watcher" workflow up and running
  • update_manifest.py fetches the latest manifest and its difference with all previous revisions

Development Roadmap

• [Backend] Video captioning
• [Backend] Extracting multiple audio fragments from ACB
• [Backend] YAML downloading schema (e.g., "namecard kit")
• [Backend] Story script parser
• [Frontend] Deploy on cloud
• [Frontend] 'Manifest Load/Export' buttons
• [Frontend] Filter search by assetbundle/resource
• [Frontend] Embedded Unity model preview

GkmasObjectManager v1.0

[Refactor] Implement canon-repr-based manifest diff

- This assumes no deletion in newer manifests

Object-oriented gkmasToolkit v0.4beta

An object-oriented refactoring of kishidanatsumi/gkmasToolkit, with advanced support for manifest versioning, downloading by regex, and image exporting.

Example Usage

from gkmasToolkit import GkmasManifest
m = GkmasManifest('EncryptedCache/octocacheevai_v89')
m.export('DecryptedCache/v89/')
m.download("img_general_cidol.*thumb-landscape-large.*", path="wallpaper/cidol-landscape", categorize=False, img_resize="16:9", img_format="jpg")
m.download("img_general_cidol.*full.*",                  path="wallpaper/cidol-portrait",  categorize=False, img_resize="9:16", img_format="jpg")
m.download("img_general_csprt.*full.*",                  path="wallpaper/csprt",           categorize=False, img_resize="16:9", img_format="jpg")
m.download("img_general_meishi_base.*full.*",            path="wallpaper/base",            categorize=False, img_resize="16:9", img_format="jpg")

from gkmasToolkit import LATEST, VERSION
m = GkmasManifest(LATEST)
for i in range(m.revision):
    GkmasManifest(VERSION(i)).export(f'samples_v{m.revision:03d}/{i:03d}')  

Changelog

• [Feature] Implement ProtoDB API request (courtesy of DreamGallery/HatsuboshiToolkit)
• [Fix] Copy list in Diclist to avoid destructive rip_field
• [Style] Split large classes (GkmasManifest, GkmasAssetBundle, GkmasResource) into multiple files
• [Style] Use urljoin to handle all URL concatenations
• [Style] Ensure all path arguments in client-side accessible methods accept both str and Path types
• [Style] Use google.protobuf.internal.builder to simplify & organize octodb_pb2.py

Object-oriented gkmasToolkit v0.3

An object-oriented refactoring of kishidanatsumi/gkmasToolkit

Example Usage

from gkmasToolkit import GkmasManifest
m = GkmasManifest('EncryptedCache/octocacheevai_v81')
m.export('DecryptedCache/v81/')
m.download("img_general_icon_contest-rank.*",              path="namecard/profile",             categorize=False)
m.download("img_general_meishi_illust_idol.*",             path="namecard/idol/full",           categorize=False)
m.download("img_general_meishi_illust_sd.*",               path="namecard/idol/mini",           categorize=False, img_resize="2:3")
m.download("img_general_cidol.*full.*",                    path="namecard/idol/produce",        categorize=False, img_resize="9:16", img_format="jpg")
m.download("img_general_cidol.*thumb-landscape-large.*",   path="namecard/idol/produce",        categorize=False, img_resize="16:9", img_format="jpg")
m.download("img_general_cidol.*thumb-portrait.*",          path="namecard/idol/produce",        categorize=False, img_resize="3:4")
m.download("img_general_meishi_illust_sign.*",             path="namecard/idol/sign",           categorize=False)
m.download("img_general_csprt.*full.*",                    path="namecard/support",             categorize=False, img_resize="16:9", img_format="jpg")
m.download("img_general_meishi_base_story-bg.*",           path="namecard/base/commu",          categorize=False, img_resize="16:9", img_format="jpg")
m.download("img_general_meishi_base_(?!story-bg).*full.*", path="namecard/base",                categorize=False, img_resize="16:9", img_format="jpg")
m.download("img_general_achievement_char.*",               path="namecard/achievement/idol",    categorize=False)
m.download("img_general_achievement_produce.*",            path="namecard/achievement/produce", categorize=False)
m.download("img_general_achievement_common.*",             path="namecard/achievement/misc",    categorize=False)
m.download("img_general_meishi_illust_music-logo.*",       path="namecard/parts/logo",          categorize=False, img_resize="3:2")
m.download("img_general_meishi_illust_pict-icon.*",        path="namecard/parts/icon",          categorize=False)
m.download("img_general_meishi_illust_stamp.*",            path="namecard/parts/misc",          categorize=False)
m.download("img_general_meishi_illust_event.*",            path="namecard/parts/event",         categorize=False)
m.download("img_general_meishi_illust_highscore.*full.*",  path="namecard/parts/highscore",     categorize=False)
m.download("img_general_skillcard.*",                      path="namecard/produce/skillcard",   categorize=False)
m.download("img_general_pitem.*",                          path="namecard/produce/pitem",       categorize=False)
m.download("img_general_pdrink.*",                         path="namecard/produce/pdrink",      categorize=False)

from gkmasToolkit import ALL_ASSETBUNDLES, ALL_RESOURCES
m.download(ALL_ASSETBUNDLES, ALL_RESOURCES)

Changelog

• [Feature] Add assetbundle extraction options in blob.GkmasAssetBundle.download()
  • categorize=False prevents categorizing blobs into subfolders
  • extract_img=True attempts to extract a single image from img_.*\.unity3d assetbundles
  • img_format controls the format of output images
    • WARNING: Texture2D's are forcefully converted to RGB mode if alpha channel is not supported in the given format
  • img_resize adjusts the images' aspect ratio when necessary
    • New size is determined in maximize mode
    • ensure_fit and preserve_npixel modes are also available, but the kwargs are not propagated to top level
    • Refer to blob.GkmasAssetBundle._determine_new_size() for details
• [Feature] Write script for downloading the "customizable namecard kit"
  • Post-categorize selected folders by lambda'ed filenames (that highlights character identifiers)
• [Fix] Implement manifest diff with utils.Diclist.diff() with ignored fields
• [Fix] Revise logic in determine_subdir (moved into blob.GkmasResource)
  • Everything after the first number or the first "character identifier" is ignored
  • Subdirectories are split by '_' (allows '-' in subtypes)
• [Style] Relocate isolated utility functions
  • manifest._list_diff(), manifest._rip_field() -> utils.Diclist
  • manifest._make_diff_manifest() -> manifest.__sub__()
  • blob._ab2png_and_write_bytes(), blob._ab2png() -> blob._export_img()
  • utils.determine_subdir() -> blob._determine_subdir()
  • utils.resize_by_ratio() -> blob._determine_new_size()
• [Docs] Provide extensive docstrings for all classes and methods

Object-oriented gkmasToolkit v0.2

An object-oriented refactoring of kishidanatsumi/gkmasToolkit

Example Usage

from gkmasToolkit import GkmasManifest
m81 = GkmasManifest('EncryptedCache/octocacheevai_v81')
m81.export('DecryptedCache/v81/')
m81.download('adv.*ttmr.*', 'sud_vo.*fktn.*', 'mdl.*hski.*', nworker=8)

from gkmasToolkit import ALL_ASSETBUNDLES, ALL_RESOURCES
m81.download(ALL_ASSETBUNDLES, ALL_RESOURCES)

m48 = GkmasManifest("EncryptedCache/octocacheevai_v48")
mdiff = m81 - m48
mdiff.export('DecryptedCache/diff/')

Changelog

• [Feature] Implement multithread downloading
  • Number of workers defaulted to multiprocessing.cpu_count()
• [Feature] Implement magic methods for GkmasManifest
  • NEW! Now two manifests can be differentiated, exported, and downloaded (new manifest has no raw)
  • dependencies and uploadVersionId fields are ignored during comparison, but all fields are retained in output
  • Side effect: __init__() now accepts Tuple[diff-dict: dict, rev1: int, rev2: int], meant for internal use only
• [Fix] Merge exported CSV's for assetbundles and resources
  • Since GkmasManifest.jdict['assetBundleList'][i]['crc'] is unused as of now
• [Style] Accept a variable number of arguments in GkmasManifest.download()
  • This implies that GkmasManifest.download_all() is redundant and now removed
• [Style] Import manifest and macros at module init (saves client some effort)
• [Style] Move constants to const.py

Object-oriented gkmasToolkit v0.1

An object-oriented refactoring of kishidanatsumi/gkmasToolkit

Example Usage

from gkmasToolkit.manifest import GkmasManifest
manifest = GkmasManifest('EncryptedCache/octocacheevai')
manifest.export('DecryptedCache/')
manifest.download_all(['adv_.*_ttmr_.*', 'sud_vo_.*_fktn_.*'])

from gkmasToolkit.utils import ASSETBUNDLES, RESOURCES
manifest.download_all([ASSETBUNDLES, RESOURCES])

Changelog

• [Refactor] gkmasToolkit.manifest.GkmasManifest is the top-level entity that handles all tasks, including
  • protobuf decryption,
  • export to JSON or CSV [NEW],
  • assetbundle/resource downloading, and
  • assetbundle deobfuscation.
• [Feature] Add regex support for downloading selected files
• [Fix] Objects are now directly downloaded as intended filenames, instead of md5 identifier
• [Chore] Remove deprecated eDiffMode, eWorkingMode, AssetTypeDict, ImageConvertor, and StorageDecryptor, along with support for SQLite DB export
• [Style] Wrap redundant pretty-print logic in rich.Console into gkmasToolkit.utils.Logger
• [Format] Use Black Formatter to clean up the codebase