This repo contains Scala CLI scripts for generating songdb TSV files used by audacious-uade (https://github.com/mvtiaine/audacious-uade).

* Directories:

- songdb/ - Scala CLI, SQL scripts and raw source TSVs to generate the final processed TSV files
- tsv/ - the songdb TSV files used by audacious-uade. The files are "encoded" to almost binary format to optimize for size and fast in-memory songdb initialization
- tsv/pretty/ - pretty printed / clear text versions of the TSV files. See "spec" below.
- misc/ - misc bash scripts

* Songdb TSV files:

- tsv/md5idx.tsv - index of MD5 hashes (48 bits) (not used for pretty printed TSVs)
- tsv/modinfos.tsv - module file format and channel info
- tsv/songlengths.tsv - subsong and songlengths info
- tsv/amp.tsv - author metadata sourced from AMP
- tsv/modland.tsv - author/album metadata sourced from Modland
- tsv/unexotica.tsv - author/publisher/album/year metadata sourced from UnExotica
- tsv/demozoo.tsv - author/publisher/album/year metadata sourced from Demozoo
- tsv/combined.tsv - AMP/Modland/UnExotica/Demozoo metadata merged to single TSV

* Raw TSV source files:

- songdb/sources/aminet.tsv - module infos and songlengths for Aminet
- songdb/sources/amp.tsv - module infos and songlengths for AMP
- songdb/sources/demozoo.tsv - Demozoo metadata generated with SQL query (songdb/scripts/sql/demozoo.sql) from Demozoo postgres database dump
- songdb/sources/demozoo_leftovers.tsv - module infos and songlengths for Demozoo downloads (excluding AMP/Modland downloads). Link list is generated with songdb/scripts/sql/demozoo_leftovers.sql
- songdb/sources/modland.tsv - module infos and songlengths for Modland
- songdb/sources/modland_incoming.tsv - modules infos and songlengths for Modland incoming directory
- songdb/sources/modsanthology.tsv - module infos and songlengths for Mods Anthology
- songdb/sources/unexotica.tsv - module infos and songlengths for UnExotica
- songdb/sources/wantedteam.tsv - module infos and songlengths for Wanted Team
- songdb/sources/zakalwe.tsv - module infos and songlengths for Zakalwe chip git repo

Except for demozoo.tsv, the raw TSV source files are generated using the precalc binary+script https://github.com/mvtiaine/audacious-uade/blob/master/src/plugin/cli/precalc/ from my local copy/mirror/snapshot of the various sites/sources.

Note that some additional required files not included in Github, specifically local mirror of some of the AMP and UnExotica web pages are needed to actually run the Scala songdb.sc script. The script runs completely locally and does not download anything from internet.

Here are example snippets and short spec for the pretty printed TSVs. Example parsing code can be found in songdb/scripts/pretty.sc

songlengths.tsv
```
fffd7a7d8547	1	250840,p+s
fffd9ea13bc6	1	162860,p
fffdc1d765c3	0	40880,l 117860,l 8780,s 79340,l 8080,s 19000,s
```
[md5]<TAB>[minsubsong]<TAB>[[songlength(ms),songend]<SPACE>[songlength(ms),songend]<SPACE>[...]]

modinfos.tsv
```
fffdc1d765c3	CustomPlay	
fffdd3c2bef3	Scream Tracker 3.2x (GUS)	8
fffe869a7f8d	AHX v2	
```
[md5]<TAB>[format]<TAB>[channels]

combined.tsv (and amp/modland/unexotica/demozoo.tsv)
```
feaa9d2a4869	Scorpik	Alchemy	Toxic Ziemniak	1992
feaba2f4c992	Jazz			
feabaabf8a62	Mantronix~Tip	Blue House Productions~Rebels~Sonic Projects	Blue House 2	1991
```
[md5]<TAB>[authors]<TAB>[publishers]<TAB>[album]<TAB>[year]
- multiple authors or publishers are separated by ~

The TSV files use UTF-8 encoding.

I reserve the right to change the format or location in Github of any of the TSV or other files at any time.

* License

The Scala and SQL scripts are licensed unded GPL-2.0-or-later.
For any applicable sui generis rights or copyrights I may have over the database files, they are provided under CC BY-NC-SA 4.0 license.

Sources used for the database:
- AMP - https://amp.dascene.net/
- Modland - http://ftp.modland.com/
- UnExotica - https://www.exotica.org.uk/wiki/UnExoticA
- Demozoo - https://demozoo.org/
- Mods Anthology - https://archive.org/details/cdrom-amiga-mods-anthology-1
- Wanted Team - http://wt.exotica.org.uk/
- Aminet - https://aminet.net/
- Zakalwe - git://zakalwe.fi/chip