Question about the INDYWV audio format

[justedni]

Hello!
First, thank you so much for all your hard work with this incredible collection of tools.
I was investigating the INDYWV file format on another game (Star Wars Episode 1 The Phantom Menace), and found your github. Your work on reverse-engineering the header and decompression algorithms is really impressive! I saw in indywv.h that there was a "TODO: Implement ADPCM decompression method". Sadly I believe that Phantom Menace doesn't use WVSM compression, the magic numbers don't match, unknown1 is always a positive number. Also, your algorithm only covers stereo files, but in my case, I'm trying to decompress dialog files so they are likely to be mono.
I was just wondering if you could give me some pointers on how you managed to figure out how the decompression worked for Infernal Machine and maybe I can try and implement the rest of the algorithm.
Thanks in advance for your time!

[smlu]

Hey!
My starting point was here: https://web.archive.org/web/20160404215239/http://wiki.xentax.com/index.php?title=Indiana_Jones_And_The_Infernal_Machine_CND. After I learnt IndyWV decompression algorithm I jumped into RE and try to decompiled the rest of the code there. Unfortunately I wasn't able to decompile the other decompression algorithm (no game sound is compressed in that format) which I believe it's some variation of ADPCM compression algorithm.
Fyi this is the function signature of the other decompression method: void __cdecl AudioLib_Uncompress(tAudioCompressorState *pCompressorState, BYTE *pOutData, const BYTE *pInData, int size, unsigned int numChannels, int bStateInited). AudioLib module is part of LEC's code. The tAudioCompressorState type is VWCompressorState struct in my code.

I hope this info helps you in any way!

[justedni]

Btw I've recently tested ChatGPT and Claude 3 Sonnet to decompile spinets of assembly code to C pseudo code. The produce quite good struct-ed C code, though you have be curious and double check the outputted code because they can be wrong in certain cases (integer/float comparison for instance). But overall it can help you speed up reverse engineering, especially in cases when decompiler software fails.

This is really interesting! I really need to try that. I'm currently looking at another RE project on the side, that involves assembly code from an old MS-DOS game (Cyberia, 1994, Xatrix Entertainment), and I have to say, the 16-bit MS-Dos low-level functions are not well integrated into Ghidra, and IDA stopped handling them a long time ago, so it's making my RE efforts very slow. I've managed to isolate the one procedure that I want to extract (there are a few audio assets in the game are compressed with a proprietary file format called Z94). I'll try putting that into Claude 3 Sonnet! :)

[smlu]

I don't think I really every tried to disassemble MS-DOS app before 🤔 I can only guess the support for this platform is veining nowadays. Tho now that I think, there are plenty of dos games I can think which would be fun to peek at it's internals.

Btw another AI tool I can suggest is mistral (https://chat.mistral.ai). If you gonna try it out it make sure you select codestral model which is better tuned/trained for coding.

[smlu]

Could you maybe paste the code line in your source code where this multiplication occurs?

Sure! In my code it appears in indywv.cpp on line 399. If you want to find the source, in my copy of Indy3D.exe, the instructions are around the data offset 004e0108. It looks like that in IDA: The local variable v42 is a __int8, but the data_0054bcf0 is an array of integers, which is think is the reason why v42 should be multiplied by 4 before being used as an index? I don't know why IDA didn't pick it up. But Ghidra does: And by the way, this is one cool example of RE code that I had to heavily modify in order to make it work in C, after carefully analyzing the static data. The calculated index looks inside an array that contains a list of addresses, that needs to be dereferenced in order to get the actual data! That calculated index can only be 8, 12, 16, 20, 24 or 28 (which is why I've created the "getIndex" function). This was fun to investigate! :) And one last thing, the problem with the *4 in IDA appears both in SWE1TPM's decompressADPCM, and in Infernal Machine's compressADPCM (I'm pretty sure it's also in IM's decompressADPCM). But honestly don't spend too much time on this, this could well be a noob mistake on my end! What matters is that it works now :)

Looking at my de-compiled code I have this:

v36 = v22;
*(_WORD *)v29 = v22;
v29 += sampleSize;
v24 = AudioLib_aIndexTableTable[stepBits][v18] + a2a;
if ( v24 > 0 ) {
  if ( v24 >= 88 )
      v24 = 88;
}
else {
  v24 = 0;
}

The AudioLib_aIndexTableTable should be data_0054bcf0 in your case, I think.
And I think you're right that the array contains the addresses to other arrays.

This is the definition of AudioLib_aIndexTableTable :

const int8_t *AudioLib_aIndexTableTable[8] =
{
  NULL,
  NULL,
  &AudioLib_aIndex2Bit,
  &AudioLib_aIndex3Bit,
  &AudioLib_aIndex4Bit,
  &AudioLib_aIndex5Bit,
  &AudioLib_aIndex6Bit,
  &AudioLib_aIndex7Bit
};

The AudioLib_aIndex* are arrays of different dimensions.

In my case I think the decompiler got it right there by accessing the AudioLib_aIndexTableTable like it's 2D array.
Btw the I just remembered that the symbol names (e.g.: AudioLib_aIndexTableTable) I got from linux version of Grim Fandango Remastered. Fyi it contains all debugging symbols 😃

Also recently I discovered debug version of Indy3D.exe on archive.org
Here's the link: https://archive.org/download/2-indiana-jones/2%20Indiana%20Jones.rar/La%20Macchina%20Infernale%2FResource%2FCD1.GOB
You have to extract content of gob, and the larger one (~2MB) of the 2 Indy3D.exe is debug version.
What's great about this version is that due to being combined in debug mode the code is not optimized and decompiled code could better resemble the original code in some cases.

[justedni]

Sorry for the late reply! Thanks a lot for looking into it! It makes perfect sense to me now, I think the problem came from my re-implementation of AudioLib_aIndexTableTable (indywv_data.cpp:47) I was using both Ghidra and IDA to re-implement this and I probably got confused at some point. The "raw" version of AudioLib_aIndexTableTable contained 4-bytes addresses, which is why the index parameter to access the array (stepBits in your case) needed to be multiplied by 4 in order to point to the wanted address in the raw array. But when it's represented as a 2D array like in your case, then the code shouldn't multiply by 4. Thanks for your help!

And I had no idea that there were versions of those games with debug symbols and everything! That's an amazing find!! I need to check it out!! :)

By the way, thanks again for suggesting Claude AI, I've been using it on a regular basis since you mentioned it, it's great for quick prototyping (C++ or python in my case), and it also looks very promising for RE. I sent it raw MS-Dos assembly files and it was able to analyse and decompile portions of the code. The MS-Dos RE project I'm working on atm is a bit complex but it definitely speeds things up, having an AI giving quick insights on some obscure functions that I'd have to google one-by-one otherwise. I'll check out Mistral as well one of those days!

[smlu]

And I had no idea that there were versions of those games with debug symbols and everything! That's an amazing find!! I need to check it out!! :)

Unfortunately in the case of Indy3D.exe the pdb file is missing but the function structure resembles the original code (e.g.: preserved functions which should be inlined). But the remastered linux version of Grim Fandango is I guess is gold digger in terms of debug symbols :)