MIF format support

[Carmina16]

I have some data on the MIF/RMD formats; are you interested?

[afritz1]

Sure! I did some preliminary investigation a few months ago to get an idea of the layout. They are still not completely understood, but I believe they are used for defining interior locations (especially main quest areas), as well as for generating chunks of cities. They're kind of like "prefabs" in a way, I guess.

All that the .MIF parser can retrieve right now is just the map dimensions (width and depth). Each file appears to contain a map header and an array of levels. I think that "flor" represents the voxels in the ground floor and "map1" represents the voxels in the main floor. There's a lot of other miscellaneous data like trigger locations, probably for displaying messages when the player walks into a voxel, and coordinates for defining which doors are locked. Are there entity definitions as well, like positions of creatures and torches? I'm pretty sure all of the creatures in main quest dungeons have pre-defined spawn points.

I assume that each voxel is like 4 bits, and each floor uses some kind of compression?

I haven't looked into .RMD files very much, but I think they define "chunks" of wilderness. I was planning on programming the wilderness only after all the city and dungeon generation works.

[Carmina16]

Yes, it's the same compression as in type 8 images.

There is a WORD for each voxel on the level, stored right-to-left from the top right corner. They refer the objects and textures defined in the corresponding INF file.
NUMF stores the number of the floor textures.
MAP1 contains the walls of dungeons and the 1st story of buildings.
If the 0x8000 bit is not set:
0000 means empty block
XXYY, if XX&0x7F == YY&0x7F defines a solid block with the XX-1 texture on the walls.
XXYZ otherwise defines a raised platform: X apparently is the height, Y is the texture on the top, defined by *BOXCAP Y in the INF file, Z is the texture on the side, from *BOXSIDE Z.
If the 0x8000 bit is set:
If the most significant nibble is 8, the lower byte is the index of the FLAT that defines a loot pile, a monster, a key, or a decoration.
9 is a transparent block which shows a 1-sided texture on all its sides.
A is a transparent block which shows a 2-sided texture on one of its sides.
B is a door with the texture defined by the 6 lowest bits-1
C is unknown
D is a diagonal wall: D0 /, D1 , lowest bits are the texture index+1

FLOR values contain the floor texture index in the high byte, or C for the dry chasm, D for the water chasm and E for the lava chasm. The low byte contains the index of FLAT at this location + 1, or 0 if empty (used for placing objects on platforms).

MAP2 contains the second story tiles. Two additional flags are used: 0x80 used to expand the block to the additional story, 0x8000 to expand it to 2 additional stories. So 0x8080 will display as 4 stories for 5 stories in total.

TRIG section consists of 4-byte records: X coordinate, Y coordinate, *TEXT index, and sound index.

LOCK section consists of 3-byte records: X, Y and lock level. The key name is calculated as (locklevel-1) mod 12.

In RMD files, the first word is the uncompressed length. The file itself is RLE-compressed using WORDs instead of bytes. If NNNN is positive, N literal words follow, if negative, the next word is repeated -N times.
The data represent 3 arrays: FLOR, MAP1 and MAP2 data for a 64x64 block.

[Carmina16]

All the cities use the predefined INF file, ABC.inf, where A is the tileset (M, T, or D), B is City or Wilderness, and C is the weather type: Normal, Rain, Snow, or W.

[afritz1]

This is great information! Thanks for the details. I'll look into .MIF files again soon and see how much more of the decoder I can implement. It's fun demystifying these arcane formats.

My first guess on the wilderness' layout was that it's split up into 64x64 chunks, and that seems to be verified based on your explanation. I guessed those numbers in particular because, if you continually press F2 while walking through the wilderness, you can see the player's coordinates looping between 32 and 96 (presumably a quirk of the coordinate system used).

Any idea what TARG means in the .MIF files? I assume it's for a target of some kind. Also, any details on the other numbers in MHDR besides the dimensions of the map?

[Carmina16]

TARG records look like X,Y coordinates to place random loot and quest monsters.

I couldn't find what most numbers in the header are for:

struct MHDR { BYTE unknown1; BYTE nEntries; // valid entries that follow WORD X[4]; // no idea what coordinate is this WORD Y[4]; BYTE iStartingLevelIndex; WORD nLevels; WORD wWidth; WORD wHeight; BYTE unknown[]; }

[afritz1]

We've made quite a lot of progress on .INF and .MIF files since this issue was opened, and I can't thank you enough for your assistance so far. I have a few more questions still:

1. How are *BOXSIDE textures for floors handled? Currently, the floor sides are just using the texture index from the floor top. I don't know when to use the dry/wet/lava chasm or pit textures.
2. How are city blocks generated? It looks like some permutation of the BSBD___.MIF, TVBD___.MIF, etc. files, probably using a similar formula to the main quest .MIF filenames (bit shifting, rotation, ...). I believe that villages are 4x4, towns are 5x5, and city-states are 6x6.
3. How are .INF sounds indexed? Currently, some places like Murkwood cause an out-of-bounds access for certain sounds (like index 32). Are some sound indices supposed to have a modification applied?

On another note, I've been wondering about the depth of your knowledge regarding Arena. You have quite a few details in certain places, especially with compressed formats like .MIF files, which makes me wonder how you came across it to begin with. Maybe you figured it out on your own years ago, or maybe you have a connection to the original developers? 😄

[Carmina16]

• Floors do not have walls, only the chasms have small textures on the walls, taken from *XXXCHASM. They are transparent on the bottom, so they allow to see the water/lava/unrendered image in the bottom part of the screen.
• That one is quite complicated. Maybe you can give me the wiki access to document it?
• The numbers in the INF file after sound filenames correspond to one of the 64 slots for sounds. The unassigned slots are ignored by the sound routine. There are some shenanigans around the slot 32, but I haven't figured that one out yet.

[afritz1]

I sent you collaborator access.

So the floors themselves only have their top face textured? If that's the case, then chasms would appear to be context-sensitive (correct me if I'm wrong). In other words, they need to look at adjacent voxels to determine which faces are textured. I wasn't sure if this is what happens because this pattern isn't found anywhere else in Arena (I think. I've been looking into how doors are rendered, and they seem to depend on surrounding voxels for determining which door faces to display). So if chasms are context-sensitive, then there would need to be a texture index for each of the wall faces on the chasm, and a second pass over the floor data would need to be done for determining each adjacency.

With regards to .INF sounds, I'll just add a warning when an out-of-range access is attempted, and it'll resort to some default sound for now.

Are you familiar with C++11? You're free to make changes and do a pull request if there's something you know how to do in the code. I'm assuming you haven't tried to compile anything in the project yet.

[Carmina16]

Well, you can see whether the floor tile is adjacent to a chasm, when constructing the floor, and set its walls accordingly?

Unfortunately, I don't know C++.

[afritz1]

Well, you can see whether the floor tile is adjacent to a chasm, when constructing the floor, and set its walls accordingly?

Right. There are probably a few ways to do this. I just wanted to brainstorm a bit before I start implementing something, since this case seems a little peculiar and I wanted to look before I leap. I'm just framing in my mind how it'll be done based on the fact that non-chasm floors don't have boxsides. According to what you said above, each chasm should be implemented by having an optional wall on each of the four sides. This data will likely also be used with collision detection at some point.

Once we get to where we're generating wilderness, the perimeter of each chunk will need to be checked against adjacent chunks for updating the walls of chasms.

[afritz1]

I just added renderer support for type 0xA voxels (store signs, bed curtains, etc.) in commit f3202ca, but they don't have the correct texture IDs. This is how I'm currently obtaining data from their .MIF voxel:

// (map1Voxel & 0xF000) == 0xA000.
int textureIndex = map1Voxel & 0x000F; // Wrong.
int orientation = (map1Voxel & 0x00F0) >> 4; // 0: North, 4: West, 8: South, C: East.

The bed curtains in STKEEP.INF are at index 11 in the file, but with the method above I get 12 from the voxel data. Any idea how Arena calculates the texture index there? Maybe with a slightly different mask?

Edit: I experimented with this:

int textureIndex = max((map1Voxel & 0x003F) - 1, 0);
int orientation = (map1Voxel & 0x00C0) >> 4;

which is how door textures are calculated (lowest 6 bits), and edge textures are 99% correct then. But there's one type 0xA voxel in IMPERIAL.MIF with a texture index of 0 that forces the usage of max() because it otherwise goes negative.

[Carmina16]

Well, the game renders a gray square at that place, so it's better just to ignore that invalid 0 index.

[afritz1]

Yeah, that's a better idea. I'll make that voxel assignment depend on a condition instead.

[Allofich]

I saw the "not sure what this is" comment in the source about the hyphen before some FLAT declarations in .INF files, so I experimented with removing and adding them in EQUIP.INF.

When I removed all the hyphens, there wasn't any visual difference (that I noticed) in an equipment store, but DOSBox's logging showed the files that normally have hyphens before them being opened by A.EXE, whereas when running with an unmodified EQUIP.INF they weren't opened.

I also tried hyphening declarations of files that were used for visible sprites in the equipment store. It caused the files to not be loaded. In one case the relevant sprite was replaced in-game by another (contextually nonsensical) one, and in the other the sprite was invisible and the program crashed.

Tentative conclusion is that the hyphens just comment out these lines and cause the files to not be loaded.

[afritz1]

Added support for wilderness chunks (.RMD files) in commit ec4549c. It's able to load four chunks into a fixed 128x128 grid for testing. The WILD###.RMD files are picked using some simple random integers for now.

[afritz1]

@Carmina16, I'd like to start chipping away at city generation soon. I see your notes on the wiki, and they are great! Could you tell me some more about where the data comes from for values like:

• cityId in templateCount <- cityId in coastalCities ? (isCity(cityId) ? 3 : 2) : 5
• cityX and cityY in citySeed <- (cityX << 16) + cityY

Also, how is isCity() implemented?

Are cityX and cityY the location values stored in CITYDATA.00?

Also, I'm not sure, but is WATER1 at template ID 5 in the reserved block list a typo? Should it be CITYW1, TOWNW1, or VILLAGW1?

[Carmina16]

Each province has 32 settlements, first 8 are "cities" (internally 0), next 8 are "towns" (1), and the last 16 are "villages" (2). isCity is for example ((cityId & 0x1F) < 8).

Yes, cityX/Y are coordinates from CITYDATA.0x.

"Water1" just means that it is the first template for coastal settlements: either CITYW1, TOWNW1 or VILLAGW1.

[afritz1]

Oh, okay. I see now. I thought maybe cityId was a global value, like 0-200 or something, but it's actually just an index into the province locations array. And isCity() is easier than I thought, too. I'll look into city generation soon.

[Carmina16]

cityId is (province << 5) + localId, so every settlement in Arena has its id in 0..256 range, with 256 being the Imperial City.

[Carmina16]

For the testing purposes, I propose North Hall, a town in Hammerfell. Here's its properties:
cityId: 0x2E
Coordinates: 250, 131
Global coordinates: 120, 85
Terrain: 3 (desert)
City Seed: 0x00780055
Ruler seed: 0x00550078
Map: 8, 4, 2, 6, 2, 8, ....
Blocks: bsbd10c, nbbd3b, eqbd6b, tvbd6c, eqbd1b, bsbd8b, ....

[afritz1]

Made some progress. I was able to generate city blocks, but some of them were incorrect and/or had incorrect voxel data (probably my fault). Need to look into it some more.

I used your test properties and got:

• cityId: 0x2E
• Name: North Hall
• Coordinates: 250, 131
• Global coordinates: (where do these come from?)
• Terrain: (where does this come from? Province quadrant?)
• Seed: 0x00FA0083 (incorrect)
• Ruler seed: nothing yet
• Map: 8, 4, 2, 6, 2, 8, ...
• Blocks: bsbd14c, nbbd3d, eqbd7d, tvbd10d, eqbd3a, bsbd9a (all incorrect)

I'm not sure why the seed is wrong because I did (cityX << 16) + cityY, with cityX=250 and cityY=131.

Also I'm not sure why my cityId is 0x2E and not 0x2D.

[Carmina16]

My bad! The seed derived from the global coordinates is used elsewhere. The value used for the generation is indeed 0xFA0083, and the cityId is 0x2E.

As global coordinates, rulers and terrain are not of immediate importance, I will document them in wiki.

More control values:
After generating the plan, the seed value is 0xe91d2657
After generating all the blocks, it is 0x94a4697f
Complete plan (mirrored):

8 4 2 6 2
8 2 5 8 2
8 5 8 5 5
1 6 3 8 6
7 1 8 6 8

[afritz1]

Alright, city generation seems to be more or less working in commit 90574fd. There are still a couple problems with some starting positions of blocks, and floor voxels being air instead of a wet chasm, but for the most part, Arena's cities can now be loaded! Thanks for helping me through it, @Carmina16.

I was getting the wrong block .MIF names before because I was accidentally getting the variation's random value before the rotation's random value, but all of that seems to be working now.

So about the starting positions of blocks, could you check that the values are correct here? I think I'm using the wrong index for each location type.

And the missing wet chasms, I'm not sure how Arena clears a block before it writes to it (because some blocks get written into more than once), so I'm just zeroing the floor voxels for now as part of the clearing.

[Carmina16]

I found out the array in the executable has different ordering: towns, villages, and finally cities. So the first element is 3, 4 TOWN1 , sixth is 3, 6 TOWNW1, etc. ending in 3, 5 CITYW3. Same ordering applies to the filenames too.

I'm not sure what the second issue is; the level data are just overwritten by the block copied.

[afritz1]

Oh, the problem with starting positions was from an off-by-one bug in the executable reading, so TOWN1 was 0, 3 instead of 3, 4, etc..

And the second issue, I think I just need to change the block writing a little bit so it checks adjacent blocks when determining chasm walls (maybe?).

[afritz1]

For blocks in the reserved block list that go outside the plan, are those just ignored? I.e.:

for block in reservedBlocks
   plan[block] <- RESERVED

There are some blocks with too high a value that then cause out-of-bounds writes; i.e., in villages with 4x4 blocks (writing to index 21 when there's only 16 blocks).

[Carmina16]

Yes, just ignore those if you use a dynamic array to hold the city plan.

[afritz1]

I'm looking into dungeon generation now. One small question: what does getRandomSeed() mean in this? How is it defined?

newSeed <- getRandomSeed()
transitions <- []
for i <- 0 to depth - 1
...

[Carmina16]

Oh, it's just the current seed value. I've also noticed the small error in the algorithm, so watch for the change.

[afritz1]

Does Arena use one integer for all location IDs (that is, both cities and dungeons)? I'm not sure if I can have a locationID that's 0-47 instead of a localCityID that's 0-31 and a dungeonID that's 0-15. I'm wondering about this because I'm about to implement the travel time code and localToGlobal() takes a local X and Y that comes from a 0-31 local city, but it looks like it could also take a 0-47 value (for all possibilities in a province, not just cities).

[Carmina16]

There are two separate location ids used: the first for cities, the other for dungeons (0..15 + (province<<5)).
The travel code can handle the 0..47 value for the travel start/destination.

[afritz1]

Do you know where the wilderness block lists you mentioned in the "Wilderness" wiki are? Also you said there are four lists but I think you meant five (normal, village, dungeon, inn, and temple). I'm assuming they're just lists of integers between 5 and 70 for generating WILD0##.RMD names.

[Carmina16]

Added!

[Carmina16]

More info on INF flat flags:

• 2 reflective: a vertically mirrored 2:1 copy of the image above is painted over it: the color index 30 is replaced with pixels from the even rows, 103 from the odd (for simulating ripple effect).
• 64 200% scale

[afritz1]

Thanks. I'd like to have sprites implemented before the next release, so I'll get to those things before then. I don't know if I will implement reflections the same way though (because my renderer allows some fake free-look); I'll have to investigate it at some point.

[Carmina16]

I think I got the precise expression for A-block vertical offset: it should be h*8 in interiors, and h*32 - 8 in exteriors.

[afritz1]

Works like a charm, thanks. See commit 8170e39. Store signs, laundry, bed curtains, etc. are all at the correct height.

[afritz1]

@Carmina16, how's it going? I've been looking into distant sky rendering lately (mountains, clouds, etc.) and I'm having trouble getting the placeSkyboxStatics() algorithm to give the desired results. I think the RNG seed might be wrong because even nMount seems wrong on the first rnd() call.

https://github.com/afritz1/OpenTESArena/wiki/Skybox#mountains-and-clouds

[Carmina16]

Make sure you use the correct seed of two different ones: for Rihad, the skybox seed is 0x6c0064. That gives two mounts, desert3 at angle 3, and desert1 at angle 447. The first cloud on the 0x1d day is cloud12 at angle 202 and height 19.

[afritz1]

I tried your suggestion for Rihad and the random generation and angles are correct now. Apparently the distant sky seed for cities is ror(rulerSeed, 16).

I've tried guessing and checking the seed for towns and villages but I haven't gotten it yet.

[Carmina16]

It is made by combining the global X and Y city coordinates.

[afritz1]

I'm not sure what you mean by combining them. I've tried

seed = (globalX << 16) + globalY;
seed = (globalX << 16) | globalY;
seed = ror((globalX << 16) + globalY, 16);
...

and they all produce incorrect results for Tenmar Forest in Elsweyr.

Also, does the Imperial City have a special distant sky seed?

[Carmina16]

• I believe the correct formula is ((globalX<<16) + globalY) * province. It turns out all High Rock cities have the same landscape.
• No, IC doesn't have a special seed.

[afritz1]

I think you are correct. I used that formula and traveled to various cities/towns/villages and they all appear to be correct, and yes, all places in High Rock only have one mountain (ironic -- maybe that's where "High Rock" came from. "Hey guys, what should we call this province?" "Well there's a big mountain in the distance" "Let's call it High Rock").

[afritz1]

Hey @Carmina16. I recently made some good progress on wilderness generation. Every wilderness chunk now appears as it does in the original game, but two things I'm not sure about are .MIF filename variant numbers and the display name for interiors.

The .MIF variant number calculation for cities is (y << 8) + (x << 1), but I'm not sure about wilderness interiors.

Interior display names appear to be based on the wild X and Y chunk coordinates. For example, if you right click on two different tavern entrances near each other, they will have the same name.

Oh, one other thing. I have city gates kind of working now -- the player can go between the wilderness and the city -- but I don't remember seeing any start point data. How does the original game know where to put the player on the outside of the city gate?

Thanks for your time.

[Carmina16]

The interior layouts should be as usual, if you map their door coordinates into a 128x128 block roughly centered around the player.

Name seed for taverns and temples:
(WY<<16)+WX

Seeds for dungeons:
https://github.com/afritz1/OpenTESArena/wiki/Dungeon-Generation#wilderness-dungeons

WX and WY are the coordinates of the 64x64 tile the player is standing.

When transitioning into the wilderness, the player is moved 300 units in a direction leading through the gate block (for example, if the gates are on the right from player, move them 300 units to the right).
When transitioning into the city, the closest entry point is used.

[afritz1]

Thank you!

The interior layouts should be as usual, if you map their door coordinates into a 128x128 block roughly centered around the player.

I didn't understand this at first, but I think now I do. The "roughly centered" part is the 64x64 area in the middle of the four wilderness chunks, 32 blocks from each edge.

When transitioning into the wilderness, the player is moved 300 units in a direction leading through the gate block (for example, if the gates are on the right from player, move them 300 units to the right). When transitioning into the city, the closest entry point is used.

I thought the game might do something like this. The simplicity of the original also seems more obvious now -- I keep thinking the game does some big coordinate system change, but it seems more like the game just moves the player a little bit and changes all the blocks around them. By closest entry point, do you mean the game does a search for a city gate block? I thought the game just put the player at the city's default start point, independent of whichever gate they used.

[Carmina16]

Indeed it does. I was wrong.