Okay taking more notes on the "missing link" between how map grids are numbered, how specifically that defines the stage geometry, and how these pieces fit together. I think I at least have some predictions of how the game organizes the stage geometry, so I can put forth an hypothesis, test it and then see how right or wrong my predictions were.
So the right now, the image above is how I think the map tiles are defined. There are effectively three types. The first are "set pieces", which are larger tiles that general show a house, or a store front or something. If the tile height is zero, the game will condense these down into a unique list of tiles, so even though there are four of the same tile, the game will only draw the tile once. The numbering for these tiles is defined by the IDX file.
The second type is "floor pieces", these are probably going to be the most common type of tile. Basically it's a standard flat tile and the only thing that changes is the texture applied to it. It looks like the numbering for these is handled by the HD2 definition. Though how these are handled by look up is not yet known.
The third type are "non-flat" pieces. Or at least type is predicted. So far I've done testing with Apple Market which only used set pieces, and commenting out HED and HD2 had no effect, but that's because Apple Market only used set pieces to make the market. The next test I did was with Downtown, which has set pieces and flat pieces. But looking at HED, there are pointers to the stage file, inside there. So next test will be using a different stage, commenting out different types and see what happens. I might try it with Yass Plains and Cardon Forest.
Above is a picture from testing with the IDX file commented out. So testing actually isn't that bad. For most stages (except SUPPORT.BIN), the offsets for the different map files are:
Which means I can go in, comment out IDX, see what tiles remain, comment out HED see which tiles remain, and basically go from there to see if there is a way to filter out which kinds of tiles do what.
As for the decision tree, I think it looks something like the above image. If the height is 0, then create a set piece tile, otherwise what we're looking for is how the game looks up floor tiles and non-flat tiles.
[5:50 PM] RoundShades (⌐⚫ ͜┘⚫): 1) r6 stores 2-byte tile. 2) r6 duplicated to r8 3) just 1 byte is duplicated in upper-byte of r2 (r8 = 00 00 80 B0, r2 = B0 00 00 00) [5:52 PM] RoundShades (⌐⚫ ͜┘⚫): 4) r4 is set up with 190000, r1 is set up with 140000 Then it gets complicated... [5:56 PM] RoundShades (⌐⚫ ͜┘⚫): 5) SHL r2 r6 1h is the opcode. I believe it means "Shift Left" "r6 1h" into r2. The result for B0 is 10160 [5:59 PM] RoundShades (⌐⚫ ͜┘⚫): 6) and r2 0F8h is next. This matches only bits matching F8 against the 10160 and seems to result in r2 = 60 [5:59 PM] RoundShades (⌐⚫ ͜┘⚫): 7) add r1 r2 r1, means r2 is added to r1, and stored at r1. r1 was 140000, now it's 140060 [6:03 PM] RoundShades (⌐⚫ ͜┘⚫): 8) mov r4 [r1-60FCh], so r1 is 140060, r1-60FC is 139F64. the brackets, mean that not the 139F64 but the value AT ADDRESS 139F64 is added to r4 and stored at r4. Result is 195320. THAT sounds like a specific block providing specific offsets.
From discord it looks like Shady was able to track how the game is able to calculate the offset to the STG file from the MDT definition. In this case it looks like the type type 0xB0 with height 0x80, is pointer offset 0x195320. Since the STG file starts at 0x00194000 and each tile header has a size of 0x30, we can find my number for the tile, which is (0x195320 - 0x00194000) / 0x30 = 0x66 or 102 in decimal, so we can see if that lines up with my tile_102, and if we get enough of a dataset, we might be able to bit shift our way to victory. For now I'll continue with commenting out and testing the different lookup tables to see what patterns I can find from that.
Okay, weird. So there are a bunch of zero's in the header. I was picking that up in code and wondering if it was a mistake or not. I might try and check the original EBD file to see if there is any difference. But there are zero's, so we'll compare it to the pose.
And there are a bunch of zero's here. I should check my code to see how I managed this originally. I wonder if I just check for zero. And I should also check to see if the zero's in each table line uo.
Animations have now been implemented in the save state viewer. The way the game stores animations is with two structs. One is a list of poses, the other is a list of pose indexes per frame. For megaman legends 1, the poses are in order for most cases, so to be lazy I just read them in order. But I think there are a few cases where poses are re-used, and that caused missing frames. In theory this should address, that, but if it doesn't hopefully that indicates it's the game's fault and not mine.
Improvements
Ebd Models are now animated
Ebd Models are now scaled by 0.00125
Ebd Model animations are included in exported
Ebd Models are no longer mirrored
Mesh names have been changed to stage name + "em" + entity id
Remaining Issues
Mouth strips are missing from cut-scene meshes (Roll, Megaman, Tron Bonne and Tiesel)
Improvements
Recovered missing quads from a few tiles
Text on textures is now displayed correctly
Scale has been implemented per-flag
Mesh names have been changed to stage name + "t" + tile index
Notes: so if the tile is not the last one, find the next tile, figure out the number of quads by the offset, otherwise, use the current method for the last tile
Errors:
Library tiles still missing a few quads
st04_em2a20 throws an error
st07_em20 throws an error
Comment:
Damn I was hoping to clean up on EBD models. Looks like there are still issues with animations and mouth atlases.
After a recent Save State Viewer update, that changed the way entities and their armature were handled, several entities started to cause RangeErrors to occur when they are selected in the viewer or when attempting to export. Among the culprits are Paprika and the museum curator. The following is a list of known bad entities encountered so far.
Okay made a few changes. The tile view now calculates the number of quads as opposed to attempting to read the number from the file. So I can add yet another rant in the wiki about how nothing makes any freaking sense.
Before update:
After update
For the animations, I went ahead and threw in try,catch to prevent the page from crashing on batch export. Seems to work and the animation the caused the problem will be tossed out for now. Though I have an idea of why the animation table might not being read correctly, and if I am correct in why I think it doesn't work, then that will be another wiki rant about how none of the values in the game can be trusted.
And in yet more insanity caused by the original programmers, I found alternate hands and megamans missing mouth. The issue? These polygons weren't included in the freaking header, and i had to (you guessed it!) calculated the number of polygons from the length of the start of the table to the first quad definition offset. This is really out of hand.
Edit:
For the mouth parts, i think i can implement a patch by detecting model id's and then adding in the missing polygons, and then turn the polygons off/on.
For the animations I think the issue is that the track list is using a pose number that exceeds the number of poses defined for the animation. I'll have to log the offsets and see if that's the case. If this is the problem, then I might have to add in a check for if the number exceeds the number of possible poses and then discard the frame.
Okay, I think the game uses the same skeleton for all of the models with atlas expressions. I turned off polygon 16 (which I think is one set of eyes), and turned on polygon 18 (the mouth), when there are models with 19 bones or more and the model has the bitflag of 0x40 (cutscene). With the animation implementation, and animation tracks implemented, that pretty much wraps up EBD models. There only thing I can think of left to do would be to create a root bone for the origin, rather than using the chest like the game does. Also I've removed translation from the animations, not sure if I should add those in, or potentially export those values somewhere for reference.
This should be most of the memory for general gameplay. And only the start of each position in memory has been listed since depending on the area and the size of the files being copied in, the length of that file in memory will change. So all that's really needed to know is the start of each memory offset, what it does, and what the next one is, and everything else fits in between.
So next step for things with the Player model: - Add list items for viewing specific parts of the mesh - Add the player model to the zip export (though export current should already work, so i might skip on that) - Start looking into player animations
To start off we need to get oriented. Before i started working on the Player model mesh, i started working from 0x01000000 (start of progbin) and worked my way back. Now I have an idea of where the entities and player model are, so we'll go forward and start tracking down what's left. So to start we have:
0xc5580 - Block of data, entity related? 0xd8800 - Block of data, player position, status, health?
It looks like these each have a block of data. 0xd8800 is about where i would expect that player model data to start, but it looks like there is something, but it doesn't look like a mesh, timing or poses. So we start with:
0xD9D00 - Huge list of Animation timings 0xDBA00 - List of two animation timings 0xDC400 - List of one animation timing 0xDC880 - List of one animation timing 0xDCD00 - List of one animation timing 0xDD180 - List of four animation timing
0xDD600 - Huge List of animation poses 0xE9600 - List of two animation poses (0xebf00 end) 0xED600 - List of one animation poses 0xEF600 - List of one animation poses
0xF0000 - Something, doesn't looks like mesh, timing, or pose
0xF1600 - List of one animation poses 0xF3600 - List of four animation poses
0xF5600 - Start of Player mesh chest 0xF7800 - Start of Player mesh head 0xFA000 - Start of Player mesh left arm 0xFB500 - Start of Player mesh special weapon (right) 0xFCA00 - Start of Player mesh right arm 0xFDF00 - Start of Player mesh buster arm (left) 0xFF400 - Start of Player mesh feet
After this starts the second half of memory, with progbin at 0x01000000. So the next step is looking at animations. It looks like most of the animations are in-order. I might try reading the timings to see if there are any animations that aren't in order. If they're all in order i might try and be lazy and just read the poses in order like i did before.
I was hoping to be able to pass in Megaman's animations to my Ebd animation functions, but it looks like there are the results:
So I guess that means that Megaman could potentially be using a different encoded format for the animations, so to determine what's going on, we can start by looking at the data, find methods of testing and then hopefully determine what format the original developers programmed.
So here were have Megaman's list of animation poses, which starts at 0xdd800. The header format is the same as the Ebd files. For each animation, we have a pointer to a list of pointers to poses for that specific animation, and then we have the encoded poses, which are stored as rotation / translation values for each bone.
In in practice what that looks like is the above image. At the top we have the full list of animations. The first pointer is to the bones, and the second pointer is to the idle animation. The idle animation has a header full of pointers, and those pointers to an encoded set of translations and rotations for each bone. Right now I'm going to skip over the bones, and we are able to read them from the Megaman model in from the cut scenes. At that's really important is that Megaman has 15 bones, and looking at the length of the bone list, it seems to match up with what's there. But if we really need to be sure it's the same, we can try reading it, or directly compare against the bytes from the cut scene model's bones.
So first thing we want to do is calculate the stride to see how many bytes are used for each encoded pose. So the first three pointers for the idle animation are 0x0dd7f8, 0x0dd834 and 0x0dd870. So we want to subtract the prior pointer from the next pointer to find the difference which will give us the stride so we get:
So we get 0x3c or 60 bytes of data per encoded pose. Now since each bone is going to need a rotation per frame, 60 bytes divided by 15 bones, is about 4 bytes of data per bone for rotation. The problem is rotation isn't the only value. For the EBD animations, we have the translation for the root bone first, followed by the rotation value for each frame. So we could try throwing a few combinations at our function, and see if something sticks (which I already tried), but since we'll dealing with bits and bytes it would probably be a better idea to go to the game and start getting some test data to get some test points so we can start making some educated guesses at how the poses are encoded.
So for testing, we'll take a look at the timings that are shown above. The header is the same as the animation poses, with one main difference, there is no pointer for the bones. So the first pointer will be the timing controller for the idle animation, the second pointer will be the timing controller for the run cycle, and so on. We see the idle animation timing pointer doesn't point to a list of pointers, but it points directly to the animation tracks. The way the tracks work is they have a pose index (which matches up with the pointer number on the animation poses), the second byte is the number of frames that pose is repeated, and the last byte in the first dword controls where to repeat or not. A value of 0x80 means repeat, a value of 0xff means stop. And there are other codes that I have seen but don't know what they do, so it could "transition" or something to that effect. For testing all we need to know is where the tracks are in memory, that way we can tell the game to keep repeating the first track, and then we can edit that pose and see what the effects are. So we change the value 0x0f010000 to 0x00010080.
So what that does is freezes Megaman in position (you can still see Dustin running around in the background) by causing the game to continuously repeat his 0 index pose. Which means that anything we write to that pose will be what gets displayed on the screen. And we can use that to start tracking down which parts of Megaman get moved when different ranges of bytes are changed. So now we can start trial and error. Each encoded pose is 0x3c bytes. To make things easier we can change one dword at a time. One dword is 0x04 bytes. Which means we have can comment out each dword one at a time, and record the results.
I ended up posting most of my testing to discord. So in order to backup my progress, I will go ahead and record all of the information here as well. So what I ended up doing is using the idle animation for testing. First I changed the dword at 0xd9e00 from 0x0f010000 to 0x00010080. What this does is it causes the first frame of the idle animation to repeat. So any changes that I make in memory will be reflected once the save state is loaded. The next step is that I commented out 0x3c bytes from 0x0dd7f8. As this is the first frame in the idle animation. What that did is it put the character in a t-pose, to make it easier to track any changes in the animation.
And I took a short video of the testing process, where I create a save state, change a byte, observe the result, change a byte and observe again.
And using this bone guide that I created,
I was finally able to narrow down that this is how the animations are encoded for Megaman:
To try and summarize what the image above shows, it looks like the game uses a very unique form of encoding that seems to alternate between 4 and 5 bytes per bone. The first encoded value is the root position, which uses two bytes for the x axis value, 1 byte for the y axis value and then 2 bytes for the z axis value. Then weirdly for the root rotation alternates to 1 byte for the x axis, 2 bytes for the y axis and 1 byte for the z axis. This runs counter to the format used in the Ebd files, which uses 1.5 bytes (12 bits) per axis value. And after testing the higher and lower bytes several times, it doesn't look like the game uses 1.5 bytes, instead using 1 bytes and 2 bytes, as the most significant bits seems to be in a place that suggested that's the case.
In terms of implementation, it doesn't look like I have a complete picture. When I went to use this scheme, in my PBD animation function, I ended up with the following result:
So it looks like there is still more testing to do to try and find exactly what's going on. I went through today and commented out most of the animation, just looking into bone 1 (the head) to make it easy to debug. And I found that the x rotation seemed pretty consistent, and the y value was never used, the reason the animation looks terrible is because the z-value fluctuates causing the character's head to spin around on the side. So I think the next step will be to add more data points to the animation, try more frames in the idle animation. And comment everything out except bone 1, and see which values have what result.
Okay so a video about the save state viewer is up. In terms of things left to do:
- debug animations for ebd (apply y translation and timing fixes) - add more mesh options for megaman (special weapon, head, feet) - change megaman texture to load from framebuffer
So an interesting observation, it looks like anything that's not a floor tile is defined with IDX. So here's cardon forest:
And here's the same perspective with IDX commented out:
So I was thinking there were three types of tiles, floor ties, non-flat tiles and double sized tiles. It looks like there's just floor and non-flat tiles. So I'll have to do some digging and start data mining the MDT and IDX to see what relationships could exist between these two.
Okay did some more testing and I've started to notice something I never caught on before, the MDT file seems to have room numbers.
So this is an example of the MDT file with tile type and tile height. And you can see that 00, 01, 02 get defined by both room 2 and room 3. Now the problem is that I don't know how the game manages the look up table. SO i guess that means I need to take a look at the IDX file to see if I can find anything that would indicate which index gets used for the look up table.
Okay so in terms of tracking down references, we can try and go with a targeted approach. And if that doesn't work we can mess around with mutliple rooms in cardon forest, which is something i predict we will have to do anyways. So in the Fletter, Roll's room is stage 0x1b, room 6. And there are four types of tiles.
For each of the types of tiles we have:
So since we have the STG file, we can debug at what offset it is. And once we know what offset it is, we can loop through the IDX file and see where the reference is located.
Edit:
Tile 129 offset is 0x195830. Next we try and look it up in the IDX file.
8b06 door st1b_131 8c06 table st1b_132 8d06 machine st1b_129 8e06 bed st1b_130
Trace
--- Read from MDT ---
Starting trace with u16 at 0x801662d2 Reading u16 at 0x801662ca Reading u16 at 0x801662d0 Reading u16 at 0x801662d2
--- Read From IDX ---
Reading u32 at 0x80160ea4 Reading u16 at 0x80160ea2 Reading u16 at 0x80160e9c Reading u16 at 0x80160e9e Reading u16 at 0x80160ea0
--- Read from STG ---
Reading u32 at 0x801ad8a0 Reading u16 at 0x801ad898 Reading u16 at 0x801ad89a Reading u16 at 0x801ad89c Reading u16 at 0x801ad89e Reading u32 at 0x801ad8a0 Reading u16 at 0x801ae36c Reading u32 at 0x8008aa24 Reading u32 at 0x801ad8b0 Reading u16 at 0x801ad8a8 Reading u16 at 0x801ad8aa Reading u16 at 0x801ad8ac