kion
Arukoitan
@kion_dgl
Posts: 193
|
Post by kion on Dec 31, 2018 12:45:51 GMT -5
Video for IDX editing:
|
|
kion
Arukoitan
@kion_dgl
Posts: 193
|
Post by kion on Jan 1, 2019 7:25:09 GMT -5
Another video for MDT editing (loading wrong areas) Quick Edit for notes (not really related to the MDT video): Start of tile header: 0x001191180 Start of high LOD Tile Defition: 0x001192c10 Start of med/low Tile Defition : 0x001192c34 Start of new pointer to follow : 0x0011a969c 0x1d18 0x015824 24580100 000015820: 00000000 fffd0102 fffd0102 c4a71a80 ................ 000015830: 01000000 5ffeb1fe 5f000101 74a81a80 ...._..._...t... 000015840: 01000000 0fff61ff 0f00b100 7ca81a80 ......a.....|... 000015850: 01000000 0fff61ff 5bfffdff 84a81a80 ......a.[....... 000015860: 01000000 fffd0102 ff000102 8ca81a80 ................ 000015870: 01000000 bfff0102 fffd0102 8ca81a80 ................ 000015880: 01000000 fffd0102 fffd61fe 8ca81a80 ..........a..... 000015890: 01000000 0fff61ff 4fffb100 8ca81a80 ......a.O....... 0000158a0: 01000000 5ffeb1fe 4f000102 8ca81a80 ...._...O....... 0000158b0: 01000000 1ffea1fe fffd01ff 94a81a80 ................ 0000158c0: 01000000 fffd0102 fffd0102 9ca81a80 ................ 0000158d0: 01000000 9ffd0102 5ffe0101 a4a81a80 ........_....... 0000158e0: 01000000 00000000 00000000 00000000 ................ 0011a9690: 00000000 00000000 00000000 fffd0102 ................ 0011a96a0: fffd0102 c4a71a80 01000000 5ffeb1fe ............_... 0011a96b0: 5f000101 74a81a80 01000000 0fff61ff _...t.........a. 0011a96c0: 0f00b100 7ca81a80 01000000 0fff61ff ....|.........a. 0011a96d0: 5bfffdff 84a81a80 01000000 fffd0102 [............... 0011a96e0: ff000102 8ca81a80 01000000 bfff0102 ................ 0011a96f0: fffd0102 8ca81a80 01000000 fffd0102 ................ 0011a9700: fffd61fe 8ca81a80 01000000 0fff61ff ..a...........a. 0011a9710: 4fffb100 8ca81a80 01000000 5ffeb1fe O..........._... 0011a9720: 4f000102 8ca81a80 01000000 1ffea1fe O............... 0011a9730: fffd01ff 94a81a80 01000000 fffd0102 ................ 0011a9740: fffd0102 9ca81a80 01000000 9ffd0102 ................ 0011a9750: 5ffe0101 a4a81a80 01000000 00000000 _............... 0011a9760: 00000000 00000000 00000000 fffd0102 ................ No context edit: 000015780: 01000000 00000000 00000000 00000000 ................ 000015790: 00000000 fffe0101 fffd0102 c4a71a80 ................ 0000157a0: 01000000 fffe0101 fffd0102 5ca81a80 ............\... 0000157b0: 01000000 fffe0101 fffd0100 6ca81a80 ............l... 0000157c0: 01000000 fffe0101 ffff0102 6ca81a80 ............l... 0000157d0: 01000000 00000000 00000000 00000000 ................ 0x11a9438 0x11a94c8 0x1240488 0x1240518 Edit: UV Values? 001194cf0: df3edf00 fea0febf e0a0e0bf febfe0bf .>.............. 001194d00: fea0e0a0 1ffe1fe0 3ffe3fe0 16c11fe0 ........?.?..... 001194d10: 00c100e0 1ffe1fe0 3ffe3fe0 2dc11fe0 ........?.?.-... 001194d20: 00c100e0 1fe01ffe 00e000fe 1fe016c1 ................ 001194d30: 3fe03fc1 1fe01ffe 00e000fe 1fe02dc1 ?.?...........-. 001194d40: 3fe03fc1 e017ee21 e059ee59 febfe1bf ?.?....!.Y.Y.... 001194d50: fea0e1a0 e1bfe1a0 febffea0 bf3e803e .............>.> 001194d60: bf008000 bf428042 bf408040 96568056 .....B.B.@.@.V.V 001194d70: 96548054 bf5b805b bf448044 1ffe1fdf .T.T.[.[.D.D.... 001194d80: 3ffe3fdf 07c11fdf 00c100df 1fdf1ffe ?.?............. 001194d90: 00df00fe 1fdf37c1 3fdf3fc1 1fdf07c1 ......7.?.?..... 001194da0: 37c137c1 fe22fe5a f018f05a df3fc03f 7.7..".Z...Z.?.? 001194db0: df00c000 fe00fe17 e000e017 e0a0fea0 ................ 001194dc0: e0bffebf 1ffe1fe0 3ffe3fe0 27c11fe0 ........?.?.'... 001194dd0: 00c100e0 1ffe1fe0 3ffe3fe0 1fc11fe0 ........?.?..... 001194de0: 00c100e0 1fe027c1 3fe03fc1 1fe01ffe ......'.?.?..... 001194df0: 00e000fe 1fe01ffe 00e000fe 1fe01fc1 ................ 001194e00: 3fe03fc1 e018ee22 e05aee5a fea0febf ?.?....".Z.Z.... 001194e10: e1a0e1bf febfe1bf fea0e1a0 1ffe1fdf ................ 001194e20: 3efe3edf 1edf22c0 3edf3ec0 1efe1edf >.>...".>.>..... 001194e30: 3efe3edf 1fdf36c0 3edf3ec0 1fdf1ffe >.>...6.>.>..... 001194e40: 00df00fe 1edf1efe 00df00fe 1bc01fdf ................ 001194e50: 00c000df 08c01edf 00c000df 22c01edf ............"... 001194e60: 08c008c0 1fdf1bc0 36c01bc0 f018fe22 ........6......" 001194e70: f05afe5a 84549154 84569156 8044bf44 .Z.Z.T.T.V.V.D.D 001194e80: 805bbf5b bf408040 bf428042 bf008000 .[.[.@.@.B.B.... 001194e90: bf3e803e 91549e54 91569e56 bf008000 .>.>.T.T.V.V.... 001194ea0: bf3e803e bf408040 bf428042 8044bf44 .>.>.@.@.B.B.D.D 001194eb0: 805bbf5b 1fc11fe0 00c100e0 2ffe1fe0 .[.[......../... 001194ec0: 3ffe3fe0 0ffe1fe0 3ffe3fe0 1fc11fe0 ?.?.....?.?..... 001194ed0: 00c100e0 1fe01fc1 3fe03fc1 1fe00ffe ........?.?..... 001194ee0: 00e000fe 1fe02ffe 00e000fe 1fe01fc1 ....../......... 001194ef0: 3fe03fc1 e000fe00 e017fe17 df00c000 ?.?............. 001194f00: df3fc03f e000fe00 e017fe17 df3fdf00 .?.?.........?.. 001194f10: c03fc000 e1bfe1a0 febffea0 e1a0fea0 .?.............. Need to narrow it down. Edit: Got it. 001194e00: 3fe03fc1 e018ee22 e05aee5a fea0febf So the weird part about this is the pointers in the header only seems to point to the scale, pallet (and probably indices) definition. Which seems to leave out the vertices and UV's entirely. Also the scale struct is only 0x24 bytes in length, which leaves a massive part of the file unaccounted for. So I decided to go through and start commenting out data to see what would change, and found that yes, the UV's started to be commented out, outside of the scale pointer that I found before. Specifically the center two dwords of e018ee22 e05aee5a. Edit: Did a check on the UV values to make sure they looked correct. So we have the bytes e018ee22 e05aee5a, which when mapped out in decimal gives: U: 0xe0 (224) V: 0x18 (24) U: 0xee (238) V: 0x22 (34) U: 0xe0 (224) V: 0x5a (90) U: 0xee (238) V: 0x5a (90) So the image is only 128px in width, but we have U values that are wider than that. Since we know the game copies images to the framebuffer and then treats 256x256 areas of the framebuffer as a single texture, it could be the case that this texture is being textured into the right half of a framebuffer page, meaning we can subtract 128 to adjust. U: 0x60 (96) V: 0x18 (24) U: 0x6e (110) V: 0x22 (34) U: 0x60 (96) V: 0x5a (90) U: 0x6e (110) V: 0x5a (90) Which when mapped out in Gimp, gives the area below. So yes, these look like the UV values we would expect. Now the problem is that we can see the effect of these values, but I haven't seen anything pointing to these values. Which is alittle concerning, but I'll keep commenting things out and see what happens. Edit: So after commenting out more of the file, I'm kind of confused, because a large portion of the file just seems to be UV values. What I'm not seeing is pointers, and it seems like there is a lot I can comment out that doesn't have any visual effect. So what I think I'll do is break the file down into sections and then make notes one the effects happened for each section. The file starts around: 0x001190b50 and ends: 0x0011aabb0. 0x0011aabb0 - 0x001190b50 is a file length of 0x1A060. And if I break off each segment into a length of something like 0x800. 0x1A060 / 0x800 is around 52. Doing more bytes would probably be easier, but the ROM track data has extra data every 0x800 bytes or so that if you overwrite it, the file doesn't read correctly. But In general I might try and comment out 0x1000 bytes at a time or so and see what happens. 00000000 00000000 00000000 00000000 0x001190b50 ~ 0x001191c60 seems to be the header. 0x001191c60 ~ 0x0011937f0 seems to be scale definitions 0x0011937f0 ~ 0x0011949c0 no change? 0x0011949c0 ~ 0x001196b40 UV values 0x001196b40 ~ 0x0011998c0 a few more UV values / no change 0x0011998c0 ~ 0x00119c9c0 no change? 0x0011a8d50 ~ 0x0011aa7a0 no visual change, fell through the floor (collision?) 0x0011aa8e0 ~
Hmmm... specifically it was this area where I (and NPC's) started falling throught the floor. 0011a93d0: 00000000 c4a71a80 01000000 8fff7100 ..............q. 0011a93e0: ff008101 fca71a80 01000000 fffd0102 ................ 0011a93f0: ffff8101 04a81a80 01000000 fffd0102 ................ 0011a9400: ffff0102 0ca81a80 01000000 fffd0102 ................ 0011a9410: ffff0102 54a81a80 01000000 fffd0100 ....T........... 0011a9420: ffff0102 3ca81a80 01000000 ffff0102 ....<........... 0011a9430: ffff0102 44a81a80 01000000 00000000 ....D........... 0011a9440: 00000000 00000000 00000000 fffe0101 ................ 0011a9450: fffd0102 c4a71a80 01000000 fffe0101 ................ 0011a9460: fffd0102 5ca81a80 01000000 fffe0101 ....\........... 0011a9470: ffff0102 64a81a80 01000000 fffe0101 ....d........... 0011a9480: fffd0100 64a81a80 01000000 00000000 ....d........... 0011a9490: 00000000 00000000 00000000 fffe0101 ................ 0011a94a0: fffe0101 c4a71a80 01000000 fffe0101 ................ 0011a94b0: fffe0101 5ca81a80 01000000 fffe0101 ....\........... 0011a94c0: fffe0101 64a81a80 01000000 00000000 ....d........... 0011a94d0: 00000000 00000000 00000000 fffe0101 ................ 0011a94e0: fffd0102 c4a71a80 01000000 fffe0101 ................ 0011a94f0: fffd0102 5ca81a80 01000000 fffe0101 ....\........... 0011a9500: fffd0100 6ca81a80 01000000 fffe0101 ....l........... 0011a9510: ffff0102 6ca81a80 01000000 00000000 ....l...........
For now I'll take a note about it and continue to comment data out to see what it does. Specifically it seems to be some properties paired with a pointer. Interesting part is there is no change visually, the stage is there, you just fall through the floor.
|
|
kion
Arukoitan
@kion_dgl
Posts: 193
|
Post by kion on Jan 9, 2019 11:22:44 GMT -5
Okay, so I seem to have hit the end of the STG file. And I'm kind of surprised that I didn't see anything that seems to look like vertices, so I might have to ask Trege on Discord to see if he's been able to isolate any vertice data in memory. What I did manage was get a general sense of how the STG file format is structured. There seems to be a header, a definition, which contains the image, pallet and then what I think are the indices for the uv and the vertices. For the UV that was easy as a large portion of the file seems to dedicated to UV values. And then after that, towards the end of the file seems to be pointers for collision definition. What confuses me is that if the floor tiles are anything like the enemy model definition there should be a vertex list somewhere that the vertice ist is referecing by index. The problem is that going through the file I didn't find anything that uses the x,y,z, nop short format that the other models use. But when I don't see what I expect to see, it's probably because my expectations are wrong. One thing that I find really surprising is the scale bits in the definition. And what this seems to be indicating is that rather than 16 bit values, the developers are using less bits for a simple representation of a tile and then scaling it up to fit the map. So the vertice data could be crammed into a very small number of bytes. And in the definition part of the structure I was able to scew some of the models, but not in anyway I would expect, which is why I think those are indice values and not the vertices themselves. So the next step is to probably take another parse of the file to try and outline sections, and then maybe take a peak into other files to see if there's anything I'm missing. And I can take another look at the bytes after each tile definition. Edit: Okay, wow. Found something unexpected. Still no vertices, but something interesting none the less. So we're looking at this block of data. 001192c10: 02430100 1c00c07f 1d00887f bc7a1940 .C...........z.@ 001192c20: 84091a08 7c7c1941 c40c1a0a 4c7a1940 ....||.A....Lz.@
And the first thing we do is change the scale byte to 0x20, to give the following result:
Now from here my plan was to start looking into the four bytes after this value, to see if I could change anything and what effects it had on the tile that I had managed to isolate. 001192c10: 02430100 1c00c07f 1d00887f bc7a1940 .C...........z.@ 001192c20: 84091a08 7c7c1941 c40c1a0a 4c7a1940 ....||.A....Lz.@ And before I haven't had much luck changing these values, as any small changes seem to have a habit of weird things happening as opposed to any kind of predictive behavior. So instead of commenting out like before I figured I'd try to change bits at a time to see what happens, and found that the second byte has some really interesting properties. So changing this byte from 0x09 to 0x00 gives: 001192c10: 02430100 1c00c07f 1d00887f bc7a1940 .C...........z.@ 001192c20: 84001a08 7c7c1941 c40c1a0a 4c7a1940 ....||.A....Lz.@ And the result is that there is now a door in the middle of the room. 01 02 03 04 So there's a byte for the type definition. But it still bothers me. Where are these vertices stashed?
|
|
kion
Arukoitan
@kion_dgl
Posts: 193
|
Post by kion on Jan 13, 2019 3:45:58 GMT -5
Since I haven't been able to find the vertices or indice data in the STG file, I guess the next step is to poke around the prog file. Though as long as I'm here I might as well do some testing with the segment definitions. So for notes, this is located at 0x001192c10 in the bin file I'm using. And is right in front of the highneck record shop. So the struct has three kinds of data, a header, pallet / texture (same format as characters) and each segment. The header is pretty simple. The first byte is the number of pallets. The second byte is the number of segments. 0x40 seems to be some flag and so is there 0x01 byte after that. I rewrote this value from 0x02430100 to 0x02030000 and saw no obvious change in the game. The next two dwords are pallets / textures. I haven't done much testing with these aside from replacing values. But in general these look like the same format used in the characters. You have an unsigned short value for the pallet and the texture respectively, and then you bit mask and shift over to get the x,y positions of the pallet and texture in the framebuffer. From there you have the segment definitions for the parts that make up the tile. Each segement definition consists of 8 bytes. Each byte seems to have a pretty different function and some bytes even use the top and bottom bits. So I think the best option is to make a table for each byte and describe the changes. Byte 0
| Changing this byte causes UV values to be mapped incorrectly
| Byte 1
| Changing this byte causes UV values to be mapped incorrectly
| Byte 2
| Changing this byte causes UV values to be mapped incorrectly
| Byte 3
| Top four bits are scaling. So 0x10 is 2x size, 0x40 is 4x size, 0x80 is 8x size. Haven't tested if these can be combined. The lower four bits are the pallet select. Most often this value is 0, but if it's 0x01 or 0x02 it means the segement uses that number pallet declared at the beginning.
| Byte 4
| Chaning this byte causes the geometry to deform (in unpredictable ways)
| Byte 5
| Segment type selection. It looks like there is either a list or predefined segment types. Changing this byte will change the geometry appearance of the tile segment. Not sure currently sure where the list for these is defined.
| Byte 6
| Chaning this byte causes the geometry to deform (in unpredictable ways)
| Byte 7
| Number of quads. If this number is reduced, square shape holes will appear in the geometry. If this number is increased, extra squares (from the following definition) will be drawn with the same pallet/texture combination.
|
So aside from the geometry and uv values (that cause some seriously crazy effects), for the most part we have some generally usefule values. We have scaling, pallet number, geometry type, and the number of quads. The other bytes to to be fixed values for specfic purposes. If we don't need them for anything, then we can get by without knowing what their function is. And otherwise if we're missing information later, then that might give us a hint for what these values do. So next step is to start messing with the .prog file. Edit: So I've started making my way through the .prog file. And I'm making a few observations. One, that the file is stupidly long. And secondly that there is a lot in this file that doesn't seem to actually do anything. At the beginning of the file seems to be some kind of definition for the light and distance with respect to the fragment shader. And I whitenessed a lot of trippy effects. With all of the room turning black, to having a green tint at the end of the render distance, to having a green tint for everything. The was also several times where the color would become inverted for close Megaman, normal and inverted again. Though beyond the lighting effects declared early on, the next change I noticed was the NPC's being commented out. And beyond that I really haven't noticed any change (aside from the game causing reboots because of errors) So far I'm at offset 0x0011828c0, and making a note of it so I don't loose my spot in the file.
|
|
|
Post by Rockman Striker on Jan 15, 2019 0:27:31 GMT -5
Even though I can't quite understand everything you wrote I'm happy to see you're finding lots of stuff, for example, I always thought that the textures were shadeless, I had no idea that it actually uses some type of shading scripting. MML2 actually uses shading effects to render shadows from the clouds and making darker environments inside the ruins but the first game doesn't seem to do that, mainly to achieve the "cell shading" effect. Interesting.
|
|
kion
Arukoitan
@kion_dgl
Posts: 193
|
Post by kion on Jan 15, 2019 10:36:00 GMT -5
Even though I can't quite understand everything you wrote I'm happy to see you're finding lots of stuff, for example, I always thought that the textures were shadeless, I had no idea that it actually uses some type of shading scripting. For playing around with the prog file, I dind't know what to expect, but it definitely wasn't the result I'm seeing. I also had the impression that MML used the pallet/texture combination from the framebuffer and didn't apply any post processing effects. But if parts of the scene can be tinting green, or inverted, or turn black entirely, then that does suggest that there is some fragment multiplication going on. So it could be that the effect of the game fading off into the distance, they multiply by a white fragment to give the effect. Or something. I honestly wasn't expecting this and thought I had eaten some kind of psycodelic mushroom when running across these effects. It doesn't really fit into my normal category of documenting information for the purpose of exporting, but I might make a video for what offset to edit to get some of these phsychodelic effects. Also randomly found where the game colors the can that you kick around. I think Xinus and others have commented that it doesn't have UV values. So it looks like that's done in the program and not from data. 001182ae0: 1000a2af 8000033c 80806334 00020224 Also can scaling: I'm messing around with things blocks at a time. so if someone wants to mess with this the offset was somewhere between 0x001182cc0 and 0x001182d10. Edit: Dimming the lights 001185830: 30000000 github.com/mirror/pcsxr/blob/51b2ad7c2bdc83ed6a0f297679233bc4d19de027/libpcsxcore/misc.c
|
|
Trege
Poh
oro?
Meddling with Legends 1, Legends 2 and Mega Man 64 data.
Posts: 463
|
Post by Trege on Jan 16, 2019 20:41:26 GMT -5
Man I've only ever been to tint/affect lighting on random objects like Megaman or cars you found the whole room's light level, it's kinda a thought a lot of people had that maybe Legends was supposed to have day/night in it, because of textures in the beta versions for lit up windows and the apple market sign glowing at night, also I believe entering Cardon Ruins also affects the lighting as everything is shaded slightly darker there including Megaman himself and his face. Messing with that was how I found how to alter his armor colour which I assume is just a specific light level/colour set for him. There's even a code I made using RAM values and Shady's cheat engine to gameshark conversion method. "Armor Code" D00B531C 0000 50002602 2003 800B1169 4500 D00B531C 0003 50002602 00A1 800B1167 A000 D00B531C 0002 50002602 00A1 800B1169 0000 Either way this is pretty awesome stuff you're finding Kion, I actually want to try and recreate downtown's night look someday, because someone actually managed to force downtown at night to load in the beta so that was interesting sadly they didn't share any info about it at all for some reason.
|
|
kion
Arukoitan
@kion_dgl
Posts: 193
|
Post by kion on Feb 6, 2019 23:34:57 GMT -5
Okay so this took an unexpected turn. Still looking for where the game keeps the vertices for the maps. But my approach has taken an unexpected turn, so going to write it out here and maybe I can look back at how stupid i was later. So my first approach was to try using no$psx. And if you haven't played around with this emulator, you should. It has a VRAM viewer, and assembly stepper. My problem is that I'm not familiar enough with assembly to trace back to where the data is. So in the VRAM viewer, you can see the list of draw calls being made every frame, and it's really easy to see the texture of what's being drawn. My problem is that I don't know what's in the lower right hand corner of the view. It has the text "CPU.PC" 8007F08C. The problem is going to that offset in no$psx traces through the assembly and I can't tell what's being copied into the registers from where, and being rendered when and how. So I decided I might go back to the save state strategy I was using before with pcsxr. The are a few problems with the save states on pcsxr, one being that the save states are compressed, and the second is that on top of being compressed, everything the emulator needs to save to reproduce that state is also saved inside the emulator, so it's hard to tell specifically what part of the save state is which memory, make changes that aren't a crap shoot and then trace that back to where the files are in the ROM from there. So I started thinking, "can I take advantage of the fact that pcsxr is open source and change how the emulator works to make this easier on me? And the answer seems to be "yes". github.com/pcsxr/PCSX-Reloaded/blob/master/pcsxr/libpcsxcore/misc.cIt looks like the save states are managed in the misc.c file, and there are pretty much four function, "saveState", and "loadState" which handle the saving and loading, and "saveStateGZ" and "loadStateGz" which actually bundle the memory into a file, compress and write it, or decompress and read it. Which means I can try to simply take the decompression out of the emulator and make things easier for myself. And it looks like I can actually do one better. int SaveStateGz(gzFile f, long* gzsize) { int Size; unsigned char pMemGpuPic[SZ_GPUPIC];
//if (f == NULL) return -1;
gzwrite(f, (void *)PcsxrHeader, sizeof(PcsxrHeader)); gzwrite(f, (void *)&SaveVersion, sizeof(u32)); gzwrite(f, (void *)&Config.HLE, sizeof(boolean));
if (gzsize)GPU_getScreenPic(pMemGpuPic); // Not necessary with ephemeral saves gzwrite(f, pMemGpuPic, SZ_GPUPIC);
if (Config.HLE) psxBiosFreeze(1);
gzwrite(f, psxM, 0x00200000); gzwrite(f, psxR, 0x00080000); gzwrite(f, psxH, 0x00010000); gzwrite(f, (void *)&psxRegs, sizeof(psxRegs));
// gpu if (!gpufP)gpufP = (GPUFreeze_t *)malloc(sizeof(GPUFreeze_t)); gpufP->ulFreezeVersion = 1; GPU_freeze(1, gpufP); gzwrite(f, gpufP, sizeof(GPUFreeze_t));
// SPU Plugin cannot change during run, so we query size info just once per session if (!spufP) { spufP = (SPUFreeze_t *)malloc(offsetof(SPUFreeze_t, SPUPorts)); // only first 3 elements (up to Size) SPU_freeze(2, spufP); Size = spufP->Size; SysPrintf("SPUFreezeSize %i/(%i)\n", Size, offsetof(SPUFreeze_t, SPUPorts)); free(spufP); spufP = (SPUFreeze_t *) malloc(Size); spufP->Size = Size;
if (spufP->Size <= 0) { gzclose(f); free(spufP); spufP = NULL; return 1; // error } } // spu gzwrite(f, &(spufP->Size), 4); SPU_freeze(1, spufP); gzwrite(f, spufP, spufP->Size);
sioFreeze(f, 1); cdrFreeze(f, 1); psxHwFreeze(f, 1); psxRcntFreeze(f, 1); mdecFreeze(f, 1);
if(gzsize)*gzsize = gztell(f); gzclose(f);
return 0; } Basically anything that calls "gzWrite" is information being written to the save file. And the part I'm mostly interested in is: gzwrite(f, psxM, 0x00200000); gzwrite(f, psxR, 0x00080000); gzwrite(f, psxH, 0x00010000); gzwrite(f, (void *)&psxRegs, sizeof(psxRegs));
And so what I think I can do is in addition to the compressed save state that automatically gets written (but the less i change the easier it is), i think i can have the emulator write the non-compressed memory to a specific file on the computer. And then load that specific file again when loading. And that should allow me to build a better map of what's in memory where, search against it, and be able to edit and look for changes. It looks like the PSX has two MB of user memory, and one MB of graphical memory. We can ignore the graphical memory since it contains the screen buffer and textures, so the environment vertices should be somewhere in user memory. Rather than trying to go in and change random things, I'm going to try and map out the memory. 0 ~ 64KB PSX kernal 0x100000 - ..\PROGBIN\R3_ST04.BIN 0x194080 - ..\MAP\DATA\ST04_MAP.MDT 0x194000 --- Player --- ..\PLBIN\PL00L00S.PBD found Start index: 0xdcd00
..\PLBIN\PL00L00M.PBD found Start index: 0xf1600
..\PLBIN\PL00R00S.PBD found Start index: 0xdc400
..\PLBIN\PL00R00M.PBD found Start index: 0xed600
..\PLBIN\PL00R03S.PBD found Start index: 0xdc880
..\PLBIN\PL00R03M.PBD found Start index: 0xef600
..\PLBIN\PL00L04S.PBD found Start index: 0xdd180
..\PLBIN\PL00L04M.PBD found Start index: 0xf3600
..\PLBIN\PL00H01P.PBD found Start index: 0xf7800
..\PLBIN\PL00M00S.PBD found Start index: 0xd9d00
..\PLBIN\PL00M00M.PBD found Start index: 0xdd600
..\PLBIN\PL00B00P.PBD found Start index: 0xf5600
-- Stage --
..\PROGBIN\R3_ST04.BIN found Start index: 0x100000
..\MAP\DATA\ST04_MAP.MDT found Start index: 0x164000
..\SCR\DATA\ST04.STG found Start index: 0x194000
..\SCR\DATA\ST04.HED found Start index: 0x13a000
..\SCR\DATA\ST04.HD2 found Start index: 0x139f00
..\EMBIN\ST04_00A.EBD found Start index: 0x16c000
..\MESS\ST04_00.MSG found Start index: 0xb6db0
..\MESS\ST04_00.MSG found Start index: 0xb6db0
Wanted to find what parts of the memory matched the ROCK_NEO.EXE file: Start: 0xb080, Length: 0x800 (CD meta data) Start: 0x10000, Length: 0x72240 Start: 0x82240, Length: 0x90e0 (commenting it out causes no change) Start: 0x8b3e0, Length: 0x50a0 (commenting it out causes no change) Start: 0x904a0, Length: 0x440 Start: 0x90940, Length: 0x340 (commenting it out causes no change) Start: 0x90c80, Length: 0x1c0 (commenting it out causes no change) Start: 0x92020, Length: 0x2c0 (commenting it out causes no change) Start: 0x92340, Length: 0x860 (commenting out causes camera issues, draw issues) Start: 0x92ba0, Length: 0x4b80 (commenting out causes characters to not be drawn) Start: 0x97860, Length: 0x4e0 (commenting out removes collision) Start: 0x97d60, Length: 0x280 Start: 0x98020, Length: 0x140
|
|
kion
Arukoitan
@kion_dgl
Posts: 193
|
Post by kion on Feb 10, 2019 16:28:10 GMT -5
Did I get lucky? Editing out these lines causes the geometry to "pinch" to one location. 000024810: 00000000 00000000 00000000 00000000 ................ 000024820: 00000000 00000000 00000000 00000000 ................ 000024830: 00000000 00000000 00000000 00000000 ................ 000024840: 00000000 00000000 00000000 00000000 ................ 000024850: 00000000 00000000 00000000 00000000 ................ 000024860: 00000000 00000000 00000000 00000000 ................
. Though to be honest, that's not how I would expect the geometry to act when changed. Quick Note: ShadyRounds (⌐⚫ ͜┘⚫): Okay, so, when toggling a tile via CheatEngine and scanning changed-values between them, I get about 2000 detected. A few are in psx-ram 9C184-A9068, the rest are in psx-ram 1B8ACC-1D9DAC, and these values change when I toggle a tile-ID located in psx-ram 164314 in the subruin ladder-room right outside the apple market entrance. 0000a2f40 Okay, so commenting out from 0x9c184 and above seems to remove the NPC's from the game. So I guess this might be the location where the game stores the position of NPC's and updates their position and state each frame. Specifically commenting out these lines removed the jetlag bakery lady from the apple market. 0000a0000: 00000000 00000000 03240901 00008300 .........$...... 0000a0010: 01000100 44000000 30c71080 80f3ffff ....D...0....... 0000a0020: 740e0000 000080f3 0000ffff 0000740e t.............t. 0000a0030: 00000000 80f3ffff 740e0000 000080f3 ........t....... 0000a0040: 0000ffff 0000740e 00000000 00000000 ......t.........
So commenting out NPC's to get a clean rip (if we ever figure out an alternative way to rip frames right from the GPU) is steadily becoming a viable option. 0x0000ad100 0x0000b3100 0x0001d8000 Commented out everything from 0x0001b8000 to 0001f8000 and no change at all. So these must be temp values. Might be a good idea to clean out this range directly from pcsxr. 0x0001f8000 - 0x0001fffff(EOF) doesn't seem to contain too many values. Mostly pointers which cause the game to crash if removed. So in general though, this is a large area of the 2M user memory, that we can cross off. Okay so: 0x000000000 ~ 0x000010000 = PSX Kernal 0x000010000 ~ = ROCK_NEO.BIN 0x0000b6db0 ~ = ..\MESS\ST04_00.MSG 0x0000dba00 ~ 0x0000dc400 = Megaman Bones 0x0000dc400 ~ 0x000100000 = Megman Polygons 0x000100000 ~ 0x000139f00 = ..\PROGBIN\R3_ST04.BIN 0x000139f00 ~ 0x00013a000 = ..\SCR\DATA\ST04.HD2 0x00013a000 ~ 0x000164000 = ..\SCR\DATA\ST04.HED 0x000164000 ~ 0x00016c000 = ..\MAP\DATA\ST04_MAP.MDT 0x00016c000 ~ 0x000194000 = ..\EMBIN\ST04_00A.EBD 0x000194000 ~ 0x0001b8000 = ..\SCR\DATA\ST04.STG 0x0001b8000 ~ 0x0001f8000 = /dev/null 0x0001f8000 ~ 0x0001fffff = pointers
Wrote a program to check if how much of the files in memory match their source files from the ROM. 1. Scan complete ..\PROGBIN\R3_ST04.BIN was found in ST04.BIN at 0x800 Match length was 0x11da0 bytes
2. Scan complete ..\SCRDATA\ST04.HD2 was found in ST04.BIN at 0x2d800 Match length was 0x110 bytes
3. Scan complete ..\SCR\DATA\ST04.HED was found in ST04.BIN at 0x2c800 Match length was 0x250 bytes
4. Scan complete ..\MAP\DATA\ST04_MAP.MDT was found in ST04.BIN at 0x14000 Match length was 0x1d0 bytes
5. Scan complete ..\EMBIN\ST04_00A.EBD was found in ST04_00.BIN at 0x800 Match length was 0x22930 bytes
6. Scan complete ..\SCR\DATA\ST04.STG was found in ST04.BIN at 0x15000 Match length was 0x7c50 bytes
1. ..\PROGBIN\R3_ST04.BIN Interesting find, it looks like there is more data between the ..\PROGBIN\R3_ST04.BIN and ..\SCR\DATA\ST04.HD2. On the left is the memory, on the right is the ROM. 2. ..\SCRDATA\ST04.HD2 Matches 1:1 with ROM 3. ..\SCR\DATA\ST04.HED So ..\SCR\DATA\ST04.HED matches 1:1 with ROM, but there's additional information after it. Actually wow, there's a lot of information after it. There is a hige section of data that is in memory that doesn't seem to be from the ROM. Editing it out is probably the easiest option, I could probably try another reverse search. 4. ..\MAP\DATA\ST04_MAP.MDT This wasn't as bad as the others. There's more data in memory following this file, but it's not as massive as some of the other holes. 5. ..\EMBIN\ST04_00A.EBD ..\EMBIN\ST04_00A.EBD matches the ROM 1:1 for its entire length. 6. ..\SCR\DATA\ST04.STG ..\SCR\DATA\ST04.STG doesn't match the ROM 1:1 and splits off from the ROM around 0x7c50 bytes. From my testing before all I found in the STG file was geometry types and uv values, so I'd be doubtful the vertices are stored in here now, but I might have to do more testing later. Other than this there don't seem to be any other files between this and /dev/null. So I'll do a quick side by side manual check to see if these are the full files. 0x000000000 ~ 0x000010000 = PSX Kernal 0x000010000 ~ = ROCK_NEO.BIN 0x0000b5df0 ~ = ..\sound\moto\se\vab\r_mine.vab 0x0000b6db0 ~ = ..\MESS\ST04_00.MSG 0x0000dba00 ~ 0x0000dc400 = Megaman Bones 0x0000dc400 ~ 0x000100000 = Megman Polygons
0x000100000 ~ 0x000111d98 = ..\PROGBIN\R3_ST04.BIN 0x000139f00 ~ 0x00013a000 = ..\SCR\DATA\ST04.HD2 0x00013a000 ~ 0x00013a238 = ..\SCR\DATA\ST04.HED 0x00015c000 ~ 0x00015c1f8 = ..\MAP\DATA\ST04_MAP.IDX 0x000164000 ~ 0x0001641c4 = ..\MAP\DATA\ST04_MAP.MDT 0x00016c000 ~ 0x00018E928 = ..\EMBIN\ST04_00A.EBD 0x000194000 ~ 0x0001aaad0 = ..\SCR\DATA\ST04.STG 0x0001f8000 ~ 0x0001fffff = pointers
|
|
kion
Arukoitan
@kion_dgl
Posts: 193
|
Post by kion on Feb 16, 2019 10:25:16 GMT -5
My last post is starting to get too long, so I'll start another one. So the results of my RAM editing efforts to locate the terrain geometry has so far been a failure. And in retrospect that doesn't come as a suprise. So far from what I've seen from the file formats, I have a few guesses as to how the game handles the terrain geometry. First the game uses MDT files to describe the width (x-axis) and height (z-axis) for how many tiles and a general definition for each tile type. The game then uses STG files to describe the types of terrain segment-geometry that define each tile, and provides a list of pallet / texture cordinates and uv values to style the terrain geometry. Two aspects that are notable to how the geometry is defined is: 1. The STG file uses scale to increase the size of the geometry-segments 2. The geometry-segments themselves are often very simplistic, often being a flat surface or angle So what I think this implies is that the format the vertices are stored in is likely bytes, to create small geometry that can then be scaled up and placed in an environment. And second is there is likely a static list of geometry located inside the game executables, if all of the types are shared between stages. This means my approach was stupid for a few reasons. The first of which is a lack of vareity. I was thinking that if the list of geometry was large enough, that by going through the game's data and commenting out sections every so-many bytes, that eventually I would be able to deform the stage geometry. The problem with this thinking is that the Apple Market probably doesn't use that-many geometry-segment types, and I could have very well edited a part of the geometry definition inside the game's binary and not have known it, because the part I edited wasn't shown in the area. So if I were going on chance to try and find the geometry definition location, then I should have probably created a stage to increase the odds of doing so. Another problem with this approach is that I'm still attempting to find the terrain geometry without extensive testing. So one easy test for this is to load different areas in the game, change the geometry tile and see if the same number indexes result in the same geometry for every stage. Then that would prvide more context on if the definition is really static and in the game's programming, or if I should change direction and sort else where. Though I will admit that the reason I haven't done this is because while I've systematically shifted through STG files enough times to be familiar with their content and functionality, I'm not familiar enough with their structure enough to be able to effectively manipulate them. So I have a couple of options of how to proceed from here. One test that sticks out is that the STG file is different in RAM than from ROM. I want to try copying the ROM data into the save state and load it to see what happens. Another test to try is to wrong-load a different STG file into the game and see what happens. Though I doubt either of these will be effective in helping to determine where the terrain is located in RAM. For that my plan is to use PCSXR to try and log where memory locations are being read from. If I can change a tile and log a difference in which address is read after, then it might provide a hint as to where the geometry is located. Edit: For test 1, there is no visible change when restoring the STG file from ROM. For test 2, here's the "wrong-warp" of ST05 imported into STG04. Okay moving on to the next test: Here the geometry-segment definition byte is at 0x000195c09. So we need to set up pcsxr to recognize when this byte is being read so we can start logging. Though I'm not exactly sure which address I need to be looking at. The STG file gets copied into memory at 0x000194000. The game says this file is copied into 0x80194000, and the byte is 0x1C09 from the start of the file. So that gives us 0x80195C09. So first things first, it seems like that for a test we should test values there and make sure we can read the values there. Edit: Logs for memory access with unchanged geometry (type)
Log 01 : pastebin.com/qiKP6iq3Log 02 : pastebin.com/WpaV1AJgLog 03 : pastebin.com/mgHA1EHzLog 04 : pastebin.com/yiJJaifSEdit: Logs for memory access with changed geometry (type) Edit: Okay, so I was finally able to produce the effect I was looking for: Basically the idea was to be able to edit out part of the terrain geometry. And the it was located in the STG file, which is where it makes most sense. After doing testing on the ROM and not finding the vertices in the file, I thought I had ruled it out. However after going through the process of dumping the memory and shifting through all the memory, it looks like it was there the whole time. Though the positive effect of this happy little accident is that now I've learned more ways to debug, and improve the speed of testing by editing the memory and reloading save states. So the next step is to start documenting the MDT and STG file types. There's also the IDX format, but I'll try to avoid documenting that if I can, and include it if there is some aspect of the MDT and STG files that requires it (which it probably does).
|
|
kion
Arukoitan
@kion_dgl
Posts: 193
|
Post by kion on Feb 18, 2019 1:00:44 GMT -5
MDT DocumentationThe MDT file type is a pretty small (conside) file. And there's not a lot going on inside of it. The first dword provides the number of rooms defined in the MDT file. Following that an array of room headers. Each header consists of two dwords. The first being a dword for the properties. The second being a pointer to where the room tiles are defined. Specifically the file used in the image above is ST04_MAP.MDT found in the ST04.BIN archive. ST04 refers to the Apple Market of which there are six rooms: 0. the main apple market 1. the junk shop 2. the tailer 3. high neck record shop 4. electronics shop 5. hipbone Though to be honest I'm not sure if the file is in this order beyond the Apple Market being the first entry. I should probably set up a wrong warp to test for completeness. So order aside, a quick mock up of what the room header descriptions are as pictured: So for each Room Header definition there are only two dwords. Since the second dword is a pointer to inside the file where the tile descriptions are, that effectively means there is only four bytes being used to define the general properties of each room. I'm not sure what the first two bytes mean, and are likely some kind of pallet definition. The third byte is likely the width or the number of tiles in the x direction, and fourth byte is likely the 'height' or the number of tiles in the z direction. I'll have to do some tests to confirm the orientation of the tiles, specifically in an area that allows the player to bring up the map by pressing the select button. My prediction is that the from the top perspective offered from the select screen, the numbering starts in the lower left hand corner, and increases row by row, in the following manner. Finally, returning to the first image with the hex view of the file. The tile definition is shown in yellow. And each tile definition is likely a type definition, followed by a height value for how high to place the tile. Which means there are a few things to differentiate in what's being defined by the tile. Where is the list that defines each type (probably in the STG), how the tile geometry is defined (slopes and stuff), and if there are any limitations on what can be places on each type of tile. QuestionsI thought of a few questions for MDT files that could help with documentation. 1. Are MDT files always placed in the same location inside memory? It seems worth looping through the files to see if they all have the same copy-to address. 2. Is there a top down list of what stage numbers are used for each area? I might make testing easier. 3. What is the function of the first two byte in the MDT Room Definition? Could be pallet or spawn point. Could try replacing with 0000 and see if there is any change. 4. What is the orientation for the width and height definitions? 5. What are the dimensions of each tile?
|
|
kion
Arukoitan
@kion_dgl
Posts: 193
|
Post by kion on Feb 19, 2019 21:29:32 GMT -5
STG NotesNot at the level of documentation yet. So I'll post my notes about the STG files as I go through them. Last night I think I managed to identify the header (it took a few tries of trying different struct lengths). You can see on the left hand side, the first short starts with -1, then 0, then 1 and continues to increase down header. Something interesting to note about this is that the numbers repeat. So the shorts will go from -1 to about 9, and then repeat going from -1 to about 9. The difference is the short that follows it seems to be some kind of flags for specifically what is being defined. So it could be that the first short is the tile id, and the second short is the flag for which part of the tile is being defined. Following that, there is 0x2c of data in the header for each entry to be broken down and tested. I numbered a few points of interest. 1. Not sure what this is, there seems to be a value (0x0201) followed by 0x016b, similar numbers seem to show up in a few of the header entries. 2. This seems to be the pointers for level of detail. Though I'm not exactly sure specifically what they define, but I did notice these when I did previous testing. 3. There's a pointer following the first two shorts in some of the entries. And this short seems to be to point towards the end of the file. Edit: So the weird thing about the flags is they seem to be important except when they're not. I noticed when the flag is 0xffff (-1) seems to be when the 0x0201 0x016b pattern appears, except when it doesn't. This pattern holds true for some of the other pointers. It almost looks like there are bitflags that define which properties are defined in the header, except there doesn't seem to be an exact rule for which properties are which bits, and in some cases the flags will be 0 and different properties are defined. Edit: Looking at more of the 'flags' with what's set in the header. The values seem to almost match up with what's being set. In most cases the 0x0d flag seems to have all of the values set, but then in another header, there are almost no values set. So my first guess would be that this was a short for setting bit flags to define something, but I don't think that's the case now, the second short seems to be setting something, and that something has similar flags when the corresponding properties are defined. Edit: There are few entries that have all of the space filled, and that makes it a little easier to see what's going on. So mostly we have a bunch of pointers (in red), and the good news is that they repeat. So one nice thing about pointers is that they point to something, so you can comment them out and see what that something is and not worry about the details. And then in blue, we have what I'll refer to as properties. There about three dwords with values that we don't know what they do, so that will require some testing, when they're commented out, the stage does this, and when this value is there, the stage does that. And lastly there's the header in green which has the tile id for the first short, and then something (probably flags?) for the second short. So that will need some testing too. Also we need to line up the STG file with the MDT file to see if the tile numbers match up. Row 1 column 1 | Callback Pointer 1 | Row 1 column 3 | Row 1 column 4 | Row 2 column 1 | Row 2 column 2 | High LOD | Medium LOD | Row 3 column 1 | Row 3 column 2 | Row 3 column 3 | Row 3 column 4 |
Edit: Okay, so more testing last night. And the interesting thing is that you can pretty much comment out everything in the header, and there are no visual changes. So the two remaining pointers are for high and medium LOD. I suspect the values after that are for lower LOD, but since the Apple Market is pretty small, these don't seem to take effect. Another interesting aspect to point out is that the other values seem to be related to interaction. So for the door panel, if you comment everything out besides the high and medium LOD, the door will still be there, but then the game will freeze when you try and interact with it. That seemed to be the case for all of the doors in the Apple Market, if you comment out the non-visual properties in the header, the object would be there, but freezes when you try to interact with it. Another thing that I found surprising what editing out the id and flags after the id had no effect. I actually went ahead and did a test where I removed everything except the high and medium LOD pointers, and commented everything else out, and aside from loading zones not working, the Apple Market seemed to work perfectly fine (enjoy purgatory Megaman). So right now the 'only' property I need to follow is the high LOD pointer. Though the implications of this kind of confuse me, as I already know what the format is for the next structure from previous testing. What's curious about this is that there are pointers to the tile definitions, and then pointers to the interaction definitions, so how does the game know where the vertices are if there doesn't seem to be any pointer to them? Edit: More notes from reading through the stage file. It looks like the last pointer comes well before the end of the file, so for the vertices, it could be that the offset is read from before the EOF. Otherwise there's a possibility that the pointer to the vertices is in the IDX file, so I should look and see what the range of pointers is for that. Edit: First did a quick test to see if the IDX pointed to inside the STG file and it does. Pointer found! 0x80194420 Pointer found! 0x801943c0 Pointer found! 0x80194360 Pointer found! 0x801948a0 Pointer found! 0x80194870 Pointer found! 0x801948a0 Pointer found! 0x80194720 Pointer found! 0x80194540 Pointer found! 0x80194450 Pointer found! 0x80194630 Pointer found! 0x80194780 Pointer found! 0x801945a0 Pointer found! 0x801944e0 Pointer found! 0x801946c0 Pointer found! 0x801947b0 Pointer found! 0x801945d0 Pointer found! 0x801944b0 Pointer found! 0x80194690 Pointer found! 0x80194750 Pointer found! 0x80194570 Pointer found! 0x80194480 Pointer found! 0x80194660 Pointer found! 0x801947e0 Pointer found! 0x80194600 Pointer found! 0x80194510 Pointer found! 0x801946f0 Pointer found! 0x80194840 Pointer found! 0x80194810 Pointer found! 0x80194840 Pointer found! 0x80194330 Pointer found! 0x80194390 Pointer found! 0x801943f0 Pointer found! 0x80194cc0 Pointer found! 0x80194cf0 Pointer found! 0x80194c60 Pointer found! 0x80194c90 Pointer found! 0x80194bd0 Pointer found! 0x80194c00 Pointer found! 0x80194c30 Pointer found! 0x80194d50 Pointer found! 0x80194d20 Second is I looked into the extra space at the end of the stage file, and that seems to be where the game stores collision. So editing it out lets you walk through walls. Edit: Well this is weird, the pointers from the IDX file only seem to point to the range of the header. '0x330', '0x360', '0x390', '0x3c0', '0x3f0', '0x420', '0x450', '0x480', '0x4b0', '0x4e0', '0x510', '0x540', '0x570', '0x5a0', '0x5d0', '0x600', '0x630', '0x660', '0x690', '0x6c0', '0x6f0', '0x720', '0x750', '0x780', '0x7b0', '0x7e0', '0x810', '0x840', '0x870', '0x8a0', '0xbd0', '0xc00', '0xc30', '0xc60', '0xc90', '0xcc0', '0xcf0', '0xd20', '0xd50' Thought the good news is that it comes in incrememnts of 0x30, so it looks like my struct definitions were correct. 0x19BB80 - Start of vertices 0x19fe00 - visual effect of vertices 0001a0070: e82030e0 7e7e7e00 e018e8f4 18e8e000
|
|
kion
Arukoitan
@kion_dgl
Posts: 193
|
Post by kion on Feb 23, 2019 11:34:51 GMT -5
More STG NotesSince my last post is getting pretty long, I'm going to start a new one to continue taking notes. The general structure of the STG file seems to be - Header - Terrtain definitions - UV values - Vertice Values - Callback functions - Collision Though there are a lot of questions that arrise from this. First of all the header, doesn't actually have the number of entries defined (as far as I can tell). The way I found the end of the header was to find the smallest pointer, and then divide by the size of the header struct (0x30). I would be surprised if this is the way the programmers made it. Second is the terrain definitions. Basically for each tile, this defines what's on top of the tile. And the way the game defines what's ontop of the tile is by defining a few basic geometry objects. This definition describes how many objects there to be defined, their type, and the image/pallet to be used. Third is the UV values for the actual terrain definitions. The structure is simple enough as it uses the same format as the rest of the game data. The PSX can access one texture up to 256x256 in size at any given time. So these are byte values to define which part of the texture to be used. The weird part is that I don't see any pointers to this section from anywhere, and I don't know how these match up with their respective definitions. The fourth is vertices. This is what I'm currently working on. And there are a few oddities already. First of all, it took a really long time to start seeing any visual change when editing these values out. Since we're in the Apple Market, it could be that the geometry for inside the shops comes before the Apple Market and that's why, but either way it was weird. What's weirder is that there doesn't seem to be any pointers to this area as well, and unlike the previous section, where you have the end of a previous section to use as a placeholder, it's just kind of shoved in there with nothing pointing to it. And also the terrain seems to have a lighting definition. Callback functions are what I define as places where you can interact with the stage. Specifically meaning when and where you can load other stages. My original guess as to how the game was doing this was through meta objects, that have a geometry, but no model and are placed in the stage. So similar to the way you can walk up to a trashcan or an NPC and interact with it, loading zones are like a non-visible trash can with a specific effect. But I turned out to be wrong, this could be similar to the way they manage it, but where they manage it is in the stage file, not the objects. And last is the collision. There seems to be a lot less collision data than geometry, and that's because the game doesn't have to check collision against everything you see. So for example, you can see through the glass on high neck records shop, and see geometry in there, but you only need to detect collision against the glass. Similarly you don't need to detect collision for the Apple Market banners on the ceiling. Side note is that the term for this is "collision hull" and it's basically a simplified version of the world geometry to cutdown on the number of checks that need to be done.
Edit: Okay I think it's safe to say I finally found the vertices. In the data, I found this split between what I think are the UV values and the vertices. Though again, I think it's incredibly weird that I can't find any pointers inside the file. So going through this section of the file, I did my normal thing, where I comment things out until they start dissappearing.
Which I think it's safe to say that parts of the environment are missing. So we know the data starts with a 0x04 offset, and then the pattern repeats every 0x10 bytes. Which gives us about four dword per quad. So the first step is after I found a quad I could test with, I started going through dword to figure out the vertices. And to my surprise: It looks like the first dword is used for the lighting setting. Which means we have three dword left for the vertices. So three dwords is 4 bytes, so we have twelve bytes to work with. Since we're using quads, that means we have four vertices, which means we have about three bytes for each vertice. And three bytes per vertices means that should be one byte per axis coordinate. e0 18 e8 f4 18 e8 e0 00 e8 f4 00 e8
And it turns out that's exactly what we have. Hopefully even someone who isn't familiar with hexidecimal should see the pattern. On the z-axis, all of the vertices share the same 0xe8 coord. And then one the x and y axis yout have the corners for e0 18, f4 18, e0 00, f400. Since we're dealing with quads (which are a stupid way to manage data), all of the vertices probably need to share an axis, as when I try to change these, the whole quad dissapeared. But that aside, we've finally managed where the primitives are stored, so now we need to work backworks and try and find how the game knows to draw so many quads and place them were. Though the format itself is pretty simple. Edit: So the next step is to try and traceback how these quads are drawn. So first of all we have the MDT file which defines the tile dimensions and the tile types. So we need to check the number of tiles with the number of header entries in the STG file to see if they match up. Next we have the terrain definitions for the type and scale. For the type, we can set the type and then find the offset. Change the type and find the offset, and then see if we can figure out how they game knows (or at least find an equivalent method) of where the offset should be for the quads, and then figure out how the game knows how many quads to draw. Edit: Made a tool to take an MDT file and draw it as a table to get a quick top-down view of a stage. The next thing to test is to try and see if the number of tiles matches up with the number of entries in the STG file header. Edit: I made a quick mdt table viewer, which I'm not 100% certain is completely correct. Specifically the normal stages look correct while the cutscenes have crazy values, but what I have managed to confirm from testing is the tiles for the Apple Market look like the following: Where 0x05 are the tiles for the north door and 0x1c are the tiles for the south door. Here's a screenshot of testing where I commented out the Junkstore to make sure it matched the orientation of my tile reader. I'm noticing a few things as I'm messing with the MDT file. 1. Where do the tiles get interpretted? Likely in the IDX file so now that I know where tiles are I can try editting out the IDX file to see if it makes changes to the corresponding tiles. 2. Is geometry repeated? When I assign a different tile type, the geometry on top of the previous tile dissappears. So I wonder what the condition is, tile position plus tile type maybe? 3. Elevation. The tiles defined in the MDT file rarely have elevation defined and they seem to work fine. However if the tile type is changed, the elevation needs to be defined for them to appear. Not sure what the difference is in these case.
|
|
kion
Arukoitan
@kion_dgl
Posts: 193
|
Post by kion on Mar 6, 2019 10:20:29 GMT -5
STG TestingI created a quick and dirty MDT viewer that takes an MDT file and produces a table with the defined number of cells and labels each one with the tile type. The Apple Market looks like the following: So the two tiles in front of the North Door are of tile type 5. Which makes it easy for testing. So I wondered where the game is defining the actual file type. So I thought that the tile numbers defined in the MDT file might match up with the IDX file. So I took another look at the IDX file. The first entry is three empty dwords, which indicates the struct is three dwords long. And it's pretty easy to see that there are two dwords followed by a pointer. So I counted out five structs, and commented out where I expected the fifth tile to be and the tiles dissapeared. So experiment confirmed, the MDT file uses the IDX file as a lookup table to define type. But what I found next surprised me. It turns out you can move the geometry in the IDX file. So the geometry can be moved left and right. And it can also be moved forward and back. Which goes against the model of how I thought the file type was laid out. Because I thought that the MDT defined the location of the tile, and then the tile was defined on top of each one. But that doesn't seem to be how it works. The IDX file defines where the geometry is, and then contains a pointer to the STG file that contains where the quads, vertices and uv's are defined. The MDT file seems to be used for two purposes, as a representation of the pause menu, and the game also seems to use the tile type for managing the interactions of the entities on top of the tile. But in general to draw the environment, it looks like you only need a list of the unique tile tpyes in the MDT file to look up in the IDX file. So a quick overview for the structure of the IDX file is the first value 0xf2 is the left/right position. The second value 0x0d is the forward/back position, 0x01 is the collision yes/no flag. And the last value is the pointer to the header location in the STG file for the geometry. Since the x and z location of each tile type (at least the stuff on top), that seems to imply that any given tile type only needs to be defined once for the geometry to be drawn. So to give an example. We can comment out one of these 0x05 tiles in the MDT file: And then when we run the game, the stage displays normally, the only difference is that walking where the tile would normally be, causes Megaman to fall through the floor. Which seems to indicate the game is actually drawing the same geometry several tiles per frame. So to test this I think we can try to see if the game draws the Apple Market twice. So first we comment out all of the surrounding tiles: Then we change one of the 0x05 tiles to 0x04. We copy the 0x05 IDX entry over the 0x04 entry. We move one to the right and one to the left: And there we go, two Apple Market signs. So yeah, it looks like we need a list of unique indexes from the MDT file, and then look them up in the IDX file. So the next step will be how to trace out the STG file and find the offset references for the uv's and vertices. Edit: This was unexpected. I remembered it's possible to change the height of tiles. So I tried to have two 5 tiles and change the height on one but not the other to see if it would cause the Apple Market banner to draw twice. Turns out no, the game only draws each type of geometry once. The tile height seems to be for an actual tile. So it looks like when the tile height is 0, the game doesn't actually draw the tile and only uses it fo meta data, where as if the tile has height, the tiles themselves are used to make the stage. Edit: Managed to get part of the junk shop to load into the Apple Market. So the way the game seems to handle stages is with tiles and with stage geometry. A tile represents a square area on the floor, and it's what you see when you bring up the start screen in a dungeon. Tiles have two properties a tile type, and a tile height. In a lot of cases, the tile height will actually be zero, meaning the tile itself isn't actually visible to the player. For each tile type, there is a IDX definition. The IDX definition will define an x and z location where a specific stage geometry is defined. For for every unique tile type, a geometry is (potentially) defined. So to test this I replaced two of the null tiles outside the normal map with the tile types 0x22 and 0x23 which are only used in the junkshop. I then went to the IDX definition for these tile types and moved the z and x position to infront of the north gate. So I think I have a model for how the game manages stages. The next step is going to be breaking down the STG file to see where to find the right offsets.
|
|
kion
Arukoitan
@kion_dgl
Posts: 193
|
Post by kion on Mar 8, 2019 23:16:33 GMT -5
STG AnalysisSo I think I have a decent idea of how the stage geometry is handled by the game. The next step is to get a better understanding of how the STG file is structured. The image above gives the general layout of the file structure. At the top is the header, this is where the IDX file points to for a given tile type. The header contains the tile type id (this is ignored by the game), pointers to callbacks, and pointers to the quad definitions for the geometry for each level of detail. Going in order, the quad definitions first define the number of quads that make up part of the geometry. It also defines the image and texture pallets to use for each quad. It then defines the scale and type for each kind of geometry. The scary part about this definition is that it only defines the geometry type with an integer, and so far I haven't found anything that gives a hint as to where the game goes to look up the offset for both the uv values and vertices for each quad. Next is the uv values, this one is easy. Each value is a two byte value for the u and v values of a given texture respectively. The issue here is where in this section needs to be referenced. Next is the vertices. This one is also easy. Each entry has a lighting setting to adjust the lighting on the quad being drawn. After this each vertice is a three byte value for x, y and z respectively. Since there's a limit with how large the quads can be defined with bytes, this is why the quad definition includes scale. Next is the callbacks. Or maybe "events" would be a better term. This is mostly used for loading zones, for entering a different room, or when a text box appears saying that something is closed or can't be entered is handled here. Next is collision. I don't know is the collision vertices are actually defined here or not, but commenting this out will allow you to walk through walls, but the stage geometry is still there. IDX EntrySince we know that tile number 5 is right in front of the north gate, it makes it easy to test with. The entry for tile 5 in the IDX is at offset 0x00015c03c. 00f20000 000d0100 70481980 As stated in previous posts, 0xf2 is the east/west position, 0x0d is the north/south position. And 0x01 is whether the geometry defined in the STG file has collision or not. One interesting thing to mention is that the positions are actually devided into sub-tiles. So even numbers will be aligned along the edges of tiles, while odd numbers will align the geometry with respect to the center of a tile. Though I can comeback and try to fine tune and test positioning once I actually have some geometry to test with. What were's interested right now is the pointer 0x80194870 which points to the header of the STG file. STG HeaderAnd now we find our way into the STG header. I'm not sure if it's a good idea to start with tile type 5 as we end up pretty far down into the file. But I guess if we need to we can always comment out other tiles and/or change the pointer at 0x00015c048 to what ever we need for testing (any index really). So we might as well record what's here. This is probably getting ahead of myself, but one thing that I'm worried about is that if there are no pointers for the uv or vertices, it could be because the lists are 'stacked' so they come one after the other, and the game calculates where the value needed to read the next values from based on all of the values declared before. That would be a terrible way to do things, but based on what I've seen so far I wouldn't rule that out completely Though in general I hope there's a simple intuitive approach to how they programmed this. (start of STG file is at 0x000194000) 000194870: 00020c00 a49e1a80 00000000 00000000 ................ 000194880: 00000000 00000000 c4611980 e0611980 .........a...a.. 000194890: e0611980 c4611980 e0611980 e0611980 .a...a...a...a..
So we can start breaking down what's going on by commenting out all of the things. 00020c00 - Commenting out does nothing. I think this is a id or flag value managed by the programmers. No change when commenting out, so probably not something I need to be concerned about. a49e1a80 - Commenting this out caused two effects, one to fall through the floor, and second for the game to freeze when pressing circle to open it. So this seems like the collision/callback offset. c4611980 - High LOD e0611980 - Medium LOD e0611980 - No change (Low LOD?) c4611980 - No change e0611980 - No change e0611980 - No change One thing that's worth metionins is the game repeats the values of c4611980 e0611980 e0611980. So in this case there's no change when moving away from the camera. So I think it's mostly that Apple Market isn't big enough to have a low LOD model. These three values are then repeated again, and the effect cannot really be observed. Though in general I can narrow down that the pointers I'm interested in are the ones are 0x18, 0x1c and 0x20 positions in the struct. Quad DefinitionsSo the high LOD model is probably going to be the easiest to test with. So we head over to c4611980, which is 0x801961c4 in little endian, which is 0x1961c4 in actual memory. 0001961c0: 02820100 1c00807f 1d00887f TC.............. 0001961d0: 94951940 64431a15 3c961941 b4441a0c ...@dC..<..A.D..
Okay, there's actually quite a bit to break down here. 02820100 - This is probably better written as 02020000. The first byte is the number of textures to define. The lower four bits of the second byte is the number of geometries defined. 1c00807f - Image Pallet definition 1, the positions of the framebuffer need to be masked and bit shifted. 1d00887f - Image Pallet definition 2, the positions of the framebuffer need to be masked and bit shifted. 94951940 64431a15 - The apple market sign above the floor 3c961941 b4441a0c - The tiles in front of the north gate So let's focus on the sign above the floor we'll start with 94951940 94 - changing this messes up uv 95 - changing this messes up uv 19 - changing this messes up uv 40 - Upper 4 bits scale, lower four bits texture id So aside from the last byte, everything seems to be uv related. Next we'll move onto 64431a15 64 - changing this causes the geometry to become distorted 43 - changing this changes the geometry type 1a - changing this causes the geometry to become distorted 15 - decreasing this number causes quads to disappear from the geometry Okay, so this is where I run into an issue. We have 8 bytes for each geometry type definition. Two of these bytes are pretty straight forward, specifically scale/texture and the number of quads. Five of these cause the game to act in very unpredictive ways. Though for the moment, I can ignore them, and later I can probably build a table of acceptable values to see if the game uses specific values in certain cases and try to look for patterns there. Right now the weirdest byte is the geometry type byte. There are no pointers up to this point and none I can find in the file, so how does the game know where to look in the list of vertices to go actually draw the geometry? Since I can access the source code of pcsxr, my plan was to log when the game reads the memory for the apple market sign and then record the values of memory the game reads after that. I can then change the value of the geometry to zero, and then see if I can log the address. I can then search the memory to see if the zero location is stored somewhere. I can then change the type to 1 and check the memory, and then 2 to see if there is any kind of pattern, or set number of bytes the game jumps for each subsequent geometry type. 0x43 : pastebin.com/raw/iP2avR9U - 0x001a4364 0x00 : pastebin.com/raw/JCpwhaF8 - 0x001a0064 0x01 : pastebin.com/raw/17kvCqtD - 0x001a0164 0x02 : pastebin.com/raw/r1n9h5iD - 0x001a0264 0x03 : pastebin.com/raw/KrvwDFzb - 0x001a0364 I'm noticing a trend... Though how the hell does the game know to jump to 0x001a0064? Edit: Okay, the solution ended up being a lot easier than I might have expected. The simple answer is "little endian". So the default value is 64431a15, and the 0x15 on the end is the number of quads to draw. Which means that 64431a jumps to 0x001a4364. So the address is actually written directly because 64431a is actually 0x001a4364 when written normally. So the address the game jumps to is actually written directly in the file just with three bytes (which is weird for a pointer), and it took me a while to realize what I was looking at.
|
|