Megaman Legends 2 Research Thread ( PSX)

[kion]

I've been wanting to take another look at Megaman Legends 2, and my schedule finally opened up a little bit to start to poke at the files. With the PSX version of the game, there were two issues with the textures. One is that they we compressed. And second was these was no easy way to try and match up textures with meshes as everything was an index in the framebuffer. And I think what makes that issue worse is that with the limited framebuffer means that the game has to load in more textures which cause the texture indexes to overlap. So I think it's worth taking a look into the PSP version to see if there's any advantages to working with this version of the game. Hopefully we'll have normal TIM or GIM files to work with, but we'll have to dig in to find out.

So we can start with some simple observations for the file layout. In the Top level folder which simply contains the file "umd_data.bin" and the folder "psp_game".

When we go into the "psp_game" folder we get a few png images for PSP menu. We also have two folders "sysdir" and "usrdir". To save ourselves sometime the "sysdir" seems to contain the eboot.bin file. So I'm guessing these are the elf executable for the PSP to boot from to start the game. 

Getting into the "usrdir" we see a few familiar looking folders. The "common" folder contains assets that are used through out the game like the meshes for Megaman, the menu sprites and the gameover screen.The "dat" folder contains all of the information for each of the scenes. Specifically this is going to be the stage and actor data for each stage. The "e1" folder is for "Episode 1", which doesn't seem to be the original game, it seems to be 4 mini-games of content that I guess is supposed to take place between Megaman Legends 1 and Megaman Legends 2. Assuming the "e1" folder uses the same file formats as the other folders we can come back to this.

And the last two folders are "kmodule" and "module", these seem to include a bunch of ".prx" files, which I guess are probably like ".dll" files in that they're probably linked libraries. So nothing that we need to interact with directly. So that means we probably want to start by focusing our attention on "psp_game/usrdir/common" and "psp_game/usrdir/dat".

[kion]

Quick video for poking around the files. In terms of the assets that are in this game we have:

- player mesh

- player animations

- title and menu sprites

- entity (enemy, objects and effects) meshes

- stage tiles

- stage maps

I guess I'll go in order and start digging through the dat folder to see what can be found.

[kion]

More Notes. It's been a while since I've looked into the files so I'll take my time and make notes as I go. We'll start with the usrdata/common folder. Most of the files in this folder start with pl00 and are likely the meshes for the different parts of Megaman and special weapons. Which means the other files in this directory are:

1) demo.bin
2) game.bin
3) init.bin
4) logo.bin
5) select.bin
6) staff.bin
7) submap.bin
8) subscn.bin
9) title.bin

To get started we can look into these files, make predictions on where they are used in the game. And look for similarities and differences into where and how they're used.

Edit:

It's been a day or two. But this probably isn't enough information to create it's own post. Right now we basically have 9 files to get started with. Let's try to break down what we can figure out with the headers in terms of hints to start getting familiar with all of the files in general. The first and easiest place to start with seems to be logo.bin as it contains the logo for the title screen. It looks like logo.bin contains two TIM files.

I drew two lines on the image indicating where the break probably is. But in all likely hood, one image is probably the "Rockman Dash 2" image. And the other image is probably the subtext for "エピソード2 大いなる遺産". Let's go ahead and take a look at the headers for each of these files. Here's the first one:

And here's the second one:

We can go ahead and make a few observations. First the file format looks exactly like the PSX version. And that probably means that the textures are compressed like they were in the PSX version. I might have to track down a Japanese copy of Rockman Dash 2 for the PSX so I can look at the files for comparison. It would really suck if the files were the same and compressed, but if that's the case then we still have several options, such as using save states, or comparing against the decompressed pc version textures. One issue I had before when trying to compress the files was that I was trying to decompress to the in-order version of the PC version, but I think that the PSX/PSP might change the order of the image to make it faster to copy into the frame buffer. So I might try to use some of the non-compressed PSX TIM files to start out with. It looks like Legendss posted about it in the psx thread. Next, stating the obvious we can see that 03 is going to be the TIM type. And then we can start to break down what some of these other bytes are by trying to compare them. And we can also change the bytes in the ROM, load the game and see what the changes are.

For now I'll go in order to try and make predictions about what each dword does.

0x03 - This looks like the TIM enum

0x6620 / 0x5020 - This looks like it could potentially be the size of the decompressed image + palette

0x04 / 0x05 - This could be a texture id (ha!) or potentially a pixel type to specify the color format like ARGB1555 or RGB565

0x01f0 / 0x01f1 - This looks like framebuffer coords

0x10 / 0x01 (both) - This looks like the color count and number of palettes

0x80 - width(?) 128 seems like of small. This value will probably be multiplied by 2 or 4 depending on the bits per pixel

0x20 / 0x28 - no idea

0x88 / 0xa0 - no idea

0x60 , 0x40 - looks like the height. kind of small but we'll see.

To take a look back at the psx textures, the information in the header was Palette Coord X, Palette Coord Y, Color Count, Palette Count, Image Coord X, Image Coord Y, Image Width, Image Height. So basically we should probably try to match these up.

[kion]

My older research thread is getting kind of long. And rather than adding onto that, or creating a new thread. I'll go ahead and continue from here. I think I might have to retrace some ground with the MML2 files as it's been a while since I've looked at them. And I always enjoy finding new approaches to try and simplify and break down problems. Also admittedly reverse engineering is rarely a linear experience, so while it would be nice to be able to create tools in a clear, concise order, it's only after the fact that things come together for a complete picture. So until then it's just better to kind of keep ripping things out with what ever approach is available. 

With respect to Megaman Legends 2, the approach that I wanted to test next was to unpack all of the assets in the game. Basically from what I can tell, it looks like assets were individual models. And then those get put into a table for which model gets used in what scene, with which textures. The devs then created a script or something that would create an archive for each scene and automatically assign coordinates for where textures exist in the frame buffer. The result is that we're kind of left with the mess of trying to put those assets back into their original forms. Previously in MML1, I wasn't able to do this and ended up going with more brute force save state viewer approach to simply go around the problem. 

With MML2, I think i'll try that approach again. And i had a few simple ideas for how that could be possible. Basically anything that's inside the BIN files, we can slice up and place into their own files. We can then try and break down what it where by trying to track the id's, or do other things like render TIM textures as a black and white .png file. And then we could make a tool that would allow for these to be grouped together to try and recover the original files. The first thing I wanted to start with was textures, something i thought I had working, but looking back and trying that code, it looks like that works in some circumstances and completely breaks in others. So we'll start by trying to break down texture decompression, when it works and what to do with it.

Since TIM files are everywhere, we'll go ahead and start with DAT/ST1ET.BIN, and it looks like there is a TIM file at 0xa000. It has a decompressed size of 0x8020, which should be an image with a 16 color palette on the front. The bit field size is 0x510. The bit field starts at 0xa030. The pay load starts at 0xa540, and the payload ends around 0xf600. With the next file starting in the archive starting at 0xf800. The file size is a lot smaller than 0x8020, so we can probably assume this is compressed and not something else. 

So the first thing we want to start with is the bit field. Right now I'm not exactly sure how to read the bit field. Sure, it's one bit at a time, but do we read one byte and go low to high, or high to low. Or read two bytes, or read four bytes? Depending on the order could create a different pattern of bits in the bitfield, which might cause errors when trying to decompress. 

Okay, and I think i got it working. I think it was a bug I missed when porting over the code from gitlab.com/megamanlegends/megamanlegends.gitlab.io/-/blob/master/public/utils/mml2_psx_tex/js/main.js. the nodejs version is here: pastebin.com/fS4XTkwz. I think now that I have the decompression working, I'll try to render the framebuffer as separate png images. 

[kion]

A little bit of progress. I went ahead and wrote a script that would go through the game and convert "all" of the TIM files into PNG files, and then put them in folders grouped by Stage Id. The current version of the script is here: pastebin.com/zMHg7fMa (needs nodejs and pngjs).

Hmm, that actually looks cut off. So it might be wrong. There still could be a few issues that I might have to look into. For starters it looks like the decompression didn't crash for every TIM in the game, so I'll think I'll call that a plus for right now. The game seems to have a quirk, where it will declare a palette size of 64 or 128 and a palette count of 1. What this actually seems to mean is a palette size of 16, and a palette count of 4 or 8 respectively.

What I think these values might mean is that rather than having alternative palettes that are swapped out to create different textures, these might be used in cases to show light exposure where the same texture just gets lighter and lighter, or darker and darker. So I might try and filter these out. The other thing I should double check against is file size. The game has a few fixed width and height values. And then palettes are indexed as either one or two pixels per byte. So I can do a quick calculation to make sure the bits per pixel matches up with what's expected from the number of bytes the game has listed for the unpacked size of the image file.

Another thing to check against is the extra palettes. Sometimes palettes can be declared on their own, so to represent that, I figured I would make a square image with the 16 colors in a 4x4 grid to visually show it exists. That way we can have  a general idea of where these extra floating palettes are to keep in mind. And hopefully that makes it easier to track down when we need to reference them later.

Also I'm slicing out the ebd files. I was thinking of further slicing those down into individual files. But I think I'll leave them grouped for now. I should probably try to run a little more logistics to read through all of the EBD files, try to make some of the id's and look for completely duplicate files. In the case where the ID is the same, and the bones and mesh are the same, and the animations or gear is different, then maybe I can try and group those into a single asset that merges the differences from the two scenes. That way we can have one list of unique models.

Also I haven't looked at the map files at all for MML2 yet, but maybe I could start by trying to look at and identify if they're still using the IDX, MDT, and STG formats from MML1 and slice those out into their own files to make them easier to break down and work with.

[kion]

I probably shouldn't treat pastebin as a poor-mans Git. But whatever, current script is here: pastebin.com/LttRtiEe

I added the functionality of exporting stand-alone palettes. Which looks something like this:

The idea is generally to get an idea of where these are declared and have an idea of what color it is when looking. In terms of implementation what i'll probably do is generate a database-file thing that keep track of where everything is stored, and if a palette isn't found, the program will have to go looking for it. Not sure what the best approach for that would be, right now I'm messing around with the console. I guess we can mess around with the files, and mess around with testing before trying to optimize anything too soon.

One optimization that I am tempted to attempt to play around with is grouping multiple textures into one page image. The PSX breaks down the framebuffer into multiple 256x256 pages. So a single page with have a 256x128 texture on top and then two 64x64 images under that. But we don't need to do this too soon as we're not mapping to any models yet. So I might go ahead and start slicing out the map tile files. Or move over to the common folder to start converting some of the PBD models or something.

[kion]

Okay, so next i guess i should start looking at some of the map stage files. I'm not sure if it would be a better idea to focus on one thing and do it well, but right now I think it's kind of better to do everything half-ass and just kind of get familiarity with the different files as opposed to trying to make anything too polished right away. So I'm going to be looking at ST08, which I think is the tutorial area. But the file structure should be consistent for all of the areas.

So let's go through the file and make a note of the headers and enum types.

	0x00	0x04	0x08	0x0c
0x000000	0x0D	0x0cc24	0x11
0x000010	0x07a4
0x000020

So first file looks like it has a enum type 0x0D.

	0x00	0x04	0x08	0x0c
0x008800	0x0C	0xefd0	0x16
0x008810	0x09e8
0x008820

Second file has a enum type of 0x0c. Not much hints from glancing at the file content.

	0x00	0x04	0x08	0x0c
0x013800	0x12	0x168d	0x03
0x013810
0x013820

Third file has an enum type of 0x12.

Hmmmm, and that's it. Three files. So a couple of things to note is that the header doesn't include where these will be copied into memory. So i'm guessing it's a fixed location depending on the enum that the game interprets. Second is the the content of the file doesn't give a lot of clues and looks pretty scrambled. It can kind of be a trap to say, "hurr, durr, I don't know what's going on, so it's probably compressed". But in this case, since the textures are compressed, it wouldn't surprise me if the devs decided to abuse their compression and simply use it where ever they felt like. Though I'm genuinely curious if there are any advantages with using compression. Like it cuts down on reads, and the compression is light weight enough, so the information being streamed into memory ultimately cuts down on load times?

Right now there aren't a lot of hints for what these files do on their own. So it's probably a good idea to make a save state, separate the memory, and then try comparing these files from the disk versus what's in memory to see if the data get's loaded in as-is, or if something happens. Also I can always try commenting out a lot of information in the ROM file, and then load the level and then try to confirm that errors or something happens to make sure i'm in the correct area. Though having all of the textures exported for all of the areas definitely helps with respect to being able to tell what's where in terms of navigating the game's scene numbers.

Edit:

It's nice to have a little bit of confirmation that i'm not completely retarded. It looks like the map files actually are compressed. Left shows what the file looks like on disk, the right side is the decompressed version.  I think this looks like the IDX format from MML1, I might have to go back and double check to see how similar they are.

Also I'll keep using pastebin as a lazy git. Made a few small improvements for filtering out false positive TIM/PAL hits. pastebin.com/hqzWec4A

[kion]

I guess while I'm looking at maps, I might try to take some time to do some testing to see what I can find about the format. My memory is kind of fuzzy on the Megaman Legends map formats, so I'll try to use the extensions from that game. Though I'm not sure yet of how similar or different they will be. On one hand it might be nice if they're the same as that would allow me to potentially make use of the tools that I have. But on the other hand, I wasn't able to get a complete picture of exactly how the maps were structured. So a different format might give me a chance to take another look at the content.



To start off, I'll start with what I'm calling the STG file, which has the magic number of 0x0C. It stuck out, since it seems pretty similar to the ebd file, so we'll look at it. At the top is a count, followed by a list of models / assets? Each entry has a length of 0x10 bytes. The first four bytes looks like a flag. At first I thought it might be an offset, but the later entries go waaay passed the end of the file. So probably flags. After that is a pointer to probably a model or something. And then after that it looks like there are two more pointers that might be for something like collision or trigger events.

I did a quick search in the game's memory and it looks like this file is at offset 0x013c800 for the opening stage. So the next step will be start messing with the save state to see what some of these offsets do.

Okay, so quick edit. I tried editing this file in memory and as far as I can tell, there was no impact on the stage being drawn. I'll try searching for the other two file types and see if that brings up anything.

[kion]

Quick note. Since the file I changed in memory didn't seem to change the stage geometry at all, I checked the other two files that were in ST08.BIN. I couldn't find them. The main reason being that the pointers in the files were updated from relative to absolute. So I should probably try searching for parts of the file that aren't pointers to see if that matches anything.

I took a little bit of time to mess around with memory last night. So I'll make a note of some of the blocks of interest:

0x07d130
0x0e7000 (looks like ebd)

0x000110800 - Megaman Disappears
0x0001125d0 - Megaman's Left Arm (mesh)
0x000123000 - Megaman Animations?
0x000134800 - No Change
0x000178800 -

Coming across Megaman's animations was a nice find. I'll try looking for the other two files, and then probably switch over to the COMMON folder to start looking at that.

-- Matched Notes ---


0x00013c800 - STG(?) ENUM 0X0C
0x000178800 - IDX(?)  ENUM 0x0D


Okay, and manage to deform the mesh with the contents of memory in 0x000178800, or the IDX(0x0D) file as I'm current calling it.

[kion]

I guess the good news is that we were able to quickly track down where the game stores the stage mesh. Now we should probably take the opportunity to break it down, and see what's going on. I think to avoid confusion with the MML1 stage types, I'll go ahead and call the MML2 file a ".MAP" type to differentiate. And there are a few things I should do.

1. I should export the .MAP file to an Excel spreadsheet so I can start making notes
2. I should export the in-memory version of the .MAP file. Any differences should give me a list of pointers in the file

So let's get going.

[kion]

I exported the ST08.MAP to a spreadsheet to be able to view it. The Google Drive link is here.
drive.google.com/file/d/17ebyvh87pS4ff2PZhqeY8zrugNSSrsAO/view?usp=sharing

The format looks a little familiar, but it's been a while since I've looked at the MML1 stage formats. So I'll have to pull those up to be able to compare the two. For now I'll try to block out what I can from the MML2 stage format.

[kion]

Didn't get around to comparing the MML2 maps to the MML1 formats. But I think that's okay because the MML2 files don't seem to have pointers as far as I can tell. Which is weird, because that's something they should definitely have. The possible options are they have some kind of byte offset. So the header lists a number of shorts, and then advances in the file by the length of the short listed. There's also a possibililty there is a table somewhere in memory. And the last option is there is some from of in-memory obfuscation where a long needs to be read, and then shifed over to get the location needed in memory. Though this might also be done for compression and hopefully not to be intentionally annoying.

Which means right now there are a few things i can do.
1. scan the psx memory for pointers to the stage file
2. compile my on version of pcsxr to log memory addresses and values read
3. split the file into sections, comment them out and view affects in game (easiest)
4. avoid getting too focused on stages and move on to the COMMON folder

I think that last one might not to too bad an idea. I can do some testing by commenting out parts of the file in memory to try and get an idea of where somethings are. And then I should probably move onto the COMMON folder to start getting familiar with that.

[kion]

*Looks at previous post date. *Dies inside.

I think one of the challenges with Megaman Legends is getting around the .BIN format. I'm not sure if it would be worth the investment to try unpacking the files to reduce the number of files that are required to inspect. There are a lot of assets to work with, but I think the easiest place to start is with the enemies characters, try to make a unique list, get names, and match textures. 

In terms of strategy I'll think I'll take the following approach. 

1. Ignore the possibility of texture swaps
2. Ignore the use of level of detail in favor of only high level of detail models
3. Focus on one data type at time

Right now my focus is trying to get all of the character / enemy entities that are in a stage. This means that all of the files with the flag type of 0x20 from the ENB file. Which gives us the list of the following 76 models. 


 '0x000020', '0x000120', '0x000320', '0x000720', '0x001520',
 '0x001620', '0x001920', '0x001a20', '0x001b20', '0x001c20',
 '0x001d20', '0x001e20', '0x001f20', '0x002020', '0x002120',
 '0x002220', '0x002320', '0x002520', '0x002620', '0x002720',
 '0x002820', '0x002a20', '0x002d20', '0x002e20', '0x002f20',
 '0x003020', '0x003120', '0x003c20', '0x003f20', '0x004220',
 '0x004e20', '0x005420', '0x005520', '0x005a20', '0x005e20',
 '0x006020', '0x006220', '0x006920', '0x006a20', '0x006b20',
 '0x006f20', '0x010420', '0x010520', '0x010820', '0x011520',
 '0x012d20', '0x013120', '0x016020', '0x022d20', '0x026520',
 '0x032d20', '0x036520', '0x046520', '0x056520', '0x060020',
 '0x066520', '0x076520', '0x086520', '0x096520', '0x0a6520',
 '0x0b6520', '0x0c6520', '0x0d6520', '0x0e6520', '0x0f6520',
 '0x106520', '0x1d0020', '0x1e0020', '0x270020', '0x2e0020',
 '0x330020', '0x340020', '0x3c0020', '0x3d0020', '0x3e0020',
 '0x440020'


The next step is that we can use my half-baked tool to try and get as much visibility as possible for which characters these are. We can create a file which shows which scene each one of these id's is used in. pastebin.com/i8VXp7FN

[kion]

A few observations after looking through the data yesterday. It looks like the game has a separate id for some of the different variations. For example the green Frongel has an id of 0x003120 while the red version has an id of 0x013120. The pattern didn't seem to hold true for Kuruguru, Mirumiji, or Horokko. If needed we can track these down and add separate entries manually to account for these.

Next is a hypothesis that hopefully makes my life easier. The thing i've been referring to as an entity id is broken down into three parts. The last byte is the handler. This seems to be a flag that tells the game engine how to process the entity. For 0x20 this seems to make it an enemy for processing movement (includes NPC's). For 0x11, this is for stationary objects. And 0x60/0x61 is for things like bullets and explosions. The second byte is the actual type, which defines which model it is. And the third byte is the variant, like green versus red Frongel. 


"0x11": [
 "0x02",
 "0x04",
 "0x05",
 "0x06",
 "0x0f"
 ],
"0x20": [
 "0x00",// NPC
 "0x01",// Data the Monkey
 "0x03",// Jakko (Reaverbot)
 "0x04",// Golbesh (Revaerbot)
 "0x05",// Horokko (Reaverbot)
 "0x07",// Giwan (Reaverbot)
 "0x08",// Fingerii (Reaverbot)
 "0x15",// Mimic (Reaverbot)
 "0x16",// Verner Von Bluecher
 "0x19",// Kuruguru (Reaverbot)
 "0x1a",// Barrell Caskett
 "0x1b",// Inmii (Reaverbot)
 "0x1c",// Megaman in Apron
 "0x1d",// Sulphur Bottom Guard
 "0x1e",// Glyde Loath in Suit
 "0x1f",// Roll in Pajamas
 "0x20",// Ruins Trap
 "0x21",// Teisel Bonne
 "0x22",// Tarble (Reaverbot)
 "0x23",// Reporter
 "0x25",// Camera Woman
 "0x26",// Puurian (Reaverbot)
 "0x27",// Sulphur Bottom
 "0x28",// Tron Bonne
 "0x2a",// Mirumijee (Reaverbot)
 "0x2d",// Birdbot
 "0x2e",// Matilda(?)
 "0x2f",// Matilda
 "0x30",// Flutter
 "0x31",// Frongel
 "0x3c",// Klaymoor
 "0x3f",// Gatz
 "0x42",// KTV Ship
 "0x4e",// Matilda Hand
 "0x54",// Glyde's Ship
 "0x55",// Joe
 "0x5a",// Geetz
 "0x5e",// Parabola Gun
 "0x60",// Roll
 "0x62",// Cannon
 "0x65",// Special Weapon
 "0x69",// Glyde Loathe in Armor
 "0x6a",// Ship Wheel
 "0x6b",// Sitting Figure
 "0x6f" // Refractor Shard
 ],
"0x60": [
 "0x00",
 "0x01",
 "0x05",
 "0x13",
 "0x16",
 "0x17",
 "0x18",
 "0x19",
 "0x1b",
 "0x22",
 "0x23",
 "0x24",
 "0x29",
 "0x2d",
 "0x2f",
 "0x34",
 "0x38",
 "0x3a",
 "0x3d",
 "0x3f",
 "0x41",
 "0x4d",
 "0x50"
 ],
"0x61": [
 "0x01",
 "0x02",
 "0x03"
 ]

In the case of NPC's, they have a type of 0x00. And then they use the variant to separate which model should be loaded. I should note that this list is skipping over the bosses, as unlike Megaman Legends 1, bosses have their own file as opposed to being in the list of enemies per scene. 

Right now, not really sure about the 0x11, 0x60 and 0x61 flags. That seems like something I would need to look into separately. For now I think it makes sense to create a list of characters, give them an id, and then provide pointers for where they can be found in the game's data, along with which textures they should be loading in. That also allows me to simplify my data model where I only need to store the game files in the tool, and then everything is read and converted. 

[kion]

Now that we've managed to isolate a list of characters, we know each one of the characters'id's, we can start asking questions about how the data is managed.

1. Which archives contain the data?

We can use Data the Monkey as an example, which has an id of 0x120 and can be found in ST1705.BIN, ST1900.BIN, and ST3C00.BIN. When looking at this model using the Sulphur Bottom tool, we can see that the tool is able to resolve the texture for  ST1900.BIN, and ST3C00.BIN, but not ST1705.BIN. 

2. Is the data consistent between each archive?

I'd have to write a program to check for this. But i think the question is, are the differences in which the same id is used in different archives meaningfully different? Probably not. They might be in different positions in the file, so the pointers will be different, but that doesn't mean anything. The texture references and uv's might be different, but the texture itself is going to be the same. So we can probably says that the geometry, bones and texture will be the same for every time the id is used.

3. Which textures are being referenced?

This is to make a note that we're probably going to have to load in the textures manually. We're going to need a way to record the textures that Sulphur Bottom is able to load correctly, and then use them for that id. The other thing is for characters where I was not able to find a unique id for a variation, we're going to have to specifically provide an entry for that variation (such as the green mirumiji). 

4. Are there any unique animations?


This is going to be the main point of interest in this case. I think it's safe to assume that geometry and textures will act the same for the same id, even when used in different archives. We're probably going to need to write some code that will loop through the other archives where the same id is used to see if the animations are the same or not.