kion
Arukoitan
@kion_dgl
Posts: 193
|
Post by kion on Jul 3, 2019 8:47:29 GMT -5
So after doing the tracing I was able to make a tiny bit of progress in the direction of figuring out how the game assigns terrain vs. flat tiles. After commenting out the IDX list, it looks like every non-flat tile is a terrain tile, of which the offset to the STG geometry definition is defined in the IDX table. Up until now I thought the format for the MDT table was [ tile type 8 bits ] [ tile height 8 bits ]
But from my trace that doesn't seem to be the case. The game reads 2 bytes for each tile type. And the game goes directly from reading the MDT table to the IDX table, which means it's not doing any look ups inbetween. After checking the offset in the IDX file for Roll's room tiles, it looks like they are are indices 0x068b, 0x068c, 0x068d, and 0x068e respectively. So the whole 16 bit value for the tile is the index to the IDX table.
Which then raises the question of how does the game manage flat tiles, if the height can be adjusted. Would that indicate that the game is looking for a specific bit to be set. Or the start of Stage IDX is 0x0015c000 followed by Stage MDT at 0x00164000. Which means the max possible length of the IDX file is 0x8000 and divide that by 12 (the length of an IDX entry) and we get that 0x2730 is the largest theoretical IDX entry. So everything over could be flat ground.
Edit:
I think there's a simple answer. And I think that if the highest bit, 0x8000 is set, the game interprets the tile as flat, otherwise it reads it as an IDX index.
|
|
kion
Arukoitan
@kion_dgl
Posts: 193
|
Post by kion on Aug 7, 2019 7:03:19 GMT -5
Okay so things have randomly started coming together for the stages. Stage terrain tiles are a 16 bit index to the IDX file. Which means the next step is to figure out how to pair the flat tiles to the terrain tiles. And I think the easiest way is to just start putting down numbers and looking for patterns. So room 0 is on the right. Room 1 is around the Flutter. And room 2 is towards the Apple Market. Okay, so first things first, we have the MDT header entry. Room 0 : Object { x_dis: 45, z_dis: 12, x_len: 40, z_len: 37, ofs: 1458252 } Room 1 : Object { x_dis: 45, z_dis: 50, x_len: 40, z_len: 36, ofs: 1461212 } Room 2 : Object { x_dis: 45, z_dis: 87, x_len: 52, z_len: 28, ofs: 1464092 } So the x_len and z_len are the number of tiles in the x and z direction respectively. And the ofs, is the offset to the MDT table where the indexes are specified. That means that there are two bytes left which is where I'm guessing the game places the room definition where the flat tiles can be filled in. So I guess what I can do is make three flat planes (one for each room), and then see if there is an x and y placement using the x_dis and z_dis values that places them where the terrain is defined.
|
|
kion
Arukoitan
@kion_dgl
Posts: 193
|
Post by kion on May 6, 2020 17:20:46 GMT -5
I started looking into modifying Pcsx-Reloaded to make debugging memory easier. The save state is compressed by default, and the order the registers, system memory and graphics are packed in doesn't make it easy to edit even on the fly. So I decided to change my approach and write some scripts to make working with save states easier. 1. unpack.pyThis will take a save state, decompress it. Extract the system memory and write the system memory to its own file. It will then also append the system memory to the end of the save state file, re-compress the save state and write it back to the original file. #!/usr/bin/python
import sys import gzip
SRAM_OFS = 0x9025 SRAM_LEN = 0x200000 VRAM_OFS = 0x29B749 VRAM_LEN = 0x100000
DEFAULT_LEN = 0x431003
print '----- PCSXR Memory Unpacker by Kion -----' if len(sys.argv) < 2: print 'Error: Please provide a save state to unpack' print 'Usage: python unpack.py [save state]' sys.exit()
for index in range(1, len(sys.argv)): sstate = sys.argv[index] print index, '] Unpacking save state: ', sstate
f = gzip.open(sstate, "rb") f.seek(SRAM_OFS) mem_data = f.read(SRAM_LEN) file = open(sstate + '_mem', 'wb') file.write(mem_data) file.close() print index, '] Unpacking save state unpacked'
f.seek(0) data = f.read() f_len = len(data) f.close() if len(data) > DEFAULT_LEN: continue f = gzip.open(sstate, "wb") f.write(data) f.write(mem_data) f.close() print index, '] Patched to allow for restore later'
print '----- PCSXR Memory Unpacker Complete -----' 2. compare.pyThis is for looking at memory. It will take two binary files of the same length, and compare each byte. At the end it will print a list of the top 10 offsets with the longest chain off differences in the two files. Output: ----- PCSXR Memory Compare by Kion ----- 0) Offset: 0x00116958, Length: 0x00000cbf 1) Offset: 0x00118add, Length: 0x00000303 2) Offset: 0x00118871, Length: 0x0000026b 3) Offset: 0x000af0a4, Length: 0x00000259 4) Offset: 0x001de002, Length: 0x000001fe 5) Offset: 0x001dd802, Length: 0x000001f8 6) Offset: 0x00118e65, Length: 0x000001f5 7) Offset: 0x0011d00c, Length: 0x000001f4 8) Offset: 0x0011868f, Length: 0x000001e1 9) Offset: 0x0011955a, Length: 0x000001b8
#!/usr/bin/python
import sys from operator import itemgetter
print '----- PCSXR Memory Compare by Kion -----' if len(sys.argv) != 3: print 'Error: Please provide two files to compare' print 'Usage: python unpack.py [save state]' sys.exit()
f = open(sys.argv[1], "rb") a = f.read() f.close()
f = open(sys.argv[2], "rb") b = f.read() f.close()
list = [] index = 0
for i in range(0, len(a)): if a[i] == b[i]: if len(list) > index: index = index + 1 continue if len(list) == index: list.append({ 'offset' : i, 'length' : 0 }) list[index]['length'] = list[index]['length'] + 1
list.sort(key=itemgetter('length'), reverse=True) for i in range(0, len(list)): if i == 10: break offset = list[i]['offset'] length = list[i]['length'] print '%d) Offset: 0x%08x, Length: 0x%08x' % (i, offset, length)
3. patch.pyWhat this will do is take a save state, and first append the current system memory onto the end of the save state if it doesn't exist already. It will then read the provided external memory file, patch it into the save state, and then re-compress so that it can be read by pcsxr. #!/usr/bin/python
import sys import gzip
SRAM_OFS = 0x9025 SRAM_LEN = 0x200000
DEFAULT_LEN = 0x431003
print '----- PCSXR Memory Patcher by Kion -----'
if len(sys.argv) != 3: print 'Error: Please provide save state and memory to patch' print 'Usage: python patch.py [save state] [memory]' sys.exit()
f = gzip.open(sys.argv[1], "rb") f.seek(SRAM_OFS) mem_data = f.read(SRAM_LEN) f.seek(0) data = f.read() f.close() print 'Reading original save state data'
if len(data) == DEFAULT_LEN: f = gzip.open(sys.argv[1], "wb") f.write(data) f.write(mem_data) f.close() print 'Patched to allow for restore later'
print 'Patching memory into save state' f = gzip.open(sys.argv[1], "rb") a = f.read(SRAM_OFS) b = f.read(SRAM_LEN) c = f.read() f.close()
f = open(sys.argv[2], "rb") mem = f.read() f.close()
f = gzip.open(sys.argv[1], "wb") f.write(a) f.write(mem) f.write(c) f.close()
print '----- PCSXR Memory Patcher Complete -----' 4. restore.py#!/usr/bin/python
import sys import gzip
SRAM_OFS = 0x9025 SRAM_LEN = 0x200000 VRAM_OFS = 0x29B749 VRAM_LEN = 0x100000
DEFAULT_LEN = 0x431003
print '----- PCSXR Memory Restore Patch by Kion -----' if len(sys.argv) < 2: print 'Error: Please provide a save state to restore' print 'Usage: python restore.py [save state]' sys.exit()
for index in range(1, len(sys.argv)): sstate = sys.argv[index] print index, '] Reading Length of save state: ', sstate
f = gzip.open(sstate, "rb") data = f.read() f.close()
if len(data) == DEFAULT_LEN: print index, '] Cannot restore this data: ', sstate continue
f = gzip.open(sstate, "rb") a = f.read(SRAM_OFS) b = f.read(SRAM_LEN) diff = DEFAULT_LEN - f.tell() c = f.read(diff) d = f.read(SRAM_LEN) f.close()
f = gzip.open(sstate, "wb") f.write(a) f.write(d) f.write(c) f.write(d) f.close() print index, '] Save State has been restored', sstate
print '----- PCSXR Memory Restore Patcher Complete -----'
|
|
|
Post by crayon on Jul 6, 2020 22:30:25 GMT -5
Hi Thank you for the amazing work on the MML1 tools! And for documenting the process! Learning quite a lot from browsing these posts =). How goes the progress towards a full scene/stage viewer? Last post on the topic (on this thread) seems to have been about a year ago. Also, is there a difference between the Stage Viewer in the Save State Viewer vs the PSX Tools? I tried to open Stage 05, Tile 64 as seen in this post with the PSX Tools version, but the tile does not render correctly...
|
|
|
Post by crayon on Jul 7, 2020 22:58:22 GMT -5
Was curious what the models would look like in the viewer with sharper texture scaling. Below is an example (more can be found here). Accomplished by using `texture.magFilter = THREE.NearestFilter` and by adding an FXAA shader pass. The difference is quite noticeable on non-organic items. On characters, the effect is questionable. I am not sure if the original artists painted the textures with the expectation that they would be blurred/mixed smooth when scaled to final size. Before: After:
|
|
|
Post by betasword on Jul 8, 2020 16:21:31 GMT -5
Ok, now I definitely know that I'm doing something wrong somewhere. I posted in the stickied tools thread, but for me, the EBD viewer will only show and export models upside down and inside out. Clearly, that isn't the case for everyone, but that's been the case for me, regardless of browser used, and between both my desktop and laptop. Are you using the DAT files from the PS1 version, or the PC version?
|
|
|
Post by crayon on Jul 8, 2020 16:43:00 GMT -5
Ok, now I definitely know that I'm doing something wrong somewhere. I posted in the stickied tools thread, but for me, the EBD viewer will only show and export models upside down and inside out. Clearly, that isn't the case for everyone, but that's been the case for me, regardless of browser used, and between both my desktop and laptop. Are you using the DAT files from the PS1 version, or the PC version? I had the same problem. I had to modify the code as per here, to get it working locally. I'm not sure what changed to break the EBD viewer. Note that the change is only visual, I suspect if you export the models, they will still be upside down, with textures inside out. (I am using the PS1 files).
|
|
|
Post by betasword on Jul 8, 2020 16:47:11 GMT -5
Oh. Well, now I feel a bit silly for not realizing there was an already existing bug tracker. Thanks for the help!
|
|
kion
Arukoitan
@kion_dgl
Posts: 193
|
Post by kion on Jul 9, 2020 1:48:56 GMT -5
Oh. Well, now I feel a bit silly for not realizing there was an already existing bug tracker. Thanks for the help! I didn't even realize there were issues open. I don't want to pull a "it works on my end", but it does so I don't want to force flip everything before I have an idea of what's going on. Can you give me any information about the browser using being used, or example files? And what approach do you think is easiest to apply? Can I be lazy and get away with asking people to clone and edit the lines specified in the issue?
|
|
|
Post by crayon on Jul 9, 2020 15:15:13 GMT -5
On my end, I experience the issue regardless of browser used (latest Chrome and Firefox). All models in ebd render upside down and inside out, but in mdt render correctly. If you know of a model that appears in both, I can compare. The hack only works for what is viewed on the app, it does not fix exports. It's effectively a last minute flip of the final image before it is displayed. In the image below, the left side is what I get from using the original (web) version of the app, and on the right from the modified version. I think the proper approach would be to address how the model is generated.
Maybe there is a difference in the data files depending on where one gets them from? If so, some toggle/selection thing (to flip between how the model is generated) in the ebd viewer might be useful.
|
|
kion
Arukoitan
@kion_dgl
Posts: 193
|
Post by kion on Jul 9, 2020 19:26:36 GMT -5
I could probably throw an option to apply an upside-down patch into the EBD viewer. Out of curiosity does the same issue occur in the save state viewer?
|
|
|
Post by crayon on Jul 10, 2020 14:28:03 GMT -5
I made a mistake in the image I shared earlier. I was using `mesh.rotation -= Math.PI` and `materials.side = THREE.DoubleSide` instead of the changes shown in the hack (`mesh.scale`, `Three.BackSide`). Below (to the left) is what you would see if using the hack as is:
As you can see nothing is different from the export resulting from unmodified code. The changes are a pure last minute flip of what is rendered, and do not affect the internal model...
To the right, is what you get by using `materials.side = THREE.DoubleSide` instead of `THREE.BackSide`. The textures look okay...
Out of curiosity does the same issue occur in the save state viewer?
I haven't used the Save State Viewer. I'll give it a shot.
|
|
|
Post by crayon on Jul 10, 2020 16:13:11 GMT -5
I could probably throw an option to apply an upside-down patch into the EBD viewer. Out of curiosity does the same issue occur in the save state viewer? The models render correctly in the Save State Viewer.
|
|
|
Post by betasword on Jul 11, 2020 20:33:22 GMT -5
I can also confirm the save state viewer displays and outputs models properly. Only issue I'm having with that tool at the moment is that it doesn't want to display the correct floor tiles in most areas when looking at the room tables. Also, for whatever reason, a room exported with the save state viewer and then imported into blender defaults to either medium or low quality for its textures, but that could be any number of weird things on my end, honestly. Even with these small issues, though, these tools are absolutely incredible. Just wanted to throw that out there.
One week later edit: Like I said, these tools are awesome, and are proving to be extremely helpful at the moment.
|
|