Caring and sharing

I'm find it difficult to completely separate myself from the project. I guess that's the problem with stuff you do out of interest. I don't have as much time as I used to though, so progress will be even slower than it was before.

By request, and because I may find myself dragged away from the project completely at some point, I've put the code that I've managed to get working on Google Code. Not intended for general consumption though.

Sadness

Well, unfortunately this project is going to have to be put on hold. Don't blame me, blame my employer: I'm having to spend all my spare time working stuff that will help me get another job. Hopefully one day I'll come back to it, but I don't have any idea of the timeframe for that. Any animation / digital effects companies want to give me a job as a pipeline software engineer?

Rollercoaster

You know those tight little rollercoasters that surprise you when they make an incredibly sharp turn that you weren't expecting? I'm trying to follow the t7g exe file in OllyDbg, and it's a bit like that. I'm not really at the stage where it's useful, it's just for fun and to try out OllyDbg. Though hopefully it'll all soak in by diffusion as I stare at the screens of assembly language...

Progress

Despite thinking that I would not get a chance to do anything with the project this week, I have. I'm slowing collecting reversing tools and information, and I'm making progress. I should point out that, at the moment, I'm only studying my extract_gjd tool, but I'm getting much more to grips with the way IDA works, and some of the basic processes involved in reversing. I'm still a little confused by what the first few lines of assembly do (in seemingly every program), but I can pick out most of the rest. I'm pretty sure it's just a case of getting my head properly screwed on and really sitting and working at it.

Unforgotten

I'm currently getting through the Reversing book that I mentioned in a comment to my last post. I realized early on that it wasn't a magic wand, the process is still going to involve a lot of learning-by-doing on my part, but it certainly puts me on the right track and has some helpful tips (even though it is directed at Win32). The problem is that I'm reading an electronic copy from our Uni library, and I'm much slower at reading electronic books (and find it less pleasant): it's not too expensive, I might have to buy a copy.

Have RL events to attend to for the next couple of weeks, then I'll be back to it...

My eyes are square

I've been staring mindlessly at assembly language for hours. I've got a book, and I can figure most of the stuff in there, but when I try and translate that to the T7G exe, it seems worlds apart. I'll try and find a reverse-engineering book in our uni library tomorrow, hopefully if a book is coming from the same angle as me it might help.

Forlorn.

GJD capable

The video above celebrates the fact that my "engine" can now pull a vdx from a gjd directly and play it (previously, in case you haven't been following, I had to extract the vdx with a command line tool first).

Enough with procrastinating. Now I have no excuse but to attempt some disassembly. Well, I could work out how to do sound / xmidi, but I think I'll save them for later.

Praise Google Desktop!

So I'm just about to turn off Google Desktop, in case it overwrites the part of the hard disk that the files were stored on.

I mean, how often do I need to search for files on my computer anyway? It's not like...

Hmmm.

It lets me search for strings inside, for example, source code files. Does that mean it stores cached versions of those file?

Bingo!

OK, I don't have them all back, but the only ones I'm missing are the detection files and the makefile, which should be a piece of cake to do again.

I may have to keep google desktop for at least a little while, if only to say thank you.

STUPID

This could be my last post for a while. I have been exceptionally stupid, and managed to delete my entire engine. I'm not gonna go into details of how I did it, it's far too depressing, but I've tried "recover" and "e2undel", to no avail. I'm now in the process of grepping my entire hard drive (which has to be done in 100MB blocks, otherwise grep runs out of memory) to see if anything is salvagable.

Suffice to say, if I get everything back up and running again, or if I have to start from scratch (my last way-point was my empty engine), I will be setting up a backup system, probably an svn server.

Grrr.

Muppet city

As expected, the problem was with me being a muppet. When telling it the length of the data (for example, for the initial still image), I was giving it the length of the compressed data, rather than the uncompressed. So it was only getting partway through. Now it works, yay!

Where to go from here. There are a number of options:

1. Get it to read this vdx from the gjd file
   
2. Get it to play back the sounds from this vdx file
   
3. Try playing back the xmidi files
   
4. Start reing the executable

Although it may seem like I'm procrastinating (hell, of course I am, disassembling still scares me), I'm gonna sort out reading the vdx from the gjd. Everything is stored in gjd files, including the xmidi files, and I think it will be good to get me into the C++ mindset.

I preferred it when it when too fast

Got the timing working. Problem is that now it's running slowly enough for me to see artifacts. *sigh*. It's easier to fix them when your program outputs every frame as a bmp. Maybe I'll get scummvm doing that...

Moving pictures

If you can see it at this crap quality... I managed to get the animation frames working. Timing is not adhered to (by any means), but all the frames are in there.

ION, given I have 4 bluetooth devices, 2 of which have video cameras on, you'd think I'd be able to transfer the video easily. No. It was possible, but I had to open the linux app that received files. Would be good to have that built into the bluetooth, or at least the option to start it whenever a bluetooth device is connected. Anyway, done now. The next question is, will this video work?

It looks more impressive than it is...

OK, nobody get too excited (though I am, but I know what this is). At the moment, this is just reading the first frame of one of the animations. Don't get me wrong, this is a big step, but ... there is a whole 200 storey building to climb after this one :-). But still, yay!

Decompression woes yet again

Hmmm.
Looking at the LZSS decompresser in the scumm he engine, that works by just allocating big enough chunks of memory to decompress the entire stream. This is all fine and good. But *my* variant of LZSS doesn't tell me how big the decompressed data is, only the compressed data. This, it turns out, is a bit of an arse.
At the moment I'm planning just to say "it's never gonna be bigger than this" as a nasty hack, just to get something working. In the future, when I'm feeling a bit more confident, I'll write an LZSS-compressed stream class (unless someone beats me to it :-).

Finding my feet

Well, this is something of a non-post. Or at least, it will seem it to anyone that's not me. I'm trying to get my engine to just play a vdx file first, just to find my feet. I'm getting my head round the way everything's done in ScummVM, slowly, but it's not helped by the fact that my C++ is rusty: I wrote the extraction tools in plain C. Might have to get a book out of the uni library tomorrow. Anyway, I have figured out how to draw to the screen, & open and read from a file... but now I've got to decompress a stream of data on the fly. In the tools I wrote before, I extracted to a temporary file, then read that back in. I knew it was a hack, and that I'd have to do it properly, but just sort of ignored it. Thankfully, there is already an LZSS decompresser in the SCUMM engine: it uses different settings so I can't use it, but I should be able to use it as reference.

Man that was a boring post. Maybe next time I'll have some pretty pictures... hopefully.

Emptiness

Well, I finally managed to get an empty engine compiled into ScummVM, yay! I've not heard back from the ScummVM leads yet, but that's ok: even if it won't be part of ScummVM, the ScummVM app means I don't have to worry about getting my own graphics / sounds setups, etc.

Google

Hmmm... as pointed out on #scummvm, Google seemed to be put out by me claiming that Googling "t7gre" did not find my blog, and now it very much does. Do they only list blogs that have been posted in recently maybe? I've googled occasionally for this in the past, and it seems rather sporadic whether it appears or not.
Anyway, I've emailed the leads of ScummVM now, to get their input on whether ScummVM could include T7G at some point in the future.

Disassembling number 7

Well, disassembling doesn't scare me as much as it did. Which is obviously a good thing. I used the free version of IDA, and it looked like I could get some good stuff out of it (bit tired tonight though). The only thing is that it won't look at the dos executable (i.e. the original), because it's only a demo (the full versioin will hack into any executable you care to throw it at, by all accounts). Thankfully there is a windows executable (available from the links on the left), but I can't help feeling that I should be using the DOS one. I imagine it to be simpler, but maybe I'm making that up.

Assembly language

I don't wanna!

Scuppered!

Well, tonight I got a mention on #scummvm. To answer the questions that were asked there, the reason I haven't discussed this with the ScummVM team is because I wanted to get a foothold on it before asking. That stage was to come soon enough.

But now I'm intrigued. This blog doesn't even show up (atm) if you google for t7gre... how on earth did it get stumbled upon? Bizarre. Ah well. I just have to assume now that someone might actually be reading this :-)

Another gift!

It turns out the music, contained in the XMI.GJD file, is in xmidi format. Which scummvm already supports. Woohoo! This says to me that the next stage is probably going to be adding the engine to scummvm ... a big task.

Oh, and I'm going to rename all the files so they have small letters. internally they're all referred to in small letters, so for the time being I may as well be consistent. Eventually I guess it's gonna have to cope with both, but I prefer the idea of being short-sighted for a bit.

And I'm going to redo the database, this time with all the unknown data values, and possibly bytecode totals for all the resources. Maybe. Still haven't decided what format the database should be in, I'm having a go with OpenOffice database, seems ok. Whether it'll read in comma delimited data, who knows...

Building database

Well, I figured if I was going to get every bit of this game working, I'd need to know which part is where, etc. So I'm unpacking everything. That may not sounds very exciting (and it's not), but I was quite impressed to find that just INTRO.GJD, which starts off at about 30MB decompressed to about 900MB. So the whole game (about 650MB) will probably take up 20GB. And then I just have all the decompression data to put into some sort of usable form...

/me trundles off to investigate MySQL

And I'm spent

I managed it in the end. I figured out not only what the problem was, but why I had a problem with that frame in particular. The frame uses a particular opcode, 0x60, a lot, when others don't (at all? Not sure). That opcode gives very precise control over the image: where the previous frames have been slightly "rushed" as it were, to save space, this one is done very precisely. If you remember the intro to 7th guest, you'll know why: all the action for the next few scenes takes place on the pages of that book: it basically doesn't change for a minute or more.
But that's not important. The important thing is that it works. Now I've just got to try it on all the others and find the ones it doesn't work on...

Noooooooooooo!

It was all going so well! I'd finally got the palette completely sorted (how could it be so difficult?), the first 30 frames were perfect, then this! Where do I start trying to figure out what's wrong with this? I just don't know...

Still bad, but getting better

As you can see, I'm getting there...

My sticking point

In case anyone is trying to do the same (unlikely, I know), the thing that was confusing me was that the 0x61 opcode, which skips to the end of the line, is also required at the end of each line (like a new line character). As such, you don't need to keep track of the number of tiles in the x direction that have been processed.

Woo hoo!

Though barely recognizable, this is a breakthrough. This is the house, after we've pulled out to see it in the book (remember that bit?). It may seem like small comfort, but it pretty much just the palette that's screwed. Hopefully.

End of night post

This is to aid my memory when I come back to this tomorrow. Too late now.

Program segfaults. This is because it goes beyond the end of the image array. This is because it goes beyond the correct number of blocks in y. I do not know why this is, I must have interpreted the file wrong... right?

Progressing

Next stage. Still not really recognizable (I think it's just gonna be the mansion again, btw).

Not quite there yet

Was getting bored of writing, so thought I'd post some images. This could be an interesting transformation... I think I've screwed up the palette on this image. I'm gonna be putting it together a few pieces at a time, so we'll see!

Valgrind is teh Magic

It seems that when your code is shonky (and you have an inkling that it might be), valgrind is where to look. It'll point to things, even when your program appears to be working, and say "err, are you sure that's right? it looks like you're being a bit of a muppet there". And you can ignore it (I did, mostly cos it took me a while to figure out what its error messages meant), but eventually they'll come up and bite you in the seg fault.

In short, I feel it is going to be invaluable. Even more so than it has been tonight.

Don't Panic!

It's ok, it was a very easy thing to fix. Phew. Now, where was I...

Well, at least I liked Square 1 when I was there first time round...

There is no expletive to express this.

Up until now, I have been working with the smaller GJD files that get installed to the hard drive, but I couldn't find a delta frame there that had a palette change. So I went for a biggy: the INTRO.GJD file on disc one. It's good I did, because I've discovered that the bit that was easy, the extraction from the GJD file, doesn't work properly. It gets about two into it (out of a couple of dozen), then falls over.

Still, I've already learnt / re-learnt quite a lot since I did that bit, so it might be obvious...

Decompressing V.EXE

As a distraction from those scary delta frames, I used a combination of unlzexe and dosbox to decompress V.EXE, which was compressed using LZEXE. So now all I've gotta do is disassemble it :-)

Procrastination, that's what you need

As expected, I'm putting off delving into the delta frames format. As you might expect, it's pretty complicated, so I've been trying to make sure my program is ready for it. I've now got a proper image struct for storing the images, which makes it much easier to have a second one to store the new image. Part of me feels I should be doing this properly in C++ classes though... I'll have to eventually...

Tada!

And it was just a comparatively small step from there to here!

The video frames (i.e. each one after this, the first one) are stored differently again, I think they're going to be even more tricky to pull out, but at least I know my BMP writer works!

Picture posting

So I figure, now would be a good time to start posting pictures. Here's the mansion, as my program outputs it atm. Clearly far from right, but it is there if you stand on your head and look hard...

Decompression fixed

Yes, there was a problem with the decompression. This would have been obvious with a little bit of arithmetic: the output .dat file was 2.5k, and it should have been about 50k. Fixed now. The BMP is still screwed, but in a much less terminal way (I hope :-).

Bugger

Hmph. I think there is a problem in my decompresser. And after I was so happy that I got it working and everything.

The problem is that I have no intermediate. I can check the vdx files are correct, but after that I have to go:

vdx -> individual chunk -> decompresser -> read image -> write bmp

The only stage where I can actually check that everything is going according to plan is at the stage where it has made me a bmp file. Which is rather frustrating. I'm used to checking at every single stage along the way, to make sure all is going well, but here I don't get that option. Unless I try to take apart the pascal code... no, I think that would be going too far. This will work, it will just take care and hard work.

Ugh...

And now, the task ahead is to convert my pretty little file into a bmp file, so that I can view it. Which is a decidedly dull task, so I keep putting it off and off.

I emailed the guy who re'ed the vdx and gjd files, and he was supportive, but he only ever concentrated on those files. Looks like I've got all the help anyone's able to give me...

Right. BMP files. *sigh*.

Decompression complete

Woohoo! It was a challenge, more mentally taxing than I'm used to, but I now understand the algorithm and have it working in my program. At the moment it just dumps the decompressed files to disk rather than converting them into any sort of viewable format, but that should, I hope, be comparatively straight-forward...

Holy compression batman...

It turns out this is going to be a little more tricky than I thought. Not much, but a little. The video sequences use a form of data compression, LZSS, which I'm gonna have to write a decoder for. Which would be a huge set-back, or at least would slow me down an awful lot, if it weren't for the fact that someone (at the page linked to above) has written a pseudo-code decoder. I say pseudo-code, it's pretty much pascal. I've begun by just directly translating it, but I'm starting to get confused: I think I'm gonna have to figure out how it's working first, then implement it after.

Starting vdx extracter

Well, I've set off on the vdx file extractor. And although reasonably straightforward, it is definitely NOT going to be a one-night job like the gjd extracter.

One thing about it is that unlike the gjd file, where every byte is accounted for, there are several bytes here and there that the handy specs linked to earlier just say "unknown". Now, they didn't have to worry about them, because they had the sole purpose of extracting resources from the files, but I'm probably going to need what's in them at some stage. So I'm storing them, but I have to come up with variable names for them when I have no idea what they are / what they do. I've decided to follow the theme, and name them (initially at least) after characters in the game. I should point out that I don't know those names off the top of my head: I'm not that obsessed with the game! I looked them up on wikipedia, so if they're wrong no-one can blame me. As if anyone will notice / care :-)

Easier than expected

Using the ScummVM tools as a base, it was really easy making an extracter for the .GJD files. Admittedly, it's not difficult, it essentially just a bunch of files stuck together, but I had expected to take a while to get going, so it's good to get an easy bit out of the way. Next (but not tonight), the vxd files, which although more tricky, really shouldn't prove too much of a problem.

Hmmm...

Having not managed to get any of the game extracting things working under linux, I thought I'd write little tools first that would extract the resources from their respective files. But someone beat me to it. And they run perfectly under wine, unlike everything else I've tried this evening.

I think it's still going to be a good exercise to write the tools in C though: it'll get me to know the file formats well before starting anything more complicated. It's just that I'll be translating from Pascal to C rather than writing from scratch!

The long journey begins

There is some information on the web to get me going: the most helpful is that someone has re-ed the file formats. I've been trying to get MultiEx Commander working under linux, but it doesn't seem to like wine. Which is rather frustrating. There's no way I could do this under windows, so I'll see if I can find anything else to help me out...

Welcome to my... blog

So this is my attempt to actually get started on this: start a blog, then I'll get really depressed if I don't put anything in it for a while and be forced to do something.

I am intending to reverse-engineer The 7th Guest, the eventual intention being (if they'll have it) to add it to ScummVM.

I should point out that I have no experience of reverse engineering, little experience of ScummVM programming, and only moderate experience of programming. The thing I have in abundance, though, is a willingness to learn these things. Hopefully that will be enough!