Groovie, baby

Happy birthday

2010-08-07T09:19:00.001+01:00

I was shocked to notice that, as of yesterday, it has been a year since I posted anything on here. That's not really that surprising, given that I've been well and truly absent from the ScummVM community, but I plan for that to change. I've set aside some time each week for personal programming, either ScummVM or Android (for which I've set up a separate blog) (or possibly a combination of the two, now ScummVM has an Android port).

I also suddenly realized how out of date the title and description of this blog was (referring to "motions towards", etc), so I've made it a little more relevant. I didn't want to put a pun in, but when the engine name is Groovie it's hard not to. Forgive me?

Call for playtesters...

2009-08-06T09:22:00.002+01:00

We need more people to play The 7th Guest! ScummVM is aiming for it's 1.0.0 release, so we need everyone to test all games, to check that no bugs have recently been added. In particular to this blog, we (well, ST) just fixed a big bug in the music (that was present in the Win95 beta player too, but not in the DOS exe), and no-one has played through the whole game since, to make sure it didn't break anything.
So if you have a bit of time, download a daily build of ScummVM, and play through the game, looking out for bugs. If you complete it, post a comment here saying which operating system you played it on and whether you found any bugs.

The game just got longer

2009-07-21T10:10:00.003+01:00

Just a quick note, as I'm at work. The 7th Guest just got much harder to complete in ScummVM:

http://scummvm.svn.sourceforge.net/viewvc/scummvm?view=rev&revision=42634

Please give it a test if you can and report back!

ION, I'm in the midst of a house move, so won't be doing anything ScummVM-related for some time.

I'm not saying Apple are evil, but...

2009-06-21T00:37:00.003+01:00

I don't use the word "hate". It is too strong for most of my emotions. If I did, however, I would most likely be using it right about now, with regard to HFS.
When copying the files for the disc (with hfsutils, see previous post), there are several modes that can be used to copy the files: raw, binhex, macbinaryii, text. There is also an auto mode, which tries to make an intelligent guess as to which mode should be used. Unfortunately, it doesn't get it right for the T7G data files: it tries to copy them as text. They need to be copied with the raw mode, but the binary (which also, helpfully, contains another selection of required files) cannot be copied in raw mode. That can be copied with auto (which guesses correctly that it should be copied with the macbinaryii mode).
I think before I dig too deeply into support for the Mac version I'm going to have to learn a little more about how HFS works, and why this confusion arises. I remember a Mac-based friend trying to explain to me (about 15 years ago) all about resource forks, data forks, etc. I just sat there thinking "surely a file is a file? it start, has binary data in, then ends, and different files are interpreted in different ways". I wish I was still in contact with that friend so I could phone him and get him to repeat the conversation...

Edit: It seems I'm not alone in my view of HFS: http://www.engadget.com/2008/02/05/linus-torvalds-calls-apples-file-system-utter-crap/

HFS 2: The Revenge

2009-06-20T21:24:00.002+01:00

When I originally got my copy of T7G for Mac, I diligently copied all the files off it (twice, in fact). However, I now need to do it again (for reasons that I won't go in to), but this time I have an ISO image file (I ripped it when I was faffing with the disc last time). I have since completely reinstalled my OS and I couldn't for the life of me remember how I did it the first time round.
The ScummVM HFS wiki page is rather Windows-centric, so I had to figure out (again) how to do it on Linux (specifically Kubuntu Jaunty). I was surprised at how easy it was, but as it might be useful for others (that might be willing to play test in future ;-), I figured I'd post it here.
Once I'd installed hfsutils (e.g. sudo aptitude install hfsutils), a simple call to "hmount hfsimage.iso" does a virtual mount of the image, so any future calls to the hfsutils commands (hls, hcd, hcopy) will be referring to that disc / image.
Now, which file was it that I wanted... I knew I should have written it down...

Mac attack!

2009-06-16T23:31:00.004+01:00

Well, free time has been sparse for all of us. My work has gotten much more hectic (partly thanks to a promotion a couple of months back, so I can't really complain), but I'm trying to make more time over the next few weeks for some ScummVMing. Though I will also be moving house, so we'll see how it goes.

Anyway, my plan is to get back into it by working on T7G Mac to begin with (sighs all round, I'm sure, but 11H will come at some stage no doubt). It's filled with little tricksy problems, not even including getting the files off the disc in the first place. In approximate order of difficulty:

Some files need to be renamed to work
Files are squashed into a single data archive (jvprat has written an extractor)
Fonts are not included, as in the DOS version: native Mac system fonts were used
Music was not MIDI, but Special Mac Fancy Format™

This last one is the real biggie, so don't expect lovely Mac music anytime soon. I'll probably have got distracted into 11H before I manage to get Mac music sorted.

PS Btw, if anyone wants to follow me on Twitter, you can find me here: http://www.twitter.com/hjsb. Though it's mostly just bitching about how the artists at work don't do what the pipeline guys tell them to.

The future is bright

2009-03-11T18:29:00.002Z

Well, for those that haven't read jvprat's comment on my previous note, the list of major problems looks a little different now:

~~The engine is currently designed to play just one game~~ - jvprat has committed changes that let us use two separate opcode lists. May need changes in future (apparently), but not holding us back anymore.
~~11H uses > 8 bit graphics (i.e. more than 256 colour images)~~ - Well, technically this is still an issue, but we can view everything in greyscale / dithered for the time being
~~11H uses a completely different video format~~ - Might need some changes later, but basically working
The script opcodes have changed - The only one left, and Scott is having fun working on it. I only wish I had a bit of time to join in!

Past, present and future

2009-03-02T20:54:00.000Z

Well it finally happened. The new version of ScummVM contains the Groovie engine, capable of playing The 7th Guest. It's quite exciting really. But now I'm just thinking about where to go from here.
It's not like T7G is completely finished: see the ScummVM wiki for more info on known issues. But it's close enough for now: I'll come back and look into them at some point in the future, as for now I have bigger fish to fry.
I'm sure there are a lot of people out there who are thinking that adding support for The 11th Hour will be easy, as it uses the same engine. However, let me explain to you why that's not the case:

The engine is currently designed to play just one game.
11H uses > 8 bit graphics (i.e. more than 256 colour images).
11H uses a completely different video format.
The script opcodes have changed.
Those are just the big things that are obvious from the outset: I'm sure there are other smaller things (have you noticed the animated cursors blend together?).

Number 1 will involve refactoring the entire engine: no real changes, but involves difficult design decisions.
Number 2 is the real biggie. It's going to involve even bigger design decisions that will affect the whole of ScummVM, not just the Groovie engine. If ScummVM is accepted into the 2009 summer of code, it's one of the possible tasks, so we're going to wait and see if it gets taken up before even thinking of undertaking it.
Number 3 isn't as big as you might think: jvprat has already implemented a basic player from format descriptions on tinternet.
Number 4 will involve lots of fun reversing. We know a bit better what we're looking for this time tho!

Lots to do then, and very little free time to do it in. But there's no hurry :-)

Supported

2008-12-17T09:55:00.004Z

This post is probably pointless, as probably the ones of you that follow this, also follow ScummVM's news page. But as of yesterday, The 7th Guest is officially supported by ScummVM. This means that it will be included in the next ScummVM release (0.13.0, whenever that may be), and that we're accepting bug reports to the ScummVM bug tracker. And if anyone submits "the microscope puzzle doesn't work like the original" then I won't be impressed :-P (see previous post if you don't know why).

It's not quite what I intended...

2008-11-22T23:44:00.002Z

The microscope puzzle. As many of you might have predicted, even the non-programmers, it's one of the more difficult bits of the project. In fact, just as it was one of the more difficult bits of the game.
I'm going about it in a way that feels like cheating. Basically, I'm completely ignoring the disassembly for now, and implementing my own algorithm (with inspiration from this tool) play the game (hopefully well). The ideal thing being that once I've done this, and have a really good idea of how the game works, I then look at the disassembly and figure out more easily (though certainly not easily) how it works. But it is possible that this latter step may get... postponed. Depends how well my algorithm works (and how different its playing style is from the original's).

The End of The Beginning

2008-11-15T09:06:00.003Z

Well, as of ScummVM r35060, the Groovie engine is part of it. Yes, that's right folks, if you download the Daily Snapshot of ScummVM from now on, you'll be able to play The 7th Guest with it.

We're going to keep the google code project going, but the svn repos there might not always have the most recent code: that will be in ScummVM. And this blog will continue indefinitely :-).

Completeable!

2008-11-07T16:12:00.003Z

... kinda. Apparently. As of r328, the game should be completable without the use of the library. However, no-one has played it through from start to finish: if anyone wants to try to play the game all the way through, please let us know if it works! As stated in the last post though, disassembling the AI for the microscope puzzle is going to take quite some time as it is very complicated: at the moment Stauf may seem considerably more stupid than you remember him (in fact, his moves aren't even valid: he basically does nothing the entire game). But still. The game's completeable! Yay!

19/21

2008-11-07T09:58:00.002Z

Wow.

I leave the project for one day, and come back to find that Jordi has changed four bits in the source code, leading to the fact that at least five more puzzles are now complete. Of the two that are left, one might work (apparently, just about to test it myself :-), and the other is going to be very, very difficult. It's the microscope puzzle, which if you know the game, you'll know is very different to all the others, as it involves an albeit basic form of AI.

Call for versions...

2008-11-05T13:51:00.004Z

We're trying to figure out how many different versions of groovie games (particularly t7g) there are, and the differences between them. If you want to help out, you'll need a copy of t7g (or other Groovie game), the ability to create an iso file from it (e.g. one of these pieces of software: http://en.wikipedia.org/wiki/List_of_ISO_image_software), and the ability to generate md5 hashes from the resulting files. Then post the relevant information as a comment on the checksums page. We will also need the executable version number: this can be found by running the executable (v.exe for t7g, v32.exe for 11h) in a command window (or in Dosbox).

14/21

2008-11-04T00:25:00.000Z

http://code.google.com/p/t7gre/wiki/Status

But there has been progress!

2008-11-02T21:36:00.002Z

It has been pointed out to me, that since I last posted any actual news about t7gre, quite a lot has changed. Jordi has been working away since I last posted (about MIDI music working), and we now have a wiki page to keep track of which puzzles are working, and 13 out of the 21 puzzles in the game are working fine. That's an amazing testament to the work that he's put in over the past few months. But now he's getting distracted in other directions... take a look at this!

Trying to get back on a rolling ball

2008-11-02T15:57:00.002Z

Well, the most hectic time on The Tale of Despereaux is (probably) over, and I have a little more time. Just enough time, in fact to spend a substantial amount of this weekend installing Kubuntu 8.10, and to get pissed off at KDE 4.1 (which, despite its prettiness and everything, seems to have less features than KDE 3.5).
But I have managed to do a little on t7gre, even if it is just to fix a small regression that has been there for goodness knows how long... ever since Jordi revamped the movie player to, y'know, work properly. Shows how often we watch the intro! Well, it is ridiculously long...
And so just as I'm thinking I'll dig back into the assembly, I discover that VirtualBox, the method by which I run WinXP IDA on Linux, has stopped working. Grrr. Having re-isntalled it, it works, but my VM now BSODs on startup. I have to install Windows. AGAIN.

Calm down. Relax. Man, I wish I could afford copy of IDA for Linux...

Distracted

2008-06-14T17:21:00.002+01:00

Well, apart from only getting in from work at 8pm each evening, I now have something else to distract me from t7g: we got a Wii. I'm sure sooner or later I'll get bored of it, but atm its sucking up all my spare time :-)

But don't be downhearted: jvprat has been working in my absence, and he's implemented music! Which is v cool: remember the music when you click on the painting at the top of the stairs? I did ;-) It's not fully implemented yet, but it's a good start!

Calling Jake

2008-05-20T23:17:00.004+01:00

When I first played The 7th Guest, it was at a friend from school's house: he was the only person I knew with a cd-rom drive. We played it, I think, all the way through, and I have many happy memories of those times.
I have barely spoken to Jake since, and over the last year or so while I've been investigating the game, I've also been remembering those times. I thought I'd let him know what I was up to, and to remind him of those times too. But I can't find him on tinternet at all, it has failed me! So Jake Taylor from Hale, if you're reading this (bit of a long shot), leave a comment!

Messiness & recovery

2008-05-20T22:47:00.002+01:00

This post is predominantly an apology. In order to get the surfaces working the same way that they do in the original, I've had to hack apart some of the beautifully structured code that jvprat wrote, and my coding is not as... well, good.
Anyway, the internal structure is more suited to continuing work now, but there are extra graphical glitches for your delectation. Mmmmm.

Delayed spiders...

2008-05-18T12:09:00.003+01:00

Well, it took longer than I'd have thought (I don't have as much time to dedicate to this project as Scott does, ok?), but I finally got the spider door puzzle working. Turns out that the engine uses two surfaces, a foreground and a background, and the front one is partially transparent. Here's me thinking i wouldn't have to worry about sprites and the like. Still, done now. Probably not the most efficient way, but hey it works. Cake next? Nah, telescope I think :-)

Uncle Henry

2008-05-02T17:20:00.002+01:00

Well, jvprat found an auction of what is probably* the last groovie game: Uncle Henry's Playhouse. Astonishingly, the person selling had three shrinkwrapped copies (and it doesn't look like Butterfly-ware, before anyone asks!). I had looked everywhere for this, so I was willing to pay the (quite high, but not FM-Towns high) price. There's still one copy left, if anyone wants it:
http://cgi.ebay.co.uk/ws/eBayISAPI.dll?ViewItem&rd=1&item=380019732447&ssPageName=STRK:MEWN:IT&ih=025
ION: I have no internet at home. Which makes work on t7gre awkward (and means I have to post this from work). But I'm getting some stuff done. Hopefully I'll get internet back start of next week, and be able to commit it.

* I say probably because we're just assuming it's a groovie engine game. Seems likely, as it is just a collection of puzzles from the earlier groovie engine games, but you never know till you see a file list!

Error in the original?

2008-04-24T22:57:00.003+01:00

I think I found an error in the original (well, windows re-release) of T7G... I can't explain it any other way. If you don't understand assembly, this will probably be lost on you, but I'm going to paste it in anyway:


.text:00404B5A                 mov     eax, [ebp+var_C]
.text:00404B5D                 imul    eax, 3
.text:00404B60                 mov     ecx, [ebp+cursor_file]
.text:00404B63                 mov     dl, [ecx+eax]
.text:00404B66                 mov     byte ptr [ebp+var_1C], dl
.text:00404B69                 mov     eax, [ebp+var_C]
.text:00404B6C                 imul    eax, 3
.text:00404B6F                 mov     ecx, [ebp+cursor_file]
.text:00404B72                 mov     dl, [ecx+eax+1]
.text:00404B76                 mov     byte ptr [ebp+var_1C+1], dl
.text:00404B79                 mov     eax, [ebp+var_C]
.text:00404B7C                 imul    eax, 3
.text:00404B7F                 mov     ecx, [ebp+cursor_file]
.text:00404B82                 mov     dl, [ecx+eax+1]
.text:00404B86                 mov     byte ptr [ebp+var_1C+2], dl

Can it be true? It accidentally uses the green channel twice instead of the blue one. In something as important as the cursor handling it seems unlikely that it would go unnoticed. But then, this is in a fairly obscure part and wouldn't affect all colours in the cursor. Interesting.

Mobility

2008-04-14T22:27:00.004+01:00

Wow. Courtesy of jvprat, the mansion is now navigable. This is not *that* big a step from where we were before, but it feels enormous because it actually makes the game at least partially playable (whereas before we essentially just had a movie player). As you can see, the engine also compiled on Windows CE with no changes at all.

There is still plenty left for us to do: the MIDI music, save games and, well, the puzzles. But this is so exciting!

Employment

2008-03-25T21:52:00.002Z

OK, so i got a job. Which is good for me, but not for this project. It's likely to be mostly on ice for a while, which is a shame given the huge burst of progress recently.

If anyone figures out anything about transparency, do let me know...

Ballcocks

2008-03-20T00:01:00.002Z

I've lost my programs to extract vdx files from gjd files. This is rubbish. All the details of the file format are on the web (see the links section), and we can already play back vdx files. But the vdx playback is incomplete (can anyone say "transparency"?), so I'm trying to figure out what the exe file is doing at that point, and damn it it would be really helpful if I had them. Why didn't I put them in svn?!

Progress. Like real progress.

2008-03-16T16:24:00.003Z

Wow. Given that this time last week we had absolutely no idea about the file formats of the .grv files, we have come a loooong way. Our opcode list is now more complete than not, and jvprat has implemented a skeleton script interpreter which we can now go about adding opcodes to one by one.

It's still going to be quite some time before anything is playable, but it's on its way! In the meantime, here's a photo of some of my working...

Excitement

2008-03-12T00:22:00.003Z

It's got to the exciting bit of the disassembly where stuff starts to make sense! Almost. And thanks to coding and words of wisdom from jvprat, we're really starting to get somewhere. Here's the proof:

http://code.google.com/p/t7gre/wiki/Opcodes

There's still a long way to go (there are 88 opcodes in all), but at least now I feel like I'm getting somewhere when I'm doing it. And it certainly helps to bat ideas around with someone!

Text

2008-02-08T00:13:00.001Z

It's been a while. A lot has been going on in RL, but little in t7gre world.

But things change. Yesterday I had a play with reversing some of the font code, and actually got somewhere. I now have a (very) basic idea what the format of the font file is. Here's a picture, which proves nothing (I could have figured this bit out just from looking at the file in a hex editor).

It's an "A", in case you can't quite figure it out.

Caring and sharing

2007-11-07T00:39:00.000Z

I'm find it difficult to completely separate myself from the project. I guess that's the problem with stuff you do out of interest. I don't have as much time as I used to though, so progress will be even slower than it was before.

By request, and because I may find myself dragged away from the project completely at some point, I've put the code that I've managed to get working on Google Code. Not intended for general consumption though.

Sadness

2007-10-18T21:54:00.000+01:00

Well, unfortunately this project is going to have to be put on hold. Don't blame me, blame my employer: I'm having to spend all my spare time working stuff that will help me get another job. Hopefully one day I'll come back to it, but I don't have any idea of the timeframe for that. Any animation / digital effects companies want to give me a job as a pipeline software engineer?

Rollercoaster

2007-09-26T00:46:00.000+01:00

You know those tight little rollercoasters that surprise you when they make an incredibly sharp turn that you weren't expecting? I'm trying to follow the t7g exe file in OllyDbg, and it's a bit like that. I'm not really at the stage where it's useful, it's just for fun and to try out OllyDbg. Though hopefully it'll all soak in by diffusion as I stare at the screens of assembly language...

Progress

2007-09-05T23:05:00.000+01:00

Despite thinking that I would not get a chance to do anything with the project this week, I have. I'm slowing collecting reversing tools and information, and I'm making progress. I should point out that, at the moment, I'm only studying my extract_gjd tool, but I'm getting much more to grips with the way IDA works, and some of the basic processes involved in reversing. I'm still a little confused by what the first few lines of assembly do (in seemingly every program), but I can pick out most of the rest. I'm pretty sure it's just a case of getting my head properly screwed on and really sitting and working at it.

Unforgotten

2007-09-01T13:19:00.000+01:00

I'm currently getting through the Reversing book that I mentioned in a comment to my last post. I realized early on that it wasn't a magic wand, the process is still going to involve a lot of learning-by-doing on my part, but it certainly puts me on the right track and has some helpful tips (even though it is directed at Win32). The problem is that I'm reading an electronic copy from our Uni library, and I'm much slower at reading electronic books (and find it less pleasant): it's not too expensive, I might have to buy a copy.

Have RL events to attend to for the next couple of weeks, then I'll be back to it...

My eyes are square

2007-08-27T21:53:00.000+01:00

I've been staring mindlessly at assembly language for hours. I've got a book, and I can figure most of the stuff in there, but when I try and translate that to the T7G exe, it seems worlds apart. I'll try and find a reverse-engineering book in our uni library tomorrow, hopefully if a book is coming from the same angle as me it might help.

Forlorn.

GJD capable

2007-08-27T12:42:00.000+01:00

The video above celebrates the fact that my "engine" can now pull a vdx from a gjd directly and play it (previously, in case you haven't been following, I had to extract the vdx with a command line tool first).

Enough with procrastinating. Now I have no excuse but to attempt some disassembly. Well, I could work out how to do sound / xmidi, but I think I'll save them for later.

Praise Google Desktop!

2007-08-26T22:01:00.000+01:00

So I'm just about to turn off Google Desktop, in case it overwrites the part of the hard disk that the files were stored on.

I mean, how often do I need to search for files on my computer anyway? It's not like...

Hmmm.

It lets me search for strings inside, for example, source code files. Does that mean it stores cached versions of those file?

Bingo!

OK, I don't have them all back, but the only ones I'm missing are the detection files and the makefile, which should be a piece of cake to do again.

I may have to keep google desktop for at least a little while, if only to say thank you.

STUPID

2007-08-26T20:29:00.000+01:00

This could be my last post for a while. I have been exceptionally stupid, and managed to delete my entire engine. I'm not gonna go into details of how I did it, it's far too depressing, but I've tried "recover" and "e2undel", to no avail. I'm now in the process of grepping my entire hard drive (which has to be done in 100MB blocks, otherwise grep runs out of memory) to see if anything is salvagable.

Suffice to say, if I get everything back up and running again, or if I have to start from scratch (my last way-point was my empty engine), I will be setting up a backup system, probably an svn server.

Grrr.

Muppet city

2007-08-26T12:56:00.001+01:00

As expected, the problem was with me being a muppet. When telling it the length of the data (for example, for the initial still image), I was giving it the length of the compressed data, rather than the uncompressed. So it was only getting partway through. Now it works, yay!

Where to go from here. There are a number of options:

Get it to read this vdx from the gjd file
Get it to play back the sounds from this vdx file
Try playing back the xmidi files
Start reing the executable

Although it may seem like I'm procrastinating (hell, of course I am, disassembling still scares me), I'm gonna sort out reading the vdx from the gjd. Everything is stored in gjd files, including the xmidi files, and I think it will be good to get me into the C++ mindset.

I preferred it when it when too fast

2007-08-26T00:26:00.000+01:00

Got the timing working. Problem is that now it's running slowly enough for me to see artifacts. *sigh*. It's easier to fix them when your program outputs every frame as a bmp. Maybe I'll get scummvm doing that...

Moving pictures

2007-08-25T22:32:00.000+01:00

If you can see it at this crap quality... I managed to get the animation frames working. Timing is not adhered to (by any means), but all the frames are in there.

ION, given I have 4 bluetooth devices, 2 of which have video cameras on, you'd think I'd be able to transfer the video easily. No. It was possible, but I had to open the linux app that received files. Would be good to have that built into the bluetooth, or at least the option to start it whenever a bluetooth device is connected. Anyway, done now. The next question is, will this video work?

It looks more impressive than it is...

2007-08-25T20:39:00.000+01:00

OK, nobody get too excited (though I am, but I know what this is). At the moment, this is just reading the first frame of one of the animations. Don't get me wrong, this is a big step, but ... there is a whole 200 storey building to climb after this one :-). But still, yay!

Decompression woes yet again

2007-08-25T00:46:00.000+01:00

Hmmm.
Looking at the LZSS decompresser in the scumm he engine, that works by just allocating big enough chunks of memory to decompress the entire stream. This is all fine and good. But *my* variant of LZSS doesn't tell me how big the decompressed data is, only the compressed data. This, it turns out, is a bit of an arse.
At the moment I'm planning just to say "it's never gonna be bigger than this" as a nasty hack, just to get something working. In the future, when I'm feeling a bit more confident, I'll write an LZSS-compressed stream class (unless someone beats me to it :-).

Finding my feet

2007-08-23T23:47:00.000+01:00

Well, this is something of a non-post. Or at least, it will seem it to anyone that's not me. I'm trying to get my engine to just play a vdx file first, just to find my feet. I'm getting my head round the way everything's done in ScummVM, slowly, but it's not helped by the fact that my C++ is rusty: I wrote the extraction tools in plain C. Might have to get a book out of the uni library tomorrow. Anyway, I have figured out how to draw to the screen, & open and read from a file... but now I've got to decompress a stream of data on the fly. In the tools I wrote before, I extracted to a temporary file, then read that back in. I knew it was a hack, and that I'd have to do it properly, but just sort of ignored it. Thankfully, there is already an LZSS decompresser in the SCUMM engine: it uses different settings so I can't use it, but I should be able to use it as reference.

Man that was a boring post. Maybe next time I'll have some pretty pictures... hopefully.

Emptiness

2007-08-22T20:36:00.000+01:00

Well, I finally managed to get an empty engine compiled into ScummVM, yay! I've not heard back from the ScummVM leads yet, but that's ok: even if it won't be part of ScummVM, the ScummVM app means I don't have to worry about getting my own graphics / sounds setups, etc.

Google

2007-08-21T12:36:00.000+01:00

Hmmm... as pointed out on #scummvm, Google seemed to be put out by me claiming that Googling "t7gre" did not find my blog, and now it very much does. Do they only list blogs that have been posted in recently maybe? I've googled occasionally for this in the past, and it seems rather sporadic whether it appears or not.
Anyway, I've emailed the leads of ScummVM now, to get their input on whether ScummVM could include T7G at some point in the future.

Disassembling number 7

2007-08-21T00:08:00.000+01:00

Well, disassembling doesn't scare me as much as it did. Which is obviously a good thing. I used the free version of IDA, and it looked like I could get some good stuff out of it (bit tired tonight though). The only thing is that it won't look at the dos executable (i.e. the original), because it's only a demo (the full versioin will hack into any executable you care to throw it at, by all accounts). Thankfully there is a windows executable (available from the links on the left), but I can't help feeling that I should be using the DOS one. I imagine it to be simpler, but maybe I'm making that up.

Assembly language

2007-08-20T22:28:00.000+01:00

I don't wanna!

Scuppered!

2007-08-20T21:46:00.000+01:00

Well, tonight I got a mention on #scummvm. To answer the questions that were asked there, the reason I haven't discussed this with the ScummVM team is because I wanted to get a foothold on it before asking. That stage was to come soon enough.

But now I'm intrigued. This blog doesn't even show up (atm) if you google for t7gre... how on earth did it get stumbled upon? Bizarre. Ah well. I just have to assume now that someone might actually be reading this :-)

Another gift!

2007-08-16T20:49:00.000+01:00

It turns out the music, contained in the XMI.GJD file, is in xmidi format. Which scummvm already supports. Woohoo! This says to me that the next stage is probably going to be adding the engine to scummvm ... a big task.

Oh, and I'm going to rename all the files so they have small letters. internally they're all referred to in small letters, so for the time being I may as well be consistent. Eventually I guess it's gonna have to cope with both, but I prefer the idea of being short-sighted for a bit.

And I'm going to redo the database, this time with all the unknown data values, and possibly bytecode totals for all the resources. Maybe. Still haven't decided what format the database should be in, I'm having a go with OpenOffice database, seems ok. Whether it'll read in comma delimited data, who knows...

Building database

2007-08-15T23:40:00.000+01:00

Well, I figured if I was going to get every bit of this game working, I'd need to know which part is where, etc. So I'm unpacking everything. That may not sounds very exciting (and it's not), but I was quite impressed to find that just INTRO.GJD, which starts off at about 30MB decompressed to about 900MB. So the whole game (about 650MB) will probably take up 20GB. And then I just have all the decompression data to put into some sort of usable form...

/me trundles off to investigate MySQL

And I'm spent

2007-08-14T23:54:00.001+01:00

I managed it in the end. I figured out not only what the problem was, but why I had a problem with that frame in particular. The frame uses a particular opcode, 0x60, a lot, when others don't (at all? Not sure). That opcode gives very precise control over the image: where the previous frames have been slightly "rushed" as it were, to save space, this one is done very precisely. If you remember the intro to 7th guest, you'll know why: all the action for the next few scenes takes place on the pages of that book: it basically doesn't change for a minute or more.
But that's not important. The important thing is that it works. Now I've just got to try it on all the others and find the ones it doesn't work on...

Noooooooooooo!

2007-08-14T22:48:00.000+01:00

It was all going so well! I'd finally got the palette completely sorted (how could it be so difficult?), the first 30 frames were perfect, then this! Where do I start trying to figure out what's wrong with this? I just don't know...

Still bad, but getting better

2007-08-14T01:22:00.001+01:00

As you can see, I'm getting there...

My sticking point

2007-08-13T21:41:00.000+01:00

In case anyone is trying to do the same (unlikely, I know), the thing that was confusing me was that the 0x61 opcode, which skips to the end of the line, is also required at the end of each line (like a new line character). As such, you don't need to keep track of the number of tiles in the x direction that have been processed.

Woo hoo!

2007-08-13T21:28:00.000+01:00

Though barely recognizable, this is a breakthrough. This is the house, after we've pulled out to see it in the book (remember that bit?). It may seem like small comfort, but it pretty much just the palette that's screwed. Hopefully.

End of night post

2007-08-13T01:52:00.000+01:00

This is to aid my memory when I come back to this tomorrow. Too late now.

Program segfaults. This is because it goes beyond the end of the image array. This is because it goes beyond the correct number of blocks in y. I do not know why this is, I must have interpreted the file wrong... right?

Progressing

2007-08-13T00:10:00.000+01:00

Next stage. Still not really recognizable (I think it's just gonna be the mansion again, btw).

Not quite there yet

2007-08-13T00:01:00.001+01:00

Was getting bored of writing, so thought I'd post some images. This could be an interesting transformation... I think I've screwed up the palette on this image. I'm gonna be putting it together a few pieces at a time, so we'll see!

Valgrind is teh Magic

2007-08-12T22:02:00.000+01:00

It seems that when your code is shonky (and you have an inkling that it might be), valgrind is where to look. It'll point to things, even when your program appears to be working, and say "err, are you sure that's right? it looks like you're being a bit of a muppet there". And you can ignore it (I did, mostly cos it took me a while to figure out what its error messages meant), but eventually they'll come up and bite you in the seg fault.

In short, I feel it is going to be invaluable. Even more so than it has been tonight.

Don't Panic!

2007-08-12T16:34:00.001+01:00

It's ok, it was a very easy thing to fix. Phew. Now, where was I...

Well, at least I liked Square 1 when I was there first time round...

2007-08-12T16:21:00.000+01:00

There is no expletive to express this.

Up until now, I have been working with the smaller GJD files that get installed to the hard drive, but I couldn't find a delta frame there that had a palette change. So I went for a biggy: the INTRO.GJD file on disc one. It's good I did, because I've discovered that the bit that was easy, the extraction from the GJD file, doesn't work properly. It gets about two into it (out of a couple of dozen), then falls over.

Still, I've already learnt / re-learnt quite a lot since I did that bit, so it might be obvious...

Decompressing V.EXE

2007-08-12T12:50:00.000+01:00

As a distraction from those scary delta frames, I used a combination of unlzexe and dosbox to decompress V.EXE, which was compressed using LZEXE. So now all I've gotta do is disassemble it :-)

Procrastination, that's what you need

2007-08-12T11:45:00.001+01:00

As expected, I'm putting off delving into the delta frames format. As you might expect, it's pretty complicated, so I've been trying to make sure my program is ready for it. I've now got a proper image struct for storing the images, which makes it much easier to have a second one to store the new image. Part of me feels I should be doing this properly in C++ classes though... I'll have to eventually...

Tada!

2007-08-11T23:24:00.000+01:00

And it was just a comparatively small step from there to here!

The video frames (i.e. each one after this, the first one) are stored differently again, I think they're going to be even more tricky to pull out, but at least I know my BMP writer works!

Picture posting

2007-08-11T22:40:00.000+01:00

So I figure, now would be a good time to start posting pictures. Here's the mansion, as my program outputs it atm. Clearly far from right, but it is there if you stand on your head and look hard...

Decompression fixed

2007-08-11T22:33:00.000+01:00

Yes, there was a problem with the decompression. This would have been obvious with a little bit of arithmetic: the output .dat file was 2.5k, and it should have been about 50k. Fixed now. The BMP is still screwed, but in a much less terminal way (I hope :-).

Bugger

2007-08-11T00:27:00.000+01:00

Hmph. I think there is a problem in my decompresser. And after I was so happy that I got it working and everything.

The problem is that I have no intermediate. I can check the vdx files are correct, but after that I have to go:

vdx -> individual chunk -> decompresser -> read image -> write bmp

The only stage where I can actually check that everything is going according to plan is at the stage where it has made me a bmp file. Which is rather frustrating. I'm used to checking at every single stage along the way, to make sure all is going well, but here I don't get that option. Unless I try to take apart the pascal code... no, I think that would be going too far. This will work, it will just take care and hard work.

Ugh...

2007-08-10T20:04:00.000+01:00

And now, the task ahead is to convert my pretty little file into a bmp file, so that I can view it. Which is a decidedly dull task, so I keep putting it off and off.

I emailed the guy who re'ed the vdx and gjd files, and he was supportive, but he only ever concentrated on those files. Looks like I've got all the help anyone's able to give me...

Right. BMP files. *sigh*.

Decompression complete

2007-08-10T01:51:00.000+01:00

Woohoo! It was a challenge, more mentally taxing than I'm used to, but I now understand the algorithm and have it working in my program. At the moment it just dumps the decompressed files to disk rather than converting them into any sort of viewable format, but that should, I hope, be comparatively straight-forward...

Holy compression batman...

2007-08-09T22:58:00.000+01:00

It turns out this is going to be a little more tricky than I thought. Not much, but a little. The video sequences use a form of data compression, LZSS, which I'm gonna have to write a decoder for. Which would be a huge set-back, or at least would slow me down an awful lot, if it weren't for the fact that someone (at the page linked to above) has written a pseudo-code decoder. I say pseudo-code, it's pretty much pascal. I've begun by just directly translating it, but I'm starting to get confused: I think I'm gonna have to figure out how it's working first, then implement it after.

Starting vdx extracter

2007-08-09T18:17:00.000+01:00

Well, I've set off on the vdx file extractor. And although reasonably straightforward, it is definitely NOT going to be a one-night job like the gjd extracter.

One thing about it is that unlike the gjd file, where every byte is accounted for, there are several bytes here and there that the handy specs linked to earlier just say "unknown". Now, they didn't have to worry about them, because they had the sole purpose of extracting resources from the files, but I'm probably going to need what's in them at some stage. So I'm storing them, but I have to come up with variable names for them when I have no idea what they are / what they do. I've decided to follow the theme, and name them (initially at least) after characters in the game. I should point out that I don't know those names off the top of my head: I'm not that obsessed with the game! I looked them up on wikipedia, so if they're wrong no-one can blame me. As if anyone will notice / care :-)

Easier than expected

2007-08-09T00:43:00.000+01:00

Using the ScummVM tools as a base, it was really easy making an extracter for the .GJD files. Admittedly, it's not difficult, it essentially just a bunch of files stuck together, but I had expected to take a while to get going, so it's good to get an easy bit out of the way. Next (but not tonight), the vxd files, which although more tricky, really shouldn't prove too much of a problem.

Hmmm...

2007-08-08T21:32:00.000+01:00

Having not managed to get any of the game extracting things working under linux, I thought I'd write little tools first that would extract the resources from their respective files. But someone beat me to it. And they run perfectly under wine, unlike everything else I've tried this evening.

I think it's still going to be a good exercise to write the tools in C though: it'll get me to know the file formats well before starting anything more complicated. It's just that I'll be translating from Pascal to C rather than writing from scratch!

The long journey begins

2007-08-08T20:28:00.000+01:00

There is some information on the web to get me going: the most helpful is that someone has re-ed the file formats. I've been trying to get MultiEx Commander working under linux, but it doesn't seem to like wine. Which is rather frustrating. There's no way I could do this under windows, so I'll see if I can find anything else to help me out...

Welcome to my... blog

2007-08-08T20:01:00.000+01:00

So this is my attempt to actually get started on this: start a blog, then I'll get really depressed if I don't put anything in it for a while and be forced to do something.

I am intending to reverse-engineer The 7th Guest, the eventual intention being (if they'll have it) to add it to ScummVM.

I should point out that I have no experience of reverse engineering, little experience of ScummVM programming, and only moderate experience of programming. The thing I have in abundance, though, is a willingness to learn these things. Hopefully that will be enough!