Tool for expansion and compression of the Huffman files

Desmet Irkm · Post by **Desmet Irkm** » Wed Dec 07, 2011 11:28 am

I've written a tool in Python now that can expand and compress the Huffman files. If you're interested you can download it from: http://www.file-upload.net/download-393 ... l.zip.html

You should put the files bttool.py and btfile.py into your msdos bt1 directory, then make a backup copy of your files, and then you can try e.g.

Code: Select all

./bttools.py expand fileid=level hex

This will expand the levs file into a bunch of 'levs-00.hex, levs-01.hex, ...' files that you can edit with a normal ascii editor. If you leave out the 'hex' option you'll get binary files 'levs-00.bin, levs-01.bin, ...' that are better if you want to edit programmatically. When you're done just invoke

Code: Select all

./bttools.py compress fileid=level hex

and it will compress all levels again, and put them together into the levs file. To get some help on the program just invoke it with no options at all.

There is also an 'identity' option, which expands and directly recompresses the given file with no compression at all, using an "identity Huffman table". If you use that you can directly edit the Huffman encoded files, as the encoding is one to one.

If you want to see how you can use the API directly from Python, take a look at the files "example_8a.py"-"example_8c.py".

Have fun!

PS: As I go on vacation today, I won't be here for the next 10 days or so. When I'm back, I plan to explain in a short series exactly what's going on in the code.

PPS: There are tools to convert Python scripts directly into exe files (pyexe, pyinstaller,...). I haven't gotten around to do that, but maybe somebody else could do that, so that also the windows people here can use the tool easily?

Desmet Irkm · Post by **Desmet Irkm** » Wed Dec 07, 2011 12:23 pm

Here's some fun I had with the tool. I decoded the city map with

Code: Select all

./bttool.py expand fileid=citymap hex

and reformatted a bit so that it was 30x30. Then messed around a bit, like make a direct passage to the Grand Plaza, put the two Towers directly in front of the Guild (beside the Credits), and some more nonsense. Here's the (modified) hex file

Code: Select all

00 00 00 01 02 03 03 02 01 03 04 01 02 01 03 04 03 02 01 03 03 04 02 01 01 01 01 03 02 01 
01 01 02 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 02 03 00 00 00 04 
01 03 00 00 01 03 02 04 03 04 01 00 01 02 04 03 02 00 02 03 04 03 04 00 00 68 00 91 00 03 
01 04 00 00 60 00 60 00 00 00 02 00 04 00 00 00 01 00 00 00 00 00 00 01 00 02 00 00 00 04 
01 00 00 01 02 04 00 00 21 00 03 00 02 00 01 00 00 00 01 03 04 02 00 02 00 03 04 68 02 01 
03 00 00 04 9B 00 60 00 00 00 02 00 03 00 00 00 00 00 00 00 00 00 00 03 00 02 03 00 03 02 
01 02 00 02 03 01 00 00 01 03 04 00 04 02 01 03 00 01 02 04 00 00 01 04 00 04 01 00 04 01 
01 01 00 00 00 00 60 01 00 00 00 00 00 00 00 04 00 00 00 00 00 01 00 00 00 00 00 00 00 03 
04 00 00 00 00 00 00 02 00 03 01 00 89 02 00 00 04 03 02 01 00 02 00 02 01 00 02 01 04 01 
03 00 02 04 03 02 00 04 00 00 04 00 00 00 04 00 00 00 00 02 00 04 00 2B 02 00 03 00 02 01 
02 00 00 00 00 04 00 00 00 00 02 01 03 00 01 02 03 04 00 00 00 03 00 12 03 00 00 00 00 02 
03 00 04 01 00 02 03 04 03 00 00 12 01 00 21 21 71 71 04 03 02 02 00 00 00 00 1C 00 03 01 
02 00 01 00 00 00 00 00 00 04 00 02 21 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 03 
01 02 03 00 01 00 01 02 00 01 02 04 21 00 00 00 00 00 71 03 00 01 00 01 04 00 94 01 02 01 
A8 00 00 00 02 00 03 04 00 00 00 01 21 00 00 00 00 00 71 04 00 00 00 03 0B 00 81 01 01 01 
01 01 02 60 04 00 00 00 02 03 00 02 21 00 00 00 00 00 71 01 04 01 00 04 02 00 A4 01 03 01 
03 00 04 00 03 02 01 00 00 04 00 00 00 00 00 00 00 00 71 02 03 03 00 02 00 00 00 00 00 02 
04 00 02 00 00 00 00 04 00 01 02 04 03 21 21 21 21 00 04 00 00 02 00 01 00 00 01 02 04 01 
04 00 00 00 04 02 00 03 00 00 00 01 02 03 04 02 03 00 02 00 00 00 00 00 04 00 00 00 00 02 
03 00 00 00 00 00 00 02 00 02 00 00 00 01 03 00 00 00 01 00 01 03 02 00 00 03 02 00 03 00 
00 02 03 00 01 02 00 01 00 00 03 02 00 04 02 00 02 03 04 00 00 00 00 04 00 01 71 00 00 03 
00 12 02 00 04 03 00 01 00 00 01 04 00 00 00 00 00 00 00 01 00 00 01 02 00 00 03 00 04 00 
01 00 00 00 00 04 00 03 00 00 01 02 04 03 02 03 04 01 04 01 02 12 04 00 04 00 01 00 00 01 
00 01 03 60 01 02 60 04 00 00 00 00 00 01 02 00 00 00 00 03 12 02 00 00 01 00 02 00 12 00 
00 04 00 00 03 00 00 02 00 00 01 02 00 00 00 00 02 04 00 00 00 00 00 00 02 00 01 00 12 00 
00 02 68 01 00 03 00 03 00 00 04 02 03 04 04 03 00 00 00 00 00 00 00 02 03 00 04 00 12 00 
01 00 00 00 04 00 00 04 00 00 00 00 00 00 00 00 00 00 02 00 00 21 60 00 01 00 03 00 12 01 
02 00 A1 00 68 00 02 03 02 03 02 01 04 03 02 04 01 00 00 00 00 60 00 00 04 00 01 00 00 03 
03 78 00 00 04 03 00 00 00 00 00 00 00 00 00 00 00 02 01 04 12 04 00 00 01 00 02 04 03 00 
04 03 02 01 01 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 01 04 03 02 00 01 00 00 00

Then I compressed it again with

Code: Select all

./bttool.py compress fileid=citymap hex

(you can also you put the hex stuff directly into 'city.pat.hex' and compress it with the command above)

The nice thing is: you don't even have to exit BT to do the editing. I have a editor open, save the stuff, run the compression, enter and exit the guild and can directly see the result. Lots of fun, really

Caracas · Post by **Caracas** » Wed Dec 07, 2011 1:16 pm

Hi Desmet Irkm

I was able to expand the citymap file and modify it, but was unable to compress it again.
I'm getting: ValueError: non-hexadecimal number found in fromhex() arg at position 60

also, in you example of the citymap, you used 94 and A4 for the 2 towers.
Shouldn't that be 91 and A1?

Desmet Irkm · Post by **Desmet Irkm** » Wed Dec 07, 2011 2:04 pm

Caracas wrote: I was able to expand the citymap file and modify it, but was unable to compress it again.
I'm getting: ValueError: non-hexadecimal number found in fromhex() arg at position 60

You probably use a dos/windows editor, which uses different line feed characters than unix systems. Unix uses only '0x0A' while dos/win uses two characters '0x0A 0x0D'. As I'm working on unix I forgot to remove the 0x0D from the file as well, before converting from hex. If you go to the line

Code: Select all

byte_arr = byte_arr.translate(bytearray(range(256)), " \n")

in 'btfile.py' and change it to

Code: Select all

byte_arr = byte_arr.translate(bytearray(range(256)), " \n\r\t")

then also dos/win linefeeds and tabs are ignored when converting. I will put that into the next release, but it would be nice to know if that works for you.

Caracas wrote: also, in you example of the citymap, you used 94 and A4 for the 2 towers.
Shouldn't that be 91 and A1?

Every byte in the citymap uses bits 3-7 for anything special and bits 0-2 to specify the house type. So, e.g. for Mangars tower the special bits are 0xA1>>3=20 and the house type is 0xA1 & 0x07 = 1 (white with thatched roof). Now, I just changed the lower three bits to 4, making the code for "my" Mangars tower 20<<3 | 4=0xA4. I think the stony building fits a tower much better than the regular thatched roof building.

Caracas · Post by **Caracas** » Wed Dec 07, 2011 2:16 pm

Works like a charm now, thanks

Desmet Irkm · Post by **Desmet Irkm** » Wed Dec 07, 2011 2:28 pm

Good. I uploaded the new version too, and changed the download link in my first post.

Caracas · Post by **Caracas** » Wed Dec 07, 2011 3:10 pm

Desmet Irkm wrote:Every byte in the citymap uses bits 3-7 for anything special and bits 0-2 to specify the house type. So, e.g. for Mangars tower the special bits are 0xA1>>3=20 and the house type is 0xA1 & 0x07 = 1 (white with thatched roof). Now, I just changed the lower three bits to 4, making the code for "my" Mangars tower 20<<3 | 4=0xA4. I think the stony building fits a tower much better than the regular thatched roof building.

Heh, didn't know about this until now, so looked into it a bit further...
The guild uses 0B on the map, this is 00001011 binary. meaning 00001 for the event and 011 for the house type.
Inns use 12 on the map, 00010010 binary, meaning 00010 for the event and 010 for the house type.
and so on...
I checked them and found following events
00001, 00010, 00011, 00100, 00101, 01100, 01101, 01110, 01111, 10000, 10001, 10010, 10011, 10100, 10101

So I tried adding a code that's not in this list: 00110 and 00111
I added them to the map, rebuild the city.pat file and when I entered the square where I added the events, I got the message:
I don't know code == 6 (for 00110)
I don't know code == 7 (for 00111)

Caracas · Post by **Caracas** » Thu Dec 08, 2011 10:04 am

I've been checking the code in the LEVS file for the special events.

Each level can have up to 8 special events. In the LEVS file there are 16 bytes available for coordinates of these special events for each level -- 2 bytes per coordinate (x,y), followed by a specific event code.
I've been looking at Harkyn's castle level 3: There are 3 special events:

Code: Select all

05 0C 01 15 05 03 FF FF FF FF FF FF FF FF FF FF 
1F FF 20 FF 21 FF FF FF FF FF FF FF FF FF FF FF

05 0C are the coordinates for the 4x99 berserkers at 5N, 12E
01 15 are the coordinated for the Mad God statue at 1N, 21E
05 03 are the coordinates for an old man with a riddle at 5N, 3E

The corresponding specail events code are:
1F for the coordinates 05 0C (berserkers)
20 for the coordinates 01 15 (Mad God)
21 for the coordinates 05 03 (Old man with riddle)

It is possible to switch these codes around... If you switch 1F and 20 around, then the Mad God will be at 5N, 12E and the berserkers will be at 1N, 21E
Also, you can switch the codes with codes from different dungeons. I changed 1F into 10 and long stairs leading up appeared where the berserkers should be (stairs from sewers lvl3 to Mangar's tower).
I was also able to place 4x99 berserkers next to the stairs in the catacombs lvl 1.

This bttool.py tool is extremely useful for fast rebuilding the LEVS file.
These are all the codes in the LEVS file:

Code: Select all

11: 9N,4E Sewers lvl1
12: 4N,0E Sewers lvl2
13: 4N,3E Sewers lvl2
10: 10N,11E sewers lvl3
14: 12N,10E catacombs lvl2
15: 13N,6E catacombs lvl2
16: 19N,20E catacombs lvl3
18: 14N,9E Harkyn's castle lvl1
19: 18N,9E Harkyn's castle lvl1
1A: 6N,16E Harkyn's castle lvl1
1B: 0N,19E Harkyn's castle lvl1
1C: 9N,9E Harkyn's castle lvl2
1D: 0N,0E Harkyn's castle lvl2
1E: 0N,19E Harkyn's castle lvl2
1F: 5N,12E Harkyn's castle lvl3
20: 1N,21E Harkyn's castle lvl3
21: 5N,3E Harkyn's castle lvl3
25: 10N,13E Kylearan's tower
26: 2N,12E Kylearan's tower
27: 20N,2E Kylearan's tower
28: 1N,4E Kylearan's tower
29: 13N,17E Kylearan's tower
2B: 13N,20E Mangar's tower lvl1
2C: 15N,4E Mangar's tower lvl2
2D: 12N,19E Mangar's tower lvl3
2E: 4N,10E Mangar's tower lvl3
2F: 8N,12E Mangar's tower lvl4
30: 8N,21E Mangar's tower lvl4
31: 9N,20E Mangar's tower lvl4
32: 20N,3E Mangar's tower lvl4
33: 15N,10E Mangar's tower lvl5
34: 21N,10E Mangar's tower lvl5
35: 8N,10E Mangar's tower lvl5
36: 10N,21E Mangar's tower lvl5
37: 17N,0E Mangar's tower lvl5
38: 1N,10E Mangar's tower lvl5
39: 1N,20E Mangar's tower lvl5
3A: 20N,10E Mangar's tower lvl5

ZeroZero · Post by **ZeroZero** » Thu Dec 08, 2011 4:20 pm

@ Caracas

The level file format is exactly like on the C64 version.
It is fully described in that thread. I also have made a
viewer for the levels files for the C64 version (coz it
reads directly from a D64 disk image).

Source code in VB6 can be provided.

Caracas · Post by **Caracas** » Fri Dec 09, 2011 8:44 am

Yes, you're right. I'm sorry, I should have mentioned it.
It's just that it's the first time that I'm realy looking into the LEVS file.

As in the c64 version, there are 8 possible forced encounters.
For Catacombs lvl 3, I can find in the LEVS file:

Code: Select all

01 00 03 09 04 0D 07 11 08 05 10 0D 12 0E 15 00 
3C 07 33 24 1E 45 0A 63 0A 63 14 42 14 35 3C 08

First 16 bytes are coordinates for the forced encounter. the next 16 bytes are the type of monster and how much there are.

so at 01,00 (1N, 0E), we can find monster 3C (Wraiths) and there are 07 of them.
At 03,09, we find monster 33 (Ghoul) and there are 36 of them ($24)
and so on...

This means that for each level, there is only a maximum of 8 forced encounters... so flooding the levels with 99 Mangars on each square will not be possible.
It seems there can only be 1 type of monster in each forced encounter. Events like 4x99 berserkers will not be possible to turn into a forced encounter.

I wonder if it would be possible to put Mangar in the Wine Cellar, and what level he would be then and what treasure he would give...

EDIT: ok, just tested it. Existing forced encounters can be modified at will. If you want to create additional ones (up to a maximum of 8 ) you need to add a 04 on the event map where you want to put a new forced encounter.
If you want to relocate an existing forced encounter, you also have to make sure to put a 04 on the new location.
I just killed Mangar twice in the Wine Cellar. he gave me about 1100 exp and some gold each time. I didn't recieve any item.
First time he summoned a Storm Giant which my party disbelieved, second time he summoned a Demon Lord who stoned one of my chars (I was with a group of ungeared lvl 40 chars)

EDIT2: interestingly, you're not limited to 99 monster per forced encounter. You can go up to 255 (FF) monsters per forced encounter... 255 Mangars... imagine that!

Flanimal · Post by **Flanimal** » Sat Dec 10, 2011 12:50 am

Would 255 Vampire Lords drain one's level down to 1, and perhaps beyond?

Maven · Post by **Maven** » Sat Dec 10, 2011 3:46 am

I think you might have problems if you put more than 99 mobs in a forced encounter. The HP arrays are only 99 bytes long. Although, since you can only use one group, and the HP arrays are contiguous, it would only overwrite the next two unused HP arrays, so it might be OK. It might screw up the already bugged targeting algorithm.

Don't know about Vampire Lords specifically, but I can say from personal experience that Specters can only drain down to level 1. I expect Vampire Lords would work the same way.

Caracas · Post by **Caracas** » Thu Dec 15, 2011 1:05 pm

Creating forced encounters with 255 monsters didn't give me any issues. I created a group of 255 skeletons and let my wizard(s) use DRBR, REDE, my bard used a horn on them, my pet dragon sneezed on them, my monk smashed bones.. all without issues

Maven · Post by **Maven** » Fri Dec 16, 2011 6:25 pm

That's great. Makes for some interesting possibilities for custom-made dungeons. Which this tool makes pretty easy to do, right?

I understand there was supposed to be a forced encounter on Mangar's level with 99 Balrogs or something. I wonder what level your characters would have to be to survive an encounter with 255 Demon Lords. Or Old Men.

Is anyone working on a custom levs file with their own dungeons, using standard specials?

Caracas · Post by **Caracas** » Sat Dec 17, 2011 12:33 am

Would be fun to recreate some places from other fantasy worlds... Like a level with only a few rows of pillars like in Moria... with a balrog encounter somewhere

Or a chamber with pools of lava and a bridge, like the bridge of Khazad-dum
There's a pool of lava on the 5th floor of Mangars tower.

In theory, you could change the picture of Kylearans tower and turn it into a 1 level city park if you also change one of the DPICS file

just thinking out loud