Slotted Palettes test and the PAL64 palette format

[System64]

Hi! I did some tests with the new palette attribute per tile, and it is a success! I linked a video to show the test.
[video=discord]https://cdn.discordapp.com/attachments/808677096788328448/1040005797654056990/2022-11-09_21-47-10.mp4[/video]

I also used my custom palette format, PAL64 generated with my tool here : PAL64 Palette Editor
The reason why I made this format is because loading 8 differents ACT files is tedious. Plus, my tool allows you to use old school palettes such as the Master System Palette, or even the NES palette! It also supports the common RGB24 format.

Don't hesitate to ask about the format specification or about the code if you are interested. Caution, the code is written in Nim, but it should be straightforward.

[vonhoff]

Hey, this looks really cool! I would love to implement something like this in my own projects. Can you share the code for loading palettes like this?

I don't mind that it's written in Nim, I think it should be easy to port to other languages.

[System64]

Hey, this looks really cool! I would love to implement something like this in my own projects. Can you share the code for loading palettes like this?

I don't mind that it's written in Nim, I think it should be easy to port to other languages.

Hi! Thanks for your interest!
First, here is the file format (the screenshot of the binary)

- The 5 firsts bytes in blue define the header, it's just the string "PAL64"

- The most significant bit in the pink byte tells if the custom NES palette is embedded in the PAL64. 1 means True, 0 means False. The others bits tell which version of the format it is.

- The green byte tells how long are the palettes (up to 255, but limited to 16 in the tool)

- The yellow byte tells how many palettes there is (up to 255, but limited to 16 in the tool)

- The purple byte tells which palette type it is (this way, you also can guess how many bytes a color use)

- The red bytes are the data of the palettes. The 3-3-2 RGB, SMS and NES colors are encoded in one byte (so one byte = one color). 5 levels, WEB, MegaDrive, Amiga and SNES palettes are encoded in 2 bytes (so 2 bytes = one color). The DS and RGB24 palettes are encoded in 3 bytes (so 3 bytes = one color)

- The orange bytes represents the embedded NES palette. The custom NES palette can have an arbitrary number of colors, up to 256. Those bytes are there if you tell the tool to embed the NES Palette in the PAL64 file. The embedded NES palette is loaded into the internal NES lookup table.

Here is some code!

Code:

import Tilengine, Palette
import std/streams
import strutils

const PAL64VER = 1.uint8

type
    paletteType = enum
        RGB332,
        SMS,
        FIVELEVELS,
        WEB,
        MEGADRIVE,
        AMIGA,
        SNES,
        DS,
        RGB24,
        NES

var nesLUT = @[5592405.uint32, 6003, 1926, 3016056, 5833293, 7471121, 7208960, 4982784, 1514240, 10752, 12544, 11784, 9797, 0, 0, 0, 10855845, 22470, 2244581, 7219417, 11410086, 13768537, 13705479, 10958592, 6508800, 1599232, 29184, 29489, 27268, 0, 0, 0, 16711679, 3123455, 6128127, 10252543, 16216831, 16742333, 16744053, 16747051, 13475840, 8501250, 4048944, 1232251, 902608, 3947580, 0, 0, 16711679, 10804991, 11651327, 13418239, 16040703, 16762346, 16762825, 16764330, 15718038, 13688981, 11790245, 10480323, 10152166, 11513775, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0]

proc loadNESPal*(path: string): void =
    let
        file = open(path)
        fSize = file.getFileSize()
        fs = newFileStream(file)
    defer: fs.close()
    var data = newSeq[uint8](fSize)
    discard fs.readData(data[0].addr, sizeof(uint8) * data.len)

    for i in countup(0, nesLUT.len-1):
        if(i >= data.len div 3):
            var j = i * 3
            var msb = data[j]
            var mid = data[j + 1]
            var lsb = data[j + 2]
            var colVal = (msb.uint32 shl 16) or (mid.uint32 shl 8) or lsb.uint32
            nesLUT[i] = colVal
        else:
            nesLUT[i] = 0

proc loadHEXtoNES*(path: string): void =
    let
        file = open(path)
    defer: file.close()
    for i in countup(0, nesLUT.len-1):
        if not file.endOfFile():
            nesLUT[i] = parseHexInt(file.readLine()).uint32
        else:
            nesLUT[i] = 0

# Converts to RGB24
proc bin2rgb(bin: uint32, palType: paletteType): uint32 =
    var
        r: uint32 = 0
        g: uint32 = 0
        b: uint32 = 0
    case palType
        of RGB332: # RGB 3-3-2
            # echo "test"
            r = ((bin shr 5 and 0b00000111).float32 * 36.42).uint32
            g = ((bin shr 2 and 0b00000111).float32 * 36.42).uint32
            b = (bin and 0b00000011) * 85
        of SMS: # SMS
            r = (bin shr 4 and 0b00000011) * 85
            g = (bin shr 2 and 0b00000011) * 85
            b = (bin and 0b00000011) * 85
        of FIVELEVELS: # 5 Levels
            r = (bin shr 6 and 0b0000000000000111) * 63
            g = (bin shr 3 and 0b0000000000000111) * 63
            b = (bin and 0b0000000000000111) * 63
        of WEB: # WEB
            r = (bin shr 6 and 0b0000000000000111) * 51
            g = (bin shr 3 and 0b0000000000000111) * 51
            b = (bin and 0b0000000000000111) * 51
        of MEGADRIVE: # MegaDrive
            r = (bin shr 6 and 0b0000000000000111) * 36
            g = (bin shr 3 and 0b0000000000000111) * 36
            b = (bin and 0b0000000000000111) * 36
        of AMIGA: # Amiga
            r = (bin shr 8 and 0b0000000000001111)
            # echo bin.toHex()
            r = r or (r shl 4)
            g = (bin shr 4 and 0b0000000000001111)
            g = g or (g shl 4)
            b = (bin and 0b0000000000001111)
            b = b or (b shl 4)
        of SNES: # SNES
            r = (bin shr 10 and 0b0000000000011111) * 8
            g = (bin shr 5 and 0b0000000000011111) * 8
            b = (bin and 0b0000000000011111) * 8
        of DS:
            r = (bin shr 12 and 0b111111) * 4
            g = (bin shr 6 and 0b111111) * 4
            b = (bin and 0b111111) * 4
        of RGB24:
            r = bin shr 16 and 0xFF
            g = bin shr 8 and 0xFF
            b = bin and 0xFF
        of NES:
            return nesLUT[bin].uint32
        else:
            echo "Not supported!"
            return 0
    return ((r shl 16) or (g shl 8) or b)

# Loads the PAL64 and sets into Tilengine's global palettes
proc loadPalsAndSet*(path: string, importEmbeddedNes: bool = false): void =
    let
        file = open(path)
        fSize = file.getFileSize()
        fs = newFileStream(file)
    defer: fs.close()

    # Data of the palettes
    var data = newSeq[uint8](fSize)
   
    # Reading the data of file and put it into the data buffer
    var i = fs.readData(data[0].addr, sizeof(uint8) * data.len)

    # Header and version checks
    var str = data[0].char & data[1].char & data[2].char & data[3].char & data[4].char
    var index = 5
    if(str != "PAL64"):
        echo "Invalid PAL64 format!!"
        return
    var myVer = data[index] and 0b01111111
    if(myVer > PAL64VER):
        echo "PAL64 version too new!!!"
        return

    # Is the NES palette embedded in the file?
    var isNesEmbedded = data[index] shr 7
    inc index

    # Length of the palettes
    let len = data[index]
    inc index

    # Number of palettes
    var numPal = data[index]

    # Tilengine supports up to 8 palettes. So we clip this number.
    if(numPal > 8):
        numPal = 8

    # Global palettes initialization and setup
    var pal0 = createPalette(len.int)
    var pal1 = createPalette(len.int)
    var pal2 = createPalette(len.int)
    var pal3 = createPalette(len.int)
    var pal4 = createPalette(len.int)
    var pal5 = createPalette(len.int)
    var pal6 = createPalette(len.int)
    var pal7 = createPalette(len.int)
    discard setGlobalPal(0, pal0)
    discard setGlobalPal(1, pal1)
    discard setGlobalPal(2, pal2)
    discard setGlobalPal(3, pal3)
    discard setGlobalPal(4, pal4)
    discard setGlobalPal(5, pal5)
    discard setGlobalPal(6, pal6)
    discard setGlobalPal(7, pal7)

    index.inc

    # Getting the type of the palette (RGB 3-3-2, SMS, WEB Safe, MegaDrive, Amiga or SNES)
    let palType = data[index].paletteType

    index.inc

    # To know how much bytes a color uses.
    var bytes = 1
    var bgColor = 0.uint32

    # Applying the palettes
    for i in countup(0, (len.int * numPal.int)-1):
        var indexCol = i.uint32 mod (len)
        var palId = i div len.int
        var color: uint32 = 0
        if(palType == RGB332 or palType == SMS or palType == NES):
            color = bin2rgb(data[index + i].uint32, palType)
            bgColor = bin2rgb(data[index].uint32, palType)
            bytes = 1
        elif(palType != DS and palType != RGB24):
            var j = i * 2
            var msb = data[index + j]
            var lsb = data[index + j + 1]
            color = bin2rgb((msb.uint32 shl 8) or lsb.uint32, palType)
            bgColor = bin2rgb((data[index].uint32 shl 8) or data[index + 1].uint32, palType)
            bytes = 2
        else:
            var j = i * 3
            var msb = data[index + j]
            var mid = data[index + j + 1]
            var lsb = data[index + j + 2]
            var colVal = (msb.uint32 shl 16) or (mid.uint32 shl 8) or lsb.uint32
            color = bin2rgb(colVal, palType)
            bgColor = bin2rgb((data[index].uint32 shl 16) or (data[index + 1].uint32 shl 8) or data[index + 2], palType)
            bytes = 3
        discard getGlobalPal(palId).setPaletteColor(indexCol.int, color)

    index = index + (len.int * bytes * 8.int)
   
    # We load the embedded NES palette into the Lookup Table if the user wants and if it is a NES palette type
    if(isNesEmbedded.bool and palType == NES and importEmbeddedNes):
        for i in countup(0, nesLUT.len-1):
            var k = i * 3
            var msb = data[index + k] shl 16;
            var mid = (data[index + k + 1]) shl 8;
            var lsb = data[index + k + 2];
            nesLUT[i] = msb or mid or lsb
    setBgColor(bgColor)

Edit : if you see something like "myVariable.someType", I'm just casting a type to another type, it's the equivalent of "(someType)myVariable" in other languages.

[megamarc]

Hi!

Seems you've been busy playing with the palettes :-) thanks for sharing a video showing the process and result. May I upload it to Tilengine youtube channel with proper attribution to you?

I've been looking the documentation and source code of your PAL64 format. I didn't know Nim, but it's fairly straightforward to read. I have some questions:

1. The type of palette and number of bytes are stored in offset bytes 8 and 9, after the yellow byte with the number of palettes? I can see it in source code, but the colored map states these bytes as generic "palette data" without specific meaning.
   
2. inc index and index.inc are equivalent? Both forms are used in your source
   
3. Why there's speciffic support for built-in NES palette? Wouldn't it be more general purpose to just let the user embed the palettes he/she wants, as the format already allows it? What is the use case for this specific feature?
   

Regards,

[System64]

Hi!

Seems you've been busy playing with the palettes :-) thanks for sharing a video showing the process and result. May I upload it to Tilengine youtube channel with proper attribution to you?

I've been looking the documentation and source code of your PAL64 format. I didn't know Nim, but it's fairly straightforward to read. I have some questions:

1. The type of palette and number of bytes are stored in offset bytes 8 and 9, after the yellow byte with the number of palettes? I can see it in source code, but the colored map states these bytes as generic "palette data" without specific meaning.
   
2. inc index and index.inc are equivalent? Both forms are used in your source
   
3. Why there's speciffic support for built-in NES palette? Wouldn't it be more general purpose to just let the user embed the palettes he/she wants, as the format already allows it? What is the use case for this specific feature?
   

Regards,

Hi! You are welcome to upload it! It would be a nice demo!
About your questions :
1) Whooops! I did a small error! effectively, the palette type is represented by the byte at Offset 8. The palette data starts at 9. I'll update the data map.

2) Yeah, they are equivalent. By the way, the keyword "discard" means you don't want to use the value returned by a function.

3) At the begining, I didn't intended to allow the user to use another palette than NES palettes. I implemented this feature because there isn't an unique NES palette. But I noticed there isn't any checks that verify if the palette the user is loading is a NES palette. So it was a kind of bug, but I decided to turn it into a feature. I also wanted to implement more palettes such as Commodore 64 palette, Amstrad CPC and so on, but it's useless with this feature, since the user can load an arbitrary color up to 256 colors. It's less efforts for me, more flexibility for the user, so it's Win-Win. I can probably rename "NES" to "Custom" so it reflects the feature in a better way. The NES palette is just the default palette if nothing is loaded or if the user don't want to load the custom palette.

Edit : I fixed the picture of the map. By the way, I also have 2 suggestions : Getters to get X pos and Y pos of a layer, and getter to get a palette from one of the 8 global palettes (like TLN_Palette myPal = TLN_GetGlobalPal(palId)).
Also, I learned some time ago you are working on a Javascript port of Tilengine. How is it going? I think I can open a lot of opportunites for the web!

[megamarc]

Hi!

Thanks for permission to use your video :-) How may I credit you?Link to  GitHub account, username...?

Thanks for clarifications about palette format and motivations for NES palette. I agree that it would be better to call "custom" palette instead of NES palette, I think this naming is too specific and may create confusion. Be careful however with the format: palette data (red portion) starts at an odd offset. On ARM you can't do unaligned memory reads, that will trigger a CPU runtime exception. Intel x86 allows it, but with performance penalty. So it's a good practice to put data in a boundary aligned to the size of expected reads. In this case, it's expected that client code will do 16-bit (2 byte) reads in case of SNES/Amiga/Genesis palettes, so palette data should start in an offset that is multiple of 2.

By the way, I just pushed to github release 2.13.2 that adds getters for layer and sprite positions, and global palette with your suggestions:

https://github.com/megamarc/Tilengine/co...94107ea236

[System64]

Hi!

Thanks for permission to use your video :-) How may I credit you?Link to  GitHub account, username...?

Thanks for clarifications about palette format and motivations for NES palette. I agree that it would be better to call "custom" palette instead of NES palette, I think this naming is too specific and may create confusion. Be careful however with the format: palette data (red portion) starts at an odd offset. On ARM you can't do unaligned memory reads, that will trigger a CPU runtime exception. Intel x86 allows it, but with performance penalty. So it's a good practice to put data in a boundary aligned to the size of expected reads. In this case, it's expected that client code will do 16-bit (2 byte) reads in case of SNES/Amiga/Genesis palettes, so palette data should start in an offset that is multiple of 2.

By the way, I just pushed to github release 2.13.2 that adds getters for layer and sprite positions, and global palette with your suggestions:

https://github.com/megamarc/Tilengine/co...94107ea236

Hi!
You are welcome! Here is my github : https://github.com/system64MC
About the format, is it still a problem if I read it byte by byte? This is what I'm doing to avoid endianess problems.
Thanks for the update! I will update my builds!

Edit : I also store the files into a buffer of bytes

[megamarc]

Hi!

Reading byte by byte is not a problem, as you can read a byte on any boundary. Just personal experience, some years ago I wrote a FAT filesystem driver for an embedded device that was a bit of a headache. The FAT table has lots of word values (i.e. two-byte fields) that are unaligned. Trying to read those words on that device (a Renesas H8S) on unaligned boundaries caused CPU faults. I had to read all those unaligned words as two individual bytes, and then reconstruct the word value merging the two bytes. A simple word read should be much more elegant.

On Intel CPUs you won't have any problem. But if your format ever goes to ARM (cell phones, etc) developers writing a PAL64 handler will face these kind of problems:
https://developer.arm.com/documentation/ka003038/latest

So for portability and ease of implementation, I recommend you to place data fields on aligned boundaries relative to their natural data size

[System64]

Hi!

Reading byte by byte is not a problem, as you can read a byte on any boundary. Just personal experience, some years ago I wrote a FAT filesystem driver for an embedded device that was a bit of a headache. The FAT table has lots of word values (i.e. two-byte fields) that are unaligned. Trying to read those words on that device (a Renesas H8S) on unaligned boundaries caused CPU faults. I had to read all those unaligned words as two individual bytes, and then reconstruct the word value merging the two bytes. A simple word read should be much more elegant.

On Intel CPUs you won't have any problem. But if your format ever goes to ARM (cell phones, etc) developers writing a PAL64 handler will face these kind of problems:
https://developer.arm.com/documentation/ka003038/latest

So for portability and ease of implementation, I recommend you to place data fields on aligned boundaries relative to their natural data size

Oh alright, thanks for your advice! I'll try to rework the format and do some tests on an ARM device or emulator.