Layer Masking

[sputnikus]

Hello!

I was wondering if Tilengine has a layer masking feature similar to TLN_SetLayerClip, but where pixels would be clipped based on the assigned layer mask rather than rectangular bounds.

At a high level, I was thinking that any given layer could be assigned a "layer mask," and when drawing pixels, we could test against this assigned layer mask to see if the pixel should be drawn or not.

I'm working on adding shadows and overlay effects to my game, and many of the effects I'm considering would require masking parts of a layer to ensure they are drawn within the bounds of the sprites on the layer below.

Here is a video recording of the effects I am working with, I would essentially like to clip all of the shadows within the chessboard's bounds:


Thank you!

[megamarc]

Hi!

Short answer: the feature of per-pixel masking of sprites isn't available. But I can suggest a method to get the effect you want with currently available features.

Tilengine renders elements in the following order, using painter's algorithm. Objects drawn first get overdrawn by objects drawn later.

1. Regular layers
2. Regular sprites
3. Layers (or individual tiles) with priority flag set
4. Sprites with priority flag set

In your case, what you want is:

1. A regular layer with the checkerboard
2. Regular sprites for the piece shadows
3. A high priority layer with the the black, outside part of the checkerboard that will cover only the shadows that lie outside of the board
4. High priority sprites for the chess pieces, that won't be covered by the outside black background.

Check the funcions TLN_SetLayerPriority and TLN_EnableSpriteFlag with FLAG_PRIORITY

With this approach you will have the shadows always below the chess pieces and hidden outside of the board, but the pieces themselves will stand out outside of the board.

Let me know if this works for you!

[System64]

Hello!

I was wondering if Tilengine has a layer masking feature similar to TLN_SetLayerClip, but where pixels would be clipped based on the assigned layer mask rather than rectangular bounds.

At a high level, I was thinking that any given layer could be assigned a "layer mask," and when drawing pixels, we could test against this assigned layer mask to see if the pixel should be drawn or not.

I'm working on adding shadows and overlay effects to my game, and many of the effects I'm considering would require masking parts of a layer to ensure they are drawn within the bounds of the sprites on the layer below.

Here is a video recording of the effects I am working with, I would essentially like to clip all of the shadows within the chessboard's bounds:


Thank you!

Oh you made this with Tilengine? This is very impressive!

[sputnikus]

In your case, what you want is:

1. A regular layer with the checkerboard
2. Regular sprites for the piece shadows
3. A high priority layer with the the black, outside part of the checkerboard that will cover only the shadows that lie outside of the board
4. High priority sprites for the chess pieces, that won't be covered by the outside black background.

Check the funcions TLN_SetLayerPriority and TLN_EnableSpriteFlag with FLAG_PRIORITY

With this approach you will have the shadows always below the chess pieces and hidden outside of the board, but the pieces themselves will stand out outside of the board.

Your suggestion does sound like it would work with the current setup. However, the black background is only temporary. Once additional background layers are added, wouldn’t those new background layers also need a cutout in the shape of the checkered board? If that’s the case, it sounds like another use case for layer masking, where the background layers could use the checkered board as a mask or cutout.

There’s also another issue: the checkered board pattern is not a static image; it’s part of a larger spritesheet designed to create the illusion of 3D rotations:


I had another solution in mind if layer masks were not available. Let me know if this makes sense: Since all of the shadows are relatively simple ellipse shapes, I was thinking of using math to manually draw them onto a bitmap and then test against the checkered board to determine whether a pixel should be drawn or not.

This approach may start to fall apart for different visual effects I’m considering, but it might be good enough for now.

Oh you made this with Tilengine? This is very impressive!

Thank you!

[System64]

In your case, what you want is:

1. A regular layer with the checkerboard
2. Regular sprites for the piece shadows
3. A high priority layer with the the black, outside part of the checkerboard that will cover only the shadows that lie outside of the board
4. High priority sprites for the chess pieces, that won't be covered by the outside black background.

Check the funcions TLN_SetLayerPriority and TLN_EnableSpriteFlag with FLAG_PRIORITY

With this approach you will have the shadows always below the chess pieces and hidden outside of the board, but the pieces themselves will stand out outside of the board.

Your suggestion does sound like it would work with the current setup. However, the black background is only temporary. Once additional background layers are added, wouldn’t those new background layers also need a cutout in the shape of the checkered board? If that’s the case, it sounds like another use case for layer masking, where the background layers could use the checkered board as a mask or cutout.

There’s also another issue: the checkered board pattern is not a static image; it’s part of a larger spritesheet designed to create the illusion of 3D rotations:


I had another solution in mind if layer masks were not available. Let me know if this makes sense: Since all of the shadows are relatively simple ellipse shapes, I was thinking of using math to manually draw them onto a bitmap and then test against the checkered board to determine whether a pixel should be drawn or not.

This approach may start to fall apart for different visual effects I’m considering, but it might be good enough for now.

Oh you made this with Tilengine? This is very impressive!

Thank you!

You are welcome! I can also see Dear ImGUI running (I absolutely love this library, I wrote an entire wavetable generator with it!), which rendering backend are you using? The built-in one? SDL2? GLFW?

[megamarc]

Hi!

Let's face challenges one by one. Tilengine is a 2D graphics library that emulates -and greatly expands- the inner workings of 2D video chipsets. For faking 3D effects, one must be creative and work around individual limitations, instead of trying to find one global approach that fits all.

As I can see, the flying shadows only go outside the board during startup animation, that's the moment where you need to overlay the background. Even if you want to float pieces during gameplay, you can overlay it. The rotation/perspective change where you change the prerendered board background is not interactive, all pieces have the shadows inside the board and the player can't do anything. During the transition, you can safely get rid of the background overlay, there's nothing to cover outside the board.

What do you think?

Sprite testing against a destination bitmap mask -not against a layer- could be a cool feature to implement, but with the uses cases you suggested I think it's not required yet. No classic 2D chipsets had such feature.

What other visual effects are you considering?

[sputnikus]

As I can see, the flying shadows only go outside the board during startup animation, that's the moment where you need to overlay the background. Even if you want to float pieces during gameplay, you can overlay it. The rotation/perspective change where you change the prerendered board background is not interactive, all pieces have the shadows inside the board and the player can't do anything. During the transition, you can safely get rid of the background overlay, there's nothing to cover outside the board.

Hmm, it is true that the shadows only go outside the chessboard's boundaries during spawning; otherwise, the chess pieces remain on the board during rotations. In that case, I think TLN_SetLayerClip would be sufficient to clip most of the visible shadows out of sight.

What other visual effects are you considering?

Another effect I was thinking about is using scrolling, repeating patterns over sprites to indicate the capturing of pieces, similar to this:

You are welcome! I can also see Dear ImGUI running (I absolutely love this library, I wrote an entire wavetable generator with it!), which rendering backend are you using? The built-in one? SDL2? GLFW?

I'm currently using SDL2 for testing, but I plan to target mobile platforms in the future, so that is likely to change.

[megamarc]

Hi,

You can achieve the desired effect having an alternative sprite of each piece with interleaved diagonal stripes in alternating colors, and apply color cycle animation:

https://en.wikipedia.org/wiki/Color_cycling
https://amiga.lychesis.net/specials/ColorCycling.html