Week 57: Autotiling forever!!

Corner-type annotations drawn over-top their corresponding subtiles.

tl;dr: I continue to work on custom autotiling logic, and I am now planning to match individual subtile quadrants using a separate corner-type annotation image.

Skip to the bottom to see my normal weekly highlights and laundry list.

The many different iterations of my tileset system (so far!)

Oy, I've now been working on tilesets for more than a month! But it's worth it!

It's been taking so long because tilesets are inherently very complicated, and I'm trying to invent a completely novel approach to autotiling that will be easy-to-use for both the tileset author and the level author.

Along the way, I've continued to discover new approaches to improve or simplify my system. So each week I've been working on a different, but related, problem.

Since it's all becoming a bit too confusing for even me to keep track of, I wanted to enumerate all the design iterations I've been through over the last month.

Here are some notes about how I want my tilesets to work:

• I include subtiles with surface slopes along multiples of 90, 45, and 27 degrees.
• This makes my tilesets pretty unique, and lets the level author create unusual levels.
• But this makes the tileset much more difficult to draw and define autotiling logic for!
• I have all of my art fade from a high-contrast surface edge to a dark solid color for the repeating interior subtiles.

Hand-draw all individual subtiles, and autotile them using Godot's standard bitmask system

• How it works
• This is the normal approach supported by Godot.
• Godot has a built-in tileset editor for encoding autotile bitmasks and collision shapes for each subtile.
• An autotile bitmask is a 3x3 grid. Each cell in this grid is a binary flag which indicates whether the given subtile should be used if there is or is not a neighbor in the relative position corresponding to the bit.
• Pros
• Simple to understand.
• No custom logic required.
• Cons
• It takes a lot of time for the tileset author to draw the art, configure the bitmasks, and configure the collision and occlusion shapes for each subtile.
• Shapes with non-90-degree angles don't autotile, which makes them extremely time-consuming for the level author to use.
• The art looks bad! Because the tileset author must draw shapes that will transition smoothly from one tile to the next regardless of the relative angle difference.

Godot's built-in tileset editor.
The red parts are the 3x3 bitmasks used in autotiling.

Update tileset art to extend edges across the depth of two cells, and introduce custom autotiling logic

• How it works
• Consider a floor surface:
• The top row of tilemap cells along the floor would all show the high-contrast surface-edge art, with the beginnings of a transition to the interior art.
• The row just below that would then show interior art that fades to the solid deep-interior color.
• This requires entirely custom logic for selecting which subtile to show in a given cell, since exterior subtiles use completely different matching rules than interior subtiles.
• Godot supports custom autotiling logic via the TileSet._forward_subtile_selection method.
• Also, this now requires looking not just at neighbors one cell away, but also those that are two cells away.
• Pros
• The tileset author can create much better-looking art when they're able to spread the art across two cells rather than just one.
• By using custom autotiling logic, I can automatically determine every subtile to place, even for non-90-degree-sloped surfaces. This makes it much easier for the level author to quickly create levels.
• Cons
• As the art extends deeper before fading to a solid color, there are exponentially more corner cases to handle between adjacent subtiles with different angles. So the tileset author has to draw a lot more art!
• This requires writing a lot of custom logic in order to choose which subtile to place for every possible neighbor-subtile combination.

Custom autotiling with subtile art spanning a depth of two cells.
The middle-grey color is in separate cells than the light-grey color.

Update custom autotiling to use more declarative and flexible subtile configurations based on corner-types

• How it works
• Target-shape information is calculated separately for each corner of a given tilemap cell, based on the angle-type of the cell, and the presence and angle types of its neighbor cells.
• For example, I can calculate that a cell should include a floor with a 45-degree positive slope, because its angle type is 45-degrees, and it doesn't have a neighbor above or to the left.
• Then, for each corner-type, I define a collection of reasonable fallback corner-types that could be used in-place of it.
• Then, given the four target corner-types for a given cell, I find the subtile from the tileset that most closely matches the four target corner-types (also considering whether the subtile could match one of the corner-type fallbacks).
• Pros
• The tileset author only needs to create art for the most important subtiles (they can omit a large majority of them!).
• This is also important because there are many possible valid combinations of corner-types that could appear in a subtile, and, depending on the art design for the tileset, many of these combinations could actually render the same art. The tileset author shouldn't have to include these redundant subtiles.
• Cons
• The tileset author needs to manually configure the corner-types for every subtile in the tileset. That is error-prone and takes a long time!

Some of the many possible corner-types.

Render individual subtile quadrants

• What is a "quadrant"?
• Every subtile can be divided into four parts, with two rows and two columns.
• Each of these parts is a quadrant.
• Very few tiles in the tileset contain a unique quadrant. Most quadrants share the exact same pixels with many other subtiles.
• The amount of quadrant re-use depends on how the tileset art is designed.
• How it works
• I use my pre-existing logic for calculating the four target corner-types for a given cell.
• Then, I define a large enum for every possible subtile quadrant in the tileset.
• Then, I define a large nested switch-statement for choosing the correct quadrant enum for a given corner based off of the corner-type and the neighbor corner-types.
• I would then also define fallback quadrants for corner-type combinations that weren’t encoded in my logic and for quadrants that weren’t included in the tileset.
• Pros
• The tileset author can omit many of the possible corner-type combinations, and subtiles for these corner-type combinations can instead be automatically created on-demand by combining the corresponding quadrants.
• This is especially true for interior subtiles, because there are more possible angle combinations and the interior art is usually more symmetrical.
• Cons
• This would require writing many thousands of lines of tedious and error-prone logic for every possible quadrant.
• Introducing any new corner types would require updating logic in many different places.
• Also, the tileset author would have to manually configure the positions of each different quadrant in the tileset, which would also be a lot of work.
• This seemed to make sense originally, because, for most tileset designs, a given quadrant is often repeated in many different subtitles throughout the tileset.
• However, as I spent more time digging into all the possible alternative ways of drawing art for the tileset, I realized that the amount of quadrant re-use can potentially decrease to nearly nothing.
• So my quadrant-matching logic would essentially need to account for every individual quadrant in the tileset, which is way too much logic to handle!

An Aseprite tilemap cell forms the quadrant of a Godot tileset subtile.

Use a corner-type annotation image

• How it works
• Annotate subtile corner-types in a separate layer in the Aseprite editor.
• Then export this corner-type-annotation layer as a separate image file.
• I can then write image-parsing logic in GDScript to associate a subtile’s corner-types (from the annotation image) with the underlying quadrants (from the corresponding tileset image).
• I can then use my pre-existing target-corner logic to choose the best quadrant matches according to whatever is available in the corner-type-annotation file.
• And I can choose each quadrant separately, so they don’t all need to come from the same subtile region in the tileset image.
• This new approach uses the same basic idea as the previous quadrant approach—select separate quadrants according to corresponding corner types—but it uses automatic matching according to corner-type-match weights rather than way too much hard-coded and brittle logic!
• Pros
• The tileset author can probably create a much smaller tileset image, while still getting full coverage of all possible subtile angle combinations.
• I need to write much less subtile matching logic (which is all brittle and error-prone).
• The automatic subtile matching logic makes it easy to also choose the most closely matching fallback when the ideal quadrant isn't defined.
• Cons
• The tileset author has to also draw the corner-type annotations.
• But at least this is less work than any of the previous configurations I was going to require from the tileset author!

Corner-type annotations drawn over-top their corresponding subtiles.

 

The final tileset (for only 90-degree and 45-degree angles) is on the left,

and its corresponding corner-type annotation image is on the right.

(You can click to enlarge an image.)

What happened last week?

Highlights

• I continued creating a custom autotiling system.
• I fixed my Mailchimp emails to not go into folks' spam folders.
• I researched possible ways to use Godot Plugins to improve my Surfacer and Scaffolder frameworks.

Laundry list

• Continue creating a custom autotiling system.
• Update tileset art to fix issues with new layer depths.
• Update tileset art to fix issues with some missing 45-degree corner cases.
• Update tileset art to support exterior-layer art extending past one-quadrant.
• Discover and add some more missing corner cases to the tileset art.
• Add helper functions for printing error messages for missing corner-type-matching cases.
• Start implementing a new quadrant-based tileset system.
• Start defining SubtileQuadrant enums.
• Start writing logic for getting target subtile quadrants given the corresponding target corners.
• Update tileset art to fix issues with some missing 45-degree clipped-corner combination cases.
• Think through yet another new approach:
• I needed to abandon my latest quadrant-based approach.
• This seemed to make sense originally, because, for most tileset designs, a given quadrant is often repeated in many different subtitles throughout the tileset.
• However, as I spent more time digging into all the possible alternative ways of drawing art for the tileset, I realized that the amount of quadrant re-use can potentially decrease to nearly nothing. So my quadrant-matching logic would essentially need to account for every individual quadrant in the tileset, which is way too much logic to handle!
• I had a new idea that uses a corner-type-annotation image.
• Annotate subtile corner-types in a separate layer in my Aseprite files.
• Then export this corner-type-annotation layer as a separate image file.
• I can then write image-parsing logic in GDScript to associate a subtile’s corner-types (from the annotation image) with the underlying quadrants (from the tileset image).
• I can then use my pre-existing target-corner logic to choose the best quadrant matches according to whatever is available in the corner-type-annotation file.
• And I can choose each quadrant separately, so they don’t all need to come from the same subtile region in the tileset image.
• This new approach uses the same basic idea as the abandoned approach: select separate quadrants according to corresponding corner types. But it uses automatic matching according to corner-type-match weights rather than a way too much hard-coded and brittle logic!
• Start implementing my new corner-type-annotation-image tileset system.
• Draw most tileset corner-type annotations.
• Create a legend for the tileset corner-type annotations.
• Update my tileset templates to have more layers, and dots in every other cell, in order to disambiguate quadrants in tl, tr, bl, br corners.
• In theory, this would make it a lot easier to quickly use the template for generating new tile art.
• But I gave up on this, since the tileset author might want the quadrants to be re-used anyway, and the built-in Aseprite tilemap features handle this well-enough already.
• Fix my devlog's Mailchimp email campaigns to stop going into spam folders.
• I spent some time researching all the possible reasons that this might happen.
• I needed to "authenticate" my domains through Mailchimp.
• I also updated my subscription settings to start collecting and using audience names in addition to their emails, since including their name in the email apparently helps a lot to get past spam filters.
• I wasted a lot of time trying to set up email aliases on my subdomain (through Google Domains), but never got that working (but ultimately, I didn’t need to).
• Research Godot Plugins.
• I think there are some very useful Plugin features that I should be using with Scaffolder, Surfacer, and my new tileset library.
• I had looked into Plugins a long time ago, but had decided that I probably didn't need to use any custom editor UI, and I could already get the benefit of running logic in-editor just via `tool` scripts.
• Read the docs.
• Look at some other folks’ packages in the Asset Library.
• Meet with GGJ teammates for a retrospective.

What's next?

• Continue creating my custom autotiling system.
• There are probably two more weeks of work left for this?

🎉 Cheers!