There are roughly two ways of doing autotiling - either the autotiles are counted as separate tiles and are then edited/inserted "offline", with the game rendering the required pieces straight off the map, or the tiles are stored without the information and it is only calculated and stored in memory (think how Minecraft fences work - the level files only store the fact that a block is a fence, not what is connected to).
The editor likely does the autotiling which is then rendered as is in the actual game.
So you either want your autotiling happen at generation, or afterwards when the map is loaded.
It is a lot simpler for the generator to place the "dumb" tiles - that way it is not concerned about all these details, not to mention if it screws up the generation (that happens a lot) it won't look horribly out of place.
Here's more or less the step-by-step approach:
First, you generate/load/otherwise create your map with "dumb" tiles. In addition, each tile has an extra field associated with it (whether it is an extra field in a Tile struct or just an extra [x,y] array is an implementation detail and is not important). Call this field the AutoTile code. It needs at most a byte of space, technically fitting into 48 distinct values.
Afterwards, once you need to render the map, you iterate over every tile and check each of its 8 neigbours whether it should "link up" to that tile or not. It might only link up to itself (wall tiles connecting to other walls) or to any tile your game considers "solid" (like fences in Minecraft).
This is the algorithm I use in a project of mine, while it is specific to my platform/framework (C#, Windows, MonoGame), it should give the general idea:
public void DoAutoTile(int X,int Y)
{
//bounds checking
if (X >= this.Width || X < 0 || Y >= this.Height || Y < 0)
return;
int code = 0;
int left, right, top, bottom, tl, tr, bl, br;
left = GetWall(X - 1, Y);
right = GetWall(X + 1, Y);
top = GetWall(X, Y-1);
bottom = GetWall(X,Y+1);
tl=GetWall(X-1,Y-1);
tr = GetWall(X + 1, Y-1);
bl = GetWall(X - 1, Y+1);
br = GetWall(X + 1, Y+1);
//some cases look identical, ignore these
if (top == 0)
{
tl = 0; tr = 0;
}
if (bottom == 0)
{
bl = 0; br = 0;
}
if (right == 0)
{
br = 0; tr = 0;
}
if (left == 0)
{
tl = 0; bl = 0;
}
code = tl + top * 2 + tr * 4 + left *8 + right * 16 + bl * 32 + bottom * 64 + br * 128;
//not part of algorithm
Tiles[X, Y].WallAutoTileCode = code;
}
In my case the GetWall function returns a 0 or a 1 answering the question whether the tile at that spot is a "wall" (solid).
The AutoTileCode now uniquely refers to a sprite in a spritesheet corresponding to how the tile is supposed to look like.
For the purpose of this being a self-contained answer, I will also include the Dictionary (associative array) that maps the AutoTileCodes to offsets into a spritesheet (before multiplication by whatever actual sprite width is in pixels), the spritesheet being 10 sprites wide and 5 sprites tall (nearly optimal packing for 48 states while still being "readable").
public static Dictionary<int, Vector2> BlobIndices=new Dictionary<int, Vector2>()
{
#region BlobMappings
{0, new Vector2(5,1)},
{2, new Vector2(0,2)},
{8, new Vector2(3,3)},
{10, new Vector2(6,2)},
{11, new Vector2(3,2)},
{16, new Vector2(1,3)},
{18, new Vector2(4,2)},
{22, new Vector2(1,2)},
{24, new Vector2(2,3)},
{26, new Vector2(5,2)},
{27, new Vector2(6,4)},
{30, new Vector2(5,4)},
{31, new Vector2(2,2)},
{64, new Vector2(0,0)},
{66, new Vector2(0,1)},
{72, new Vector2(6,0)},
{74, new Vector2(6,1)},
{75, new Vector2(5,3)},
{80, new Vector2(4,0)},
{82, new Vector2(4,1)},
{86, new Vector2(6,3)},
{88, new Vector2(5,0)},
{90, new Vector2(8,1)},
{91, new Vector2(8,3)},
{94, new Vector2(9,3)},
{95, new Vector2(8,0)},
{104, new Vector2(3,0)},
{106, new Vector2(7,4)},
{107, new Vector2(3,1)},
{120, new Vector2(4,3)},
{122, new Vector2(8,4)},
{123, new Vector2(7,1)},
{126, new Vector2(3,4)},
{127, new Vector2(7,0)},
{208, new Vector2(1,0)},
{210, new Vector2(4,4)},
{214, new Vector2(1,1)},
{216, new Vector2(7,3)},
{218, new Vector2(9,4)},
{219, new Vector2(2,4)},
{222, new Vector2(9,1)},
{223, new Vector2(9,0)},
{248, new Vector2(2,0)},
{250, new Vector2(8,2)},
{251, new Vector2(7,2)},
{254, new Vector2(9,2)},
{255, new Vector2(2,1)}
#endregion
};
A simple Search and Replace is likely sufficient to convert this to whatever data types your framework would prefer.
These values map to a spritesheet with this layout:
(feel free to use this image as a placeholder or to check implementation or even in your finished game, it's very simplistic and zero effort went into it).
The last step is to tell the renderer to use this data when drawing the tiles - have it refer to the tile's AutoTileCode, consult BlobIndices (the Dictionary from earlier) for the corresponding X,Y offset, multiply to get actual pixel values if needed (or straight UVs or whatever), perhaps some other offsets if you combine multiple AutoTiles in one sheet, and render the resulting tile.
For example, the value 0 (nothing connected) will correspond to sprite [5,1] in the sheet, which is indeed an "isolated" single wall. The value 2 (only the top connected) corresponds to [0,2], which indeed looks like a wall connected to something at the top, and so on.
If you want more information on this, check out the articles on "Tile Bitmasking" (not super technically correct, but the name stuck) that got me in the right direction a while ago, and the first one even contains a version of the Dictionary I created, but with numerical indices instead).
How To Use Tile Bitmasking to Autotile Your Level Layouts
Adventures In Bitmasking - Angry Fish Game Studios
This link contains another explanation, and I think that's where the layout I am using came from originally, it also shows the simpler sets when you don't need arbitrary shapes.
Mechanic #166 - PGC: View-Tile Rulesets
This code file on one of my projects contains the autotiling code as well as rendering code, so it can be seen in action:
Map.cs on GitHub
