What do the different DOOM 4 antialiasing settings do and how do they impact performance relative to each other?

That list you're seeing is a list of different technologies for anti-aliasing the game. They're by no means exclusive to DOOM.

Different technologies perform anti-aliasing in different ways and thus achieve different results.
Usually the game lists them in order of less processing power consumption to more processing power consumption. The technology in the top is the one that requires less processing power and the one in the bottom is the one that requires the most processing power.
Generally the less processing power an anti-aliasing technology uses the less aliasing it reduces and/or introduces more blur or other defects in the image. The more processing power it uses the more aliasing it reduces without introducing many defects to the image.

There are, however, a few exceptions to theses generic rules. I just gave them to use as a rule of thumb when you don't know what they mean.

Now let's go over a few of the technologies you mentioned:

• FXAA - Fast approximate anti-aliasing - developed by Nvidia.
  • Pros: does not require large amounts of computing power and includes all pixels of the screen (even those showing transparent elements).
  • Cons: textures may not appear as sharp and must be applied before rendering the HUD elements
• SMAA - Subpixel Morphological Anti-Aliasing - it's a more efficient implementation of the traditional MLAA (Morphological Anti-Aliasing).
• TAA - Temporal Anti-Aliasing - a technique to prevent temporal aliasing which is caused by framrates being too low in comparison with moving objects in a scene.
• TSSAA - Temporal Super Sampling Anti-Aliasing - same as TAA but uses supersampling to achieve better visual results.

All of these technologies come in varying strengths indicated by their multiplier. The multiplier is the number with the TX in front of if. in TSSAA (8TX) the 8TX is the multiplier.
The higher the number the more processing power it will require and the better the results it will achieve.

For more on anti-aliasing please visit Anti-Aliasing Wikipedia, Spatial Anti-aliasing Wikipedia and Anti-aliasing article.

The T in TAA it means Temporal Anti-Aliasing.

Basically, every frames, the image is accumulated in an accumulation buffer and is merged with the current frame. This helps remove flickering (Temporal Aliasing).

TSSAA means Temporal Super Sampling Anti-Aliasing. This will compute Super Sampling Anti-Aliasing by using the accumulated frames. This will to not only reduce flickering, but will also reduce the aliasing of edges and textures. 8TX means that the aliasing is resolved using a pattern of 8 samples.

1. Temporal Anti-Aliasing Super Sampling in Unreal Engine 4
2. High Quality Temporal Supersampling - Brian Karis