Just looked at AMD's latest chip specs — still stuck at 12GB GDDR6. This isn't funny anymore. Even basic Llama 3 models won't fit in this memory with batch sizes above 1.
Compare this to NVIDIA's H200: 141GB HBM3 versus 12GB GDDR6. That's an 11.75x difference in memory capacity, yet only a 3.5x price gap. This is where real economies of scale kick in.
Speaking of alternatives, Apple's M7 Ultra at least offers unified memory architecture instead of these GDDR bottlenecks.