After reading the PC beta thread, it also seems that the Anti Aliasing options also work when the Post Processing option is turned off on the Mac!
Jaggies have now gone with 8X AA option selected, and the game looks much, much better!
So what would be the optimal setting for each option? My results are the same as squidlr's, but I must admit to ignorance about what the table setting options actually mean.