The Australians and the Dutch I'd say! Or is that perhaps because of 'stars' like Ian Thorpe and Pieter van den Hoogenband? I guess it's more fun to watch for 'general sports fans' if someone of their own country competes and wins the big prizes.
I have to say swimming has always been very popular in The Netherlands (where I'm from) as a recreational sport. But sports TV programs are dominated by soccer. Since Pieter van den Hoogenband and Inge de Bruijn started bringing in the gold since Sydney, it has become a lot more popular with the mainstream audience.
I noticed on the web that there are quite a few international fans of Pieter van den Hoogenband and since he's my favourite swimmer too, I set up an English website dedicated to him. I've translated some Dutch articles and interviews and even a few of his columns (he writes for a Dutch newspaper). I'm working on more and am posting daily, so perhaps you would like to check it out and bookmark it:
http://pietervandenhoogenband.blogspot.com.
If you link me on your website (and click once on it to activate it), I automatically link back with a nice javascript thingy.