I think the main reason why Swimming isn't that big in the states is because it is difficult to commercialize swimming. Like many of the other major sports (football, basketball, baseball) they are commercialized a lot and recieve a lot of TV time, they have sponsors and large arenas.
Another reason I see swimming not being as prevelant as other sports is because there are not that many big names. You of course have your 2 or 3 stars, but they are only going to swim 3 or 4 events at most, and during the whole swim meet(lasting a couple of hours), you are only rooting for a player/competitor for about 2-5 minutes, depending on race.
The main reason is probably Americans Fear of Water, Scary
