I used mothers aluminum polish to clean up a swingarm on a txt280 trials bike - I wish I would have seen the post on wet-sanding - it took way too long to rub out all the scratches. And the previous owner had used something caustic to clean the bike that etched the aluminum on the swingarm.
The other option would be to have the swingarm "bead blasted" to give it the uniform brushed appearance that you mention above.
jeff