I think you will find most of the noise is coming from the chain guide slapping the swingarm. Once I glued mine down, All the noise stopped.
Use a glue called "Sikaflex" It's used to bond windscreens in cars, good stuff.
(Picture was taken whilst glue was drying, that's why there is a strap holding it down)