Could be loose or off axis parts within the primary drive. Most probably the clutch, but I would think it would have to be pretty off balance to give a noticeable gyro effect, given its not very large and isn't very heavy (in relation to the rest of the bike). Also, if it was that far off axis, it would probably be banging into things and making horrible noises, and grinding up bits of metal into the oil too.
More likely is wheels which aren't true.
Do your friends notice the same thing when they ride your bike?