If the pilot jet is plugged, the engine tries to idle on the spillover from the needle. To do this the slide stop has to be screwed in almost to the point of coil binding the screw spring. It will change RPM and not really idle. This scenario does not "just happen". It usually results after the bike has been stored for a few months. The gas varnishes over time and plugs the pilot jet. With the pilot jet removed and held up to a light you should see through it clearly. I doubt this is your problem.
I don't expect a 2T to idle well. Temperature changes will affect how it runs. If you are running the bike hard, it is common for the idle to change a bit. The real question is "What's in it for jetting?". The stock N1EF needle is known to be an issue. If you have a 42 pilot jet, the air screw should be about 1.5 to 2 turns open from lightly seated. My bike runs best at 2.5 turns out on a 42 pilot jet.
On deceleration does the bike "Knack"? This is the sound of unburnt fuel igniting in the pipe. It usually means the bike is set up too rich. The problem is that your description so far would make me think your jetting is bit lean. If you were getting a kickback on starting all the time, I'd check the flywheel key as Davehuge suggests.
I don't run a thermostat in my bike so I get more variation in how my bike runs. I also don't change my jetting through the year. It is set up for spring & fall. Thus it runs slightly lean in late fall and the engine gets noticeably crisper. In hot weather it's slightly rich and feels a bit soft.
At this point I'd say you need a better needle.