Every action has a reaction, so that means some part of your body must also be turning. It could be that your shoulders are turning subconsciously whenever you pop the tail, or that your foot is moving slightly forwards or backwards.
If they're asking you questions about work, unfortunately you have to listen and answer them. If they're asking you personal or other non-work related questions, it's time to simply be...
It could be due to either a lack of water or nutrients. Try watering your peppers more frequently. If that doesn't correct the problem, add some nutrients to the soil.
Use witch hazel afterward to help alleviate the burning sensation. Next time you shave, mix the olive oil with another oil to tone it down in case you are allergic...