You can, but yogurt tends to separate when mixed with other ingredients. You should probably just make enough for one use at a time. If you have extra, let someone else do a mask or use it yourself within a day or two.
They don't need to be refrigerated if they are kept in an air-tight container. If they are not stored in an air-tight container then refrigerate them. If taking them on...
Use witch hazel afterward to help alleviate the burning sensation. Next time you shave, mix the olive oil with another oil to tone it down in case you are allergic...