The best tool for this is a rubber cement pick-up which is an inexpensive item for about $2.00. Using your fingers can reek havoc because of the oil in your skin as well as rubbing paint into the white area and generally smearing everything.
Happy Painting!
I can vouch for that! I always have trouble taking off the mask. You should post a link to the doodad you’re describing so we can see what it looks like! 🙂