It's best to have clean hands when handling cotton in my opinion. I've had times where I was in a rush to rewick, hadn't done anything 'dirty' ie used the restroom, ate with my hands, worked outside or anything. The oil from my hands gave it a funky flavor. It wasn't leftover dorito dust or anything obvious. I make sure to use dish soap, a thorough rinse and dry my hands prior to handling wicking cotton. I've also had times I thought my hands were well rinsed (didn't feel slick, no visible suds) and yet it imparted a soapy flavor.
The way I look at it, vaping is personal thing (hence personal vaporizer). It goes in our mouths, we inhale through it, handle it, touch/handle the wicks, if you're a vaper who purges their atomizer prior to inhaling moist breath blows into it and lingers. If it's not someone I'd eat or drink after, share a glass or silverware with (or vice versa) I'd prefer to keep things as sanitary as possible. Just out of common courtesy and err on the side of cleanliness.
So if it were just for my own use or close friends/family, washed hands would suffice. If it was for someone else (customer at a shop) I'd want to wear gloves and preferably not ones with a pungent latex/rubber odor that could get into the wick either. Otherwise for me washing my hands and a good rinse is my preference. Without washing hands the wick tasted funky enough I started over and tossed it. If you're wicking your own atomizer and don't care about dorito dust, doesn't bother you or give a bad enough flavor, more power to you. I'm not gonna bash on others for doing whatever works for them.