Before I put my hands in my tanks, I rinse them off. But what if there is residue on my hands when I'm touching decor or filter media? Am I just being paranoid?
I honestly never worry about it too much. I'll rinse my hands before handling anything in the tank, but that's about it. If I had been doing the dishes, or messing with chemicals before-hand, I'll be very thorough and then make sure to dry off well with a towel, and I've never had issues.
I'd say that with most tanks, everything get's so diluted that it wouldn't hurt anything but the most sensitive of creatures.