Welcome william!
Your tank is very overstocked!! You are supposed to have about 1 gallon of water for every inch of fish you have
10 gal / 28 fish!!!!!
I would say take your fish back to the petstore and start your tank over. having a new tank with that many fish can be very deadly!
Every aquarist should know of the Nitrogen Cycle here is the url,
http://www.fishlore.com/NitrogenCycle.htm
can you post a pic of one of the unknown fishys? I can help you identify it.
and to answer the original question, the mollies are probably crowded and therfor feel threatened.