It isn't necessarily a downfall, but you can't run carbon or other mechanical filtration with a sponge filter. I have not had good luck with sponge filters partially for that reason and prefer HOB, canister, or in-tank filters.
Depending on stocking levels, a sponge filter might not be able to keep up, but that is true with any filter. It might be just the trick for you and couldn't hurt to try!
You said you have mostly small fish and shrimp in your small tanks, so sponge filters may have enough mechanical filtration. The other biggest possible drawback is just that they take up more room in the tank, but small ones area available, so that issue is minimized.
You probably already know this, but if you do make the switch leave your HOB's running for a few weeks before removing them to give the sponge filter time to grow a colony of beneficial bacteria.