Remember that the beneficial bacteria needed to sustain the cycle is attached to all surface areas of the aquarium with the highest concentrations of it being in your filter.
You do not want to over clean and remove this beneficial bacteria. Maybe select a couple of ornaments and clean them and then get the rest at the next water change.
I do not have any algae issues so I do not clean my ornaments either. Too, a little algae adds to the more natural look of the aquarium and can be beneficial to some fish and critters. Algae is a good sign that there is oxygen in the water.
Chances are when you move the ornaments, any detritus on them is going to end up back in the aquarium and it will filter out.