if it is during a regular tank cleaning, you really shouldn't clean the gravel like that since a good part of your beneficial bacteria lives there. You should only change 25% of the water and use a vacuum to take the gunk out of the gravel. Taking the fish out and in again will only stress them and take the beneficial bacteria out of the gravel as well.
Same reason why the filter pad shouldn't be changed either.
Edit: I just saw that you have a 1ppm
ammonia in your tank... at this point you should increase the water changes to 50% not weakly but daily, since your tank is not cycled.