yeah, don't tear down everything. Just use the gravel vac, change about 30-50% of the water (probably 30) and if there is any gunk on the walls of the tank, wipe that down too. Make sure to dechlorinate the water before adding it so that you don't kill the cycle.
IMO I wouldn't have used the goldfish, because now you have to find them a home, and you could've cycled with just
ammonia, but you live and learn. I hope you have a heater for the tropical fish (you probably do).
Wait 24 hours after cleaning the tank and removing the goldfish before you add the tropical fish.