HI, WELCOME TO FISHLORE!
Contrary to what the local fish store will tell you goldfish need more room than a 5 gallon tank. The profile here says 10 gallons per fish. Others recommend 20 gallons for the first fish and 10 gallon for each additional fish.
Do you test your water for anything other than ammonia? If so what are the readings for nitrite and nitrate? How often and how much water do you change?
It just sounds to me like there could be a build up of nitrate in the water that requires several large water changes, and gravel vacuuming. It is always a good idea to add vita chem if a fish appears to be ailing....as well as prime, until you get the water quality worked out. Good Luck