Commons get a foot long and are active swimmers, there’s a type of fancy which I forgot it’s name but it’s just a smaller common with bigger fins.
Most of the goldfish you find in stores need at least a 50 gallon, but there are some smaller species that can be kept in tanks as small as 30 gallon tanks.
Think of it this way. Imagine getting a little bird that is designed to fly across fields and putting it in a closet-sized sealed glass box that it can never leave. Most people would think that a closet sized bird cage is actually pretty huge. It's really not. Now imagine that the bird was a hawk, that makes the closet (a huge cage!) so small the bird could not even extend its wings. And this is where it lives. Until it dies. This is all the air it has, all the room to move, all the ground to touch. How fast does it run out of oxygen before it needs fresh air? How fast does its waste make the space toxic?
I have three fancy goldfish in a 75G aquarium, which gives them a 4ftx1.5x1.5 space to swim. My largest fish is about 8" and even with being partially blind and all her big fins, all that space allows for her to actually "run" around (aka build up to full swimming speed) and turn. Normal goldfish are hefty carp that are built for speed and feel safe in a group. This is why many suggest that they should only be kept in a pond.
Think less about the "gallons" size of a tank and more about the swimming space/shape of the tank. If I had my 8" fish in a 40G breeder tank (by any standard a perfectly reasonable size for a single fancy goldfish) at 36″ x 18″ x 16H, that gives her 8 body-lengths to swim horizontally one direction and only about TWO to swim another direction. Not as big as you would think. Now add two more fish to that situation (for me) and it starts sounding pretty cramped. And what about filters? Decor? Even LESS swimming space. If that 40G tank was a 'tall' style instead, they would barely any length to swim.
A goldfish is a very curious, interactive, active kind of fish that busies its day poking around at everything and exploring. They are not 'stream' fish, they are 'slow water' fish. They swim fast, they swim slow. They go up, they go down. Keeping them in a small tank without stimulation does a great disservice to the animal. I feel they should always have at least one other fish with them, as well (which means a larger tank is a good idea anyway)