No not all fish require heaters, but the ones that are not
goldfish and koi are either shoaling fish of around 3" size or quite a lot larger (5" or larger - quite a lot larger) plus being fish that I have never seen or heard of so would probably only be available by special order or in a very large fish store. The list out the
Aquarium Fish by Dick Mills of Other Coldwater Freshwater Fishes are: Bitterling, Japanese Loach,Three-Spine Stickleback, Green Sunfish, Pumpkinseed, Orange-Spotted Sunfish, Bluegill, Spotted Sunfish, Chinese Sailfin Sucker, Red Shiner, European Minnow, Fathead Minnow, Japanese Rose Bitterling, Bitterling (another one), Oily Gudgeon, Rudd. There are none of these appropriate for a 4 gallon tank either because of size or the need to shoal.
Sorry I cannot be more help. The real minimum for a betta is 5 gallons to give them room to be active. It is okay to use a smaller tank for a quarantine or a hospital tank as you do not really want a sick fish to be terribly active, but to live in full time, it is not big enough.