It is not necessarily bad. Brown algae typically grows because of too much light. However, it could also mean your water parameters are off. How long do you keep your light on and what are your water parameters?
I had the same issue in a ten gallon. And I love watching my fish until around ten. If you didn't want to jurastically change your lighting situation, you could put an algae eater like a pleco or otocinculus depending on the tank size. I put three ottos and they had the algae gone within a day and they maintain it now.