| Keep doing what you're doing. If you were to take it out and rinse it you'd mini-cycle your tank or worse. You're never going to get all the gunk out and that's ok, you just want to get as much as possible of the leftover food/fish poop that's down there. Make sure to dig down deep and stir up the gravel so you really clean it. Yes it isn't fascinating but it's part of keeping an aquarium (kind of like I love having cats, but cleaning their litterboxes ain't exactly thrilling).
Check out the other threads on the Aquarium Maintenance board too, there's lots of good information there. | 
Similar Aquarium Fish Forum Threads 
