Yeah, I have a sponge that starts growing over my clavularia just like that. I'm not all that concerned about the clavularia...it grows like a weed....so I just scrape it off occasionally.
Taking the rock out of the water for a bit sounds like a good idea, I may give that a try myself.