That's about what we have to do.Thanks all. I didn't realise you could get sponges like that, but after your help I found out it's possibly some form of Leucosolenia sponge.
The Vermetids are a pain, I have used aiptasia x on them but they just turn up somewhere else, I've got them on most of my rocks. I used to get stressed about all the unwanted stuff (gha,gba,vermetids etc) in my tank but there's always something so I TRY not to let it bother me too much and I do enjoy all the diversity, good and not so good.