My emerald crabs (Mithraculas sculptus) used to sleep in my leathers, never bothered them. Even if the hermit picks at a polyp or two doubt it hurts the colony. Lemon gobies nip at Acropora heads, but the Acro grows faster when they are present than not. If the leather is healthy don't worry.