I'd say the toughest coral to keep is the one you can't seem to get to live in your own tank. It's kind of strange, but many people that have fantastic reef tanks are unable to keep some types of coral, even though everything else is doing well. This may even be a coral that is considered easy to keep by others, such as xenia, or GSP.
For myself, I never seem to have any success with open brain corals. I've not tried one for a long time now, and I hope I have a little better understanding of them, so I may try again. Yes, it's considered an easy coral.