Rhododendron is toxic as well, if you eat the leaves but I've heard is fine for spoons.
Walnut trees can kill competing plants that try to grow near one, but it have used a walnut spoon for ages.
I haven't looked into Holly, but still wouldn't worry too much. Toxicity is one of those words that sounds scary, but might be virtually meaningless in some instances. Maybe not the best example- Some people are allergic to poison ivy, some are not affected by it, PI is listed as a toxic plant. You have to know how the thing affects you. On the other hand toxicity can also mean very deadly.