dairy allergies can often be more complicated than just the casein - http://onlinelibrary.wiley.com/doi/10.1034/j.1398-9995.2001.00911.x/full

then there is also the issue that certain food components can be highly immunogenic without being allergenic such as gluten in celiac disease. it would be great if there were conclusive tests that could be done to determine whether something is 'safe' to eat or not but with the current state of scientific understanding and testing methods available there is still a lot of guesswork.