There is a lot of discussion in the Paleo world about dairy. Some people follow a paleo diet/lifestyle with dairy added and call themselves lacto-paleo. I think it's a personal decision for everyone that can really only be made after completing a Whole30 and reintroducing it properly. For me, dairy is really not worth it. I found that in small amounts (a slice on my JJ's unwich or little added to a salad), it wasn't bad. But, a cup of cottage cheese or scoop of ice cream had horrible affects: stomach pain, diarrhea, headache & acne...etc. So, I plan to remain (mostly) dairy free. I know that if I choose to eat more than a little dairy, I will pay for it. We have a good alternative to ice cream and I have found that I can live without cheese (although after a week of having cheese curds in my face and turning them down, I did finally eat a hand-full of them. Good thing they are several hundred miles away from me!).
So, if you're curious about how dairy may or may not be affecting your health, give the Whole30 a try and then reintroduce it. You may find that it's not a problem. Or you may decide that you can live without it after all.