Actually I'm more or less following a kind of paleo diet - I do eat gluten free grains sometimes and also legumes, but not very much any more. Main point of paleo is not to eat processed food, sugar and grains. Basically, eat things that are as free of anti-nutrients as possible and which you could eat raw. From my experience, you don't need tons of food when you eat paleo because this way of eating is much more satisfying. So even though some foods may be more expensive (grass fed beef, organic eggs), it's not really more expensive all in all. I'm totally fond of breakfast oat porridge, e.g., but I found that I do much better with a bowl of paleo pumpkin custard (pumpkin puree, half a mashed banana, 1 beaten egg, a bit coconut milk, dash of cinnamon, all slowly heated until you get a custard) or chia seed pudding (chia seeds, coconut milk, some fruit, drop of stevia, vanilla) or a vegetable frittata, or a bowl of avocado, tomato, canned tuna salad...
I do eat some dairy, but mostly sheep and goat milk produce.
Eating totally paleo also worked for me - no grains at all, no legumes, no dairy whatsoever. Main expense was organic meat and eggs (which I do buy anyway). Depending on whre you live, maybe there are farmers from whom you can get stuff. In Germany, most supermarkets have their own organic lables, which is not as expensive as buying from organic groceries.
I went back to also eating some dairy, grains and legumes because I felt good with them in moderation and like them, so I don't see why I should completely ban them from my diet.
Here are my favourite recipe sites for paleo: