It's a matter of having perspective and being realistic and your own goals as a family too. For us, our goals meant we send our kids to public schools so we can fully fund our retirement and beyond, fund our boys college 100% and take 3-4 vacations a year, DH's brother has 5 kids and chose to send them all to private schooling from K-12th plus expensive private college which meant they never take a vacation unless it's free or funded by someone else, and still take out some loans for kids college while paying some, retirement wouldn't be as fully flush but enough to live frugally. I'm an only child and all of my cousins on my mom side really vary from making it to solidly comfortable lifestyle, whereas DH's cousins are all quite wealthy but we still get together and celebrate together in different ways without spending excessively.
For us it works since we prefer to vacation solo or with close friends instead of with our either side of family. We also don't consider time spending with either set of grandparents as vacation time, but rather as spending time together as a family whether here staying in my home or visiting them at their home.
Mummy to DS1-6/11 and DS2-1/14