We started traveling overseas when DS1 was a baby and have continued to do so almost every year. We initially traveled because DH and I wanted to go but didn't want to always leave our kids behind so we weren't as concerned about them remembering everything. I feel like it depends on your expectations as it is certainly not a "vacation" but a trip. We stop at almost every park we see, get ice cream frequently and sometimes have to scrap plans to spend an evening at the hotel pool. We've found European cities to have great parks for kids to decompress and run around.
In flight entertainment has always been a savior on long flights but I've always planned to provide all the entertainment if need be. Not going to lie, some flights are easier than others but it's doable at all ages. My kids have handled jet lag better than I do. They seem to have one difficult night and then shift to the new time both at our destination and upon return. We always try to jolt ourselves to adjust quickly.
It's amazing to watch kids experience new and different cultures. While they may not remember every detail of the trips, they will take in the lessons one can only learn experiencing it first hand, being immersed in the differences and similarities.
Mom to Two Wild and Crazy Boys and One Sweet Baby Girl