The downsides as I see it....
1) You can't use the free Magical Express from the airport (which you may not care about)
2) Lack of availability of Disney Dining Plan
3) You have to pay for parking at the theme parks
4) You can't use the Extra Magic Hours. We like the early ones, but seldom use the evening ones.
5) It is MUCH easier to get to and from your resort during the day on Disney transportation and it is much easier if you want to split up. Even if you drive, that means if someone wants to leave the parks, unless you want to rely on the hotel shuttle, everyone has to leave. Especially since you have 3 kids of varying ages, I think this would be of interest to you.
The main upside as I see it is that you do get a lot more hotel for your money in an off-site resort.
Beth, mom to older DD (8/01) and younger DD (10/06) and always missing Leah (4/22 - 5/1/05)