I saw your post earlier and wanted to reply because we went through this with our family a few years ago. My DS was in K in our wonderful neighborhood school where my DD had attended since K (she was in 3rd at the time). He entered K as a pretty advanced reader and the school was working hard to accomodate him (went to the highest reading group in 2nd grade for reading 2-3x per week and spent 2 hours a week doing enrichment activities with the librarian, so basically didn't spend much time in his K class during their language arts block). We weren't entirely sure whether he was just a precocious reader or whether he might need enrichment across the board, so we went through our district's GT testing in January of K (they test all second graders, but if the teacher recommends or parents request, students can test as early as K). He ended up testing 99%+ across the board and was offered a place in one of our district's center based programs for 1st grade (basically these programs have one stand alone GT classroom at every grade co-located and somewhat integrated with a neighborhood school).
The placement we were offered was only a 10 minute drive from our neighborhood at a school where we knew a bunch of families from the community/sports. We were lucky that the start times were such that we could drop my DD at 7:45 (7:55 start time) and get my DS to school by 8:00 (8:10 start time). It was still a tough decision though - having kids at two schools meant two PTAs, two sets of events/fundraisers, two sets of families/friends. We also wondered if we had done our older DD a disservice by not having her test in K (she had tested into the GT program in 2nd grade and was getting some services in her "GT areas" which were primarily math, but we hadn't switched her because the program is 1-2 years accelerated and it's difficult to start beyond 2nd - 3rd grade). Ultimately, we made the decision to have her stay at our neighborhood school for 4th grade and on and to have our DS switch to the GT center. We considered switching our DD to the gen ed at the new school (the new school had some amazing programs and was probably more creative than our neighborhood school even in their non-GT classrooms and she actually had great friends already at the new school), but ultimately decided that we were really excited about her teacher options for 4th-6th grade at our current school and she had an amazing group of friends and a super highly academic peer group in her grade at the current school, so we felt like she was being challenged pretty well already. For our DS, some of the factors were we wanted him to have academic peers and he only had 1-2 in his grade at his current school, we wanted him to have access to the more creative/accelerated curriculum, we loved the ideas of teachers who were trained to work with GT kids and ultimately we felt like it was the best fit for him.
Now that we're in our second year of kids at two schools, I can honestly say we have no regrets. For my DS, the program has been great and even though he fit in fine socially at his old school, he fits in even better at the new school and seems to have really found his people. For my DD, we knew there was a chance she could have a teacher she'd had in 2nd grade again this year in 5th grade who is one of those once in a lifetime type teachers and it happened
She's changing and growing so much this year and we're so happy we didn't have to uproot her. Having made what we felt was the right decision at the time for our DS has made us realize that if it's feasible for you to make the right choices for each kid, it can be amazing. It's actually opened us up to looking at other options (private) for my DD for middle school (which will lead to more challenges schedule-, involvement-, and money-wise) because we feel like we can handle it and have seen what we have to plan for.
If you do end up switching, I think you have to set yourself up for the fact that some things will have to give. I split my volunteer time between the two schools and I'm much less involved at the PTA/School level at DS's school (I volunteer in the classroom, but went to my first PTA meeting only after 15 months at the school). We don't do every extra activity (e.g. DS skipped the science fair this year and I haven't made it to my DDs class parties this year for the first time ever).
Finally, if you decide not to switch, it will be OK
You sound like a great parent who cares and is involved and if you're there advocating for your DS to get the enrichment he needs, he's going to be fine. A friend of mine with two kids in college is fond of reminding me that as long as you're involved and your kids are reasonably happy/safe, they're going to learn and they'll probably end up at the same place in life no matter where they went to elementary school.
Good luck!