That's a great plan. All three days we were there, they started letting people into the park even before the stated early opening time. The HP area is waaay at the back, so just zip over there and ignore all the other rides/attractions you will see along the way. You can come back to those later. Use your first few hours exclusively at HP, when the park will be the quietest. It'll be fine. Another thing is, you can't take backpacks on the Hogwarts ride (and some of the other ones, too). A friend suggested that we not take a backpack to not waste time at the lockers. We took her advice the first day (cargo pants/fanny pack
for snacks) and since there weren't really any crowds, we would have been fine either way. But, I saw on Saturday, that it would have been a pain to deal with the lockers for backpacks.