Mystery Solved

I sent email to the good folks at NC State and Virginia Tech who would know such things and they both came back without hours with the same answer: Clethra acuminata. Seems that I was barking up the wrong tree and it's not a birch at all.

The common names are Cinnamon Clethra and Mountain Sweetpepperbush. It's a favorite of hummingbirds. Imagine that!