That's true but I think the issue lies in how slowly mathematics is taught: when you spend all of elementary and middle schools repeating the same material over and over. During elementary school I used to change all the minuses to pluses on the homeworks since I didn't like subtract but I was doing basic algebra (basic polynomials) before I entered elementary school. At that point I was reading interesting math books for fun. I was able to find some math puzzle books and was slowly making my way through those.
Your experiences aren't really typical, then, and neither are my own. It's probably a mistake to extrapolate from our own childhoods to try and figure out what will work best for the vast majority of people.
