Why do we still vary?

I notice that last summer Karl Smith asked "Why Are There Short People?" His logic is pretty good, except for the fact that the fitness variation seems to be much starker in males than females (there is some evidence I've seen that shorter women can be more fertile, though that's balanced by the fact that larger women seem to be able to manage gestation better). In any case, height seems to be a fitness enhancing trait which is highly heritable, and yet the variation in height remains! Karl's readers offered some reasons. What do you think? Mind you, something which immediately comes to mind is that the logic presented for why everyone should be tall and vary only a touch is logic. Not all the assumptions need to hold. For example, has the advantage to height been invariant at all times and places? I have posited for example that the fact that humans became smaller after the Ice Age may have something to do with increased morbidity and declining mortality, where agricultural settlements "hugged" the Malthusian boundary more consistently than hunter-gatherers. In this sort of environment smaller individuals may have gained a fitness advantage because they required fewer resources to make it through the inevitable "starving times."* But more generally the fact that height seems widely distributed across the genome brings to mind the role that pleitropy might play. Genes of small effect in height may have larger effect in many other traits which are constrained from shifting in allele frequency too much. In other words, can it be that the G-matrix is somehow maintaining this quantitative trait? I'd be curious what those of you with a quantitative genetic background might have to say.... * Another obvious issue is that larger individuals may be more favored energenetically in cold environments. As the world warmed that constraint was released.