Rich Lawler has an insightful comment:
It's interesting to note that a few of the most insightful observations about the evolutionary process were first promulgated verbally, then later proven mathematically (unlike H-W equilibrium). These include runaway sexual selection (first adumbrated by Fisher, then shown mathematically possible by Lande and Kirkpatrick), the handicap principle (first adumbrated by Zahavi, then--finally--shown to be mathematically possible by Grafen), and, of course, natural selection (first adumbrated by what's-his-face, then formalized by Wright, Fisher, and later Price, among others). And of course, all of these topics were debated back-n-forth until the math made them more clear.
I think the key here is the reference to evolutionary process, dynamics which span time and space beyond intuitive conceptualization. Verbal "lock & key" models are often sufficient to communicate the biophysical processes at work when an enzyme and substrate interact. A graphical illustration of a biophysical process is clear as the visual mapping is can be precise and accurate; the primary actors here are material. Not so for something as abstract as "fitness." Verbal descriptions of moments about distributions of abstract concepts are not sufficiently precise to allow for fruitful theoretical inference beyond the most elementary level. Of course, a non-mathematical idea can, and usually does, serve as the seed for future growth of formal theory. But when the subject is by its nature complex formalism is often simplest route.