Letters of Recommendation - Assorted Observations

Funnily enough, I've slowly been working on a post on what it is like to write letters of recommendation for people at different stages of their careers, as part of a series of posts I'm hoping to write on aspects of being a Professor. Then along comes Julianne's (rather innocuous and humorous in my opinion) post, and all kinds of craziness breaks out in the comments. So I'm still working on the other post, but I thought it might be useful to provide some unvarnished facts about letters that I've picked up over the years. Most certainly everything that I'm about to write is anecdotal, based on my own direct experiences, and acquired through conversations with colleagues. Nevertheless, I have carefully read literally many thousands of letters of recommendation for graduate students, and still more thousands for postdoctoral and faculty positions combined. So feel free to weigh that any way ...