Domestication has a long history. It predates the invention of writing by thousands of years. In the history of biology the study of domestication is closely connected to the emergence of experimental and theoretical biology out of the shadow of natural history. Chapter I of The Origin of Species is preoccupied with animal breeding. The various modifiers of selection, artificial, natural and sexual, are to some extent human constructions. When the constraint of human preferences and needs are removed feral populations of dogs and pigs tend to shift back toward an ancient modal wild type. I hesitate to use a term such as "primitive" because of its pejorative connotations. Wolves after all are more intelligent than dogs, and this is probably true of most domesticates in relation to their wild ancestors. This makes sense, human intelligence to some extent serves as a substitute for the reasoning skills of our domesticated animals, and since the brain is an energentically expensive organ perhaps there is a natural "relaxation" of selective constraint? Of course despite being duller in a general sense, domestic dogs do have special adaptations. They can read human mental states more subtly than much more intelligent animals, such as chimpanzees. Lapdogs, despite their often bizarre morphology and obnoxious behavior clearly have an adaptively fit niche. Working dog breeds, such as those who herd sheep, retain greater autonomous intelligence. And so forth. An interesting point brought up by the power of selection to reshape dogs bodily and behaviorally, and their differences from the wolf parental populations from which they derive, is how much insight we can gain of human evolution. The Nobel Prize winning ethologist Konrad Lorenz suggested that humans were "degrading" due to a process of self-domestication. Human cranial capacities may exhibit a long term secular increase, but I have heard that they may have peaked during the Last Glacial Maximum, and have been subject to decline as is true of size in general. The study of the genetics of domestication is not simply one of natural history, it is a fertile applied science. One of the most fascinating pure research programs has been that of the famous Russian "domestic foxes," who are derived from wild fox lineages but have over time been selected for traits which have turned them into analogs of domestic dogs. A new paper which explores similar dynamics in rats just came out, Genetic Architecture of Tameness in a Rat Model of Animal Domestication:

A common feature of domestic animals is tameness - i.e. they tolerate and are unafraid of human presence and handling. To gain insight into the genetic basis of tameness and aggression, we studied an intercross between two lines of rats (Rattus norvegicus) selected over more than 60 generations for increased tameness and increased aggression against humans, respectively. We measured 45 traits, including tameness and aggression, anxiety-related traits, organ weights, and levels of serum components in more than 700 rats from an intercross population. Using 201 genetic markers, we identified two significant quantitative trait loci (QTLs) for tameness. These loci overlap with QTLs for adrenal gland weight and for anxiety-related traits, and are part of a five-locus epistatic network influencing tameness. An additional QTL influences the occurrence of white coat spots, but shows no significant effect on tameness. The loci described here are important starting points for finding the genes that cause tameness in these rats, and potentially in domestic animals in general.

The basic outline was that the original rat population was separated into two breeding programs. Two lines were developed which exhibited opposite tales of the distribution in "tameness." They did truncation selection to shift the median trait value of the population over the generations, but eventually the populations stopped responding to selection. Why? Probably because the underlying standing genetic variation which they were drawing from was exhausted. The ability to recreate different cat and dog morphologies from extant variation suggests that novel mutations aren't necessary to drive this sort of quantitative trait evolution. Nevertheless after a number of generations trait value stopped changing because the underlying allele frequencies had been driven to fixation (likely in disjoint frequencies between the two populations in question). At this point the researchers noted which regions exhibited the greatest allele frequency differences, and genotyped them in a subsequent linkage study. The two lines which were the products of selection, the "tame" and "aggressive" populations, were crossed to produced hybrids, and the hybrids themselves were crossed. The results were what one would expect, as the hybrids (F1) exhibit midparent values and the variation of the original parent populations reappeared in the third generation (F2). In addition to the wide range of phenotypes in addition to tameness that they looked at, they were curious about correlations. Here is a chart which illustrates the associations (or lack of):

Blue is negative association, and red positive. The diagonal red line is of course the association of the trait with itself, which is going to be 1. These results were somewhat disappointing I suspect, it seems that they were looking for an association between white coat color and tameness. Domestication and neoteny often result in coat pattern changes, probably due to an underlying pleiotropic process. The association between adrenal gland size and tameness is suggestive because there are neurological models which might emerge out of the intersection:

Alternatively, a causal relationshipmight exist between tameness and adrenal gland size. For example, sudden increases of plasma corticosterone, which is produced by the adrenal cortex, promote aggressive behavior in rats (Kruk et al. 2004), while chronically high levels of glucocorticoids seem to inhibit aggressive behavior in several vertebrate species (Summers et al. 2005). However, postmortem corticosterone levels did not map to any locus linked to tameness or aggression. It is thus equally possible that other hormonal activities of the adrenal glands are directly or indirectly linked to the alleles that affect tameness in the rats.

Finally, they focused on two moderate effect quantitative trait loci (QTLs):

Two significant QTL for tameness...The strongest of these (termed ''Tame-1'') is located at 58 cM on chromosome 1. The difference in tameness between homozygous genotypes at Tame-1 corresponds to 20% of the difference between the tame and the aggressive line (1.2 of 6.4 units of PC1). Tame-1 explains .1% of the phenotypic variance in tameness. The second locus (''Tame-2'') is located at 78 cM on chromosome 8. Both the tameness difference between homozygous genotypes (10% of the line difference) and the portion of residual phenotypic variance in tameness it explains (2.3%, are about half of those of Tame-1.

These two QTLs are probably the largest in the trait of tameness. They're not as big as those for skin color (e.g., SLC24A5 explaining ~35% of between group variance), but much larger than for height (where "large effect" loci are on the order of 1% or less). They note that the genetic architecture of the trait, that is, lots of smaller effect QTLs, probably explains the inability to find strong associations between multiple traits and genes. They might have lacked the power in this design to detect weaker correlations of small effect which summed together might have resulted in phenotypic associations. Here's a figure which illustrates these two QTLs:

I won't even get into the details of the fact there was some gene-gene interaction, epistasis, in terms of how the trait values varied. Just goes to show that there's a lot of detailed nuts & bolts to the quantitative architecture of the phenotype. Cite:Genetics, April 10, 2009, doi:10.1534/genetics.109.102186