The human brain sets us apart from the rest of our primate relatives. Humans have large brains compared with other primate species. Specific areas of brain, especially within the neocortex, have recently undergone rapid expansion, with most of this increase occurring within the last 500,000 years. Although understanding the causes and consequences of this brain expansion has long been a preoccupation, we still have limited evidence that points to the exact forces that drove the increase in the human brain. Across primates, however, there is an increasing body of evidence which links brain architecture to social cognition. In this chapter, we will review the patterns of gross brain evolution in vertebrates as a whole, and in primates in particular. We follow this discussion with a critical evaluation of what total brain size means in terms of cognitive function. Finally, we will focus on the substructures in the brain associated with social tasks, and whether there is evidence for exceptional increases in these areas in the human brain.