Numerous anatomical and brain imaging studies find substantial differences in brain structure between men and women across the span of human aging. The ability to extend the results of many of these studies to the general population is limited, however, due to the generally small sample size and restrictive health criteria of these studies. Moreover, little attention has been paid to the possible impact of brain infarction on age-related differences in regional brain volumes. Given the current lack of normative data on gender and aging related differences in regional brain morphology, particularly with regard to the impact of brain infarctions, we chose to quantify brain MRIs from more than 2200 male and female participants of the Framingham Heart Study who ranged in age from 34 to 97 years. We believe that MRI analysis of the Framingham Heart Study more closely represents the general population enabling more accurate estimates of regional brain changes that occur as the consequence of normal aging. As predicted, men had significantly larger brain volumes than women, but these differences were generally not significant after correcting for gender related differences in head size. Age explained approximately 50% of total cerebral brain volume differences, but age-related differences were generally small prior to age 50, declining substantially thereafter. Frontal lobe volumes showed the greatest decline with age (approximately 12%), whereas smaller differences were found for the temporal lobes (approximately 9%). Age-related differences in occipital and parietal lobe were modest. Age-related gender differences were generally small, except for the frontal lobe where men had significantly smaller lobar brain volumes throughout the age range studied. The prevalence of MRI infarction was common after age 50, increased linearly with age and was associated with significantly larger white matter hyperintensity (WMH) volumes beyond that associated with age-related differences in these measures. Amongst men, the presence of MRI infarction was associated with significant age-related reductions in total brain volume. Finally, statistically significant associations were found between the volume of MRI infarcts in cubic centimeters and all brain measures with the exception of parietal lobe volume for individuals where the volume of MRI infarctions was measured. These data serve to define age and gender differences in brain morphology for the Framingham Heart Study. To the degree participants of the Framingham Heart Study are representative the general population, these data can serve as norms for comparison with morphological brain changes associated with aging and disease. In this regard, these cross-sectional quantitative estimates suggest that age-related tissue loss differs quantitatively and qualitatively across brain regions with only minor differences between men and women. In addition, MRI evidence of cerebrovascular disease is common to the aging process and associated with smaller regional brain volumes for a given age, particularly for men. We believe quantitative MRI studies of the Framingham community enables exploration of numerous issues ranging from understanding normal neurobiology of brain aging to assessing the impact of various health factors, particularly those related to cerebrovascular disease, that appear important to maintaining brain health for the general population.