The study objective was to investigate the effects of childhood residential mobility on older adult physical and mental health. In REasons for Geographic and Racial Differences in Stroke (REGARDS) Study, we used linear regression models to investigate if number of moves during childhood predicted mental and physical health (SF-12 MCS, PCS), adjusting for demographic covariates, childhood socioeconomic status (SES), childhood social support, and adverse childhood experiences (ACEs). We investigated interaction by age, race, childhood SES, and ACEs. People who moved more during childhood had poorer MCS scores, β = −0.10, SE = 0.05, p = 0.03, and poorer PCS scores, β = −0.25, SE = 0.06, p < 0.0001. Effects of moves on PCS were worse for Black people compared to White people (p = 0.06), those with low childhood SES compared to high childhood SES (p = 0.02), and high ACEs compared to low ACEs (p = 0.01). As family instability accompanying residential mobility, family poverty, and adversity disproportionately affect health, Black people may be especially disadvantaged.