Neighborhood socioeconomic status (SES) is associated with dietary habits among the residents, but few studies have examined this association separately among long-term residents and movers. We calculated cumulative neighborhood SES score weighted by residential time in each address over 6 years for non-movers (n = 7704) and movers (n = 8818) using national grid database. Increase in average neighborhood SES was associated with higher adherence to dietary recommendations in both groups. Among the movers, an upward trajectory from low to high neighborhood SES was also associated with better adherence. Our findings suggest high SES areas might offer healthier food environments than low SES areas.