The failed integration of North Korean refugees in South Korea has not been improved, despite many studies and measures created to address the issue. A different approach is required to give a new insight into alleviating the problem. Early studies demonstrated that social capital, resources accessible through social networks, generated benefits; it played a crucial role in the integration processes of refugees. However, as indicated in previous research, North Korean refugees had poor relationships with South Koreans. It is necessary to identify the reasons for the poor relationships to enhance them. Therefore, this study explores the obstacles preventing the refugees from building good relationships with South Koreans. This study conducted semi-structured interviews with eight participants consisting of seven North Koreans and one South Korean. Findings show that the refugees’ relationships with South Koreans were hindered by their different mindsets and frequent job changes. Their relationships were also hampered by South Koreans’ ignorance and cultural and linguistic differences. This study provides valuable indications for how to improve the refugees’ relationships with South Koreans.