In this paper, we discuss usability lessons in designing physical activity games for the elderly. Physical activity games in this study refer to game-based activity that can enhance the elderly’s experiences in doing physical exercise and improve their physical well-being. We learned these usability lessons through the usability testing of different physical activity games including commercial games as well as the games produced by the Gamified Solutions in Healthcare (GSH) project. The discussion is based on the findings from the observational study, interview, and questionnaires in the usability testing. These usability lessons are insightful and useful for our future game design and development as well as for researchers working on game usability for the elderly.