Analyzing antigen-specific T cell responses at scale has been challenging. Here, we analyze three types of T cell receptor (TCR) repertoire data (antigen-specific TCRs, TCR-repertoire, and single-cell RNA + TCRαβ-sequencing data) from 515 patients with primary or metastatic melanoma and compare it to 783 healthy controls. Although melanoma-associated antigen (MAA) -specific TCRs are restricted to individuals, they share sequence similarities that allow us to build classifiers for predicting anti-MAA T cells. The frequency of anti-MAA T cells distinguishes melanoma patients from healthy and predicts metastatic recurrence from primary melanoma. Anti-MAA T cells have stem-like properties and frequent interactions with regulatory T cells and tumor cells via Galectin9-TIM3 and PVR-TIGIT -axes, respectively. In the responding patients, the number of expanded anti-MAA clones are higher after the anti-PD1(+anti-CTLA4) therapy and the exhaustion phenotype is rescued. Our systems immunology approach paves the way for understanding antigen-specific responses in human disorders.