Immune-checkpoint blockade (ICB) combined with neoadjuvant chemotherapy improves pathological complete response in breast cancer. To understand why only a subset of tumors respond to ICB, patients with hormone receptor-positive or triple-negative breast cancer were treated with anti-PD1 before surgery. Paired pre- versus on-treatment biopsies from treatment-naive patients receiving anti-PD1 (n = 29) or patients receiving neoadjuvant chemotherapy before anti-PD1 (n = 11) were subjected to single-cell transcriptome, T cell receptor and proteome profiling. One-third of tumors contained PD1-expressing T cells, which clonally expanded upon anti-PD1 treatment, irrespective of tumor subtype. Expansion mainly involved CD8+ T cells with pronounced expression of cytotoxic-activity (PRF1, GZMB), immune-cell homing (CXCL13) and exhaustion markers (HAVCR2, LAG3), and CD4+ T cells characterized by expression of T-helper-1 (IFNG) and follicular-helper (BCL6, CXCR5) markers. In pre-treatment biopsies, the relative frequency of immunoregulatory dendritic cells (PD-L1+), specific macrophage phenotypes (CCR2+ or MMP9+) and cancer cells exhibiting major histocompatibility complex class I/II expression correlated positively with T cell expansion. Conversely, undifferentiated pre-effector/memory T cells (TCF7+, GZMK+) or inhibitory macrophages (CX3CR1+, C3+) were inversely correlated with T cell expansion. Collectively, our data identify various immunophenotypes and associated gene sets that are positively or negatively correlated with T cell expansion following anti-PD1 treatment. We shed light on the heterogeneity in treatment response to anti-PD1 in breast cancer.