Neoadjuvant immunotherapy represents promising strategy in the treatment of esophageal squamous cell carcinoma (ESCC). However, the mechanisms underlying its impact on treatment sensitivity or resistance remain a subject of controversy. In this study, we conducted single-cell RNA and T/B cell receptor (scTCR/scBCR) sequencing of CD45+ immune cells on samples from 10 patients who received neoadjuvant immunotherapy and chemotherapy. We also validated our findings using multiplexed immunofluorescence and analyzed bulk RNA-seq from other cohorts in public database. By integrating analysis of 87357 CD45+ cells, we found GZMK + effector memory T cells (Tem) were relatively enriched and CXCL13+ exhausted T cells (Tex) and regulator T cells (Treg) decreased among responders, indicating a persistent anti-tumor memory process. Additionally, the enhanced presence of BCR expansion and somatic hypermutation process within TNFRSF13B + memory B cells (Bmem) suggested their roles in antigen presentation. This was further corroborated by the evidence of the T-B co-stimulation pattern and CXCL13-CXCR5 axis. The complexity of myeloid cell heterogeneity was also particularly pronounced. The elevated expression of S100A7 in ESCC, as detected by bulk RNA-seq, was associated with an exhausted and immunosuppressive tumor microenvironment. In summary, this study has unveiled a potential regulatory network among immune cells and the clonal dynamics of their functions, and the mechanisms of exhaustion and memory conversion between GZMK + Tem and TNFRSF13B + Bmem from antigen presentation and co-stimulation perspectives during neoadjuvant PD-1 blockade treatment in ESCC.