Quiescent cancer stem cells (CSC) play important roles in tumorigenesis, relapse, and resistance to chemoradiotherapy. However, the determinants of CSC quiescence and how they sustain themselves to generate tumors and relapse beyond resistance to chemoradiotherapy remains unclear. Here, we found that SET domain-containing protein 4 (SETD4) epigenetically controls breast CSC (BCSC) quiescence by facilitating heterochromatin formation via H4K20me3 catalysis. H4K20me3 localized to the promoter regions and regulated the expression of a set of genes in quiescent BCSCs (qBCSC). SETD4-defined qBCSCs were resistant to chemoradiotherapy and promoted tumor relapse in a mouse model. Upon activation, a SETD4-defined qBCSC sustained itself in a quiescent state by asymmetric division and concurrently produced an active daughter cell that proliferated to produce a cancer cell population. Single-cell sequence analysis indicated that SETD4+ qBCSCs clustered together as a distinct cell type within the heterogeneous BCSC population. SETD4-defined quiescent CSCs were present in multiple cancer types including gastric, cervical, ovarian, liver, and lung cancers and were resistant to chemotherapy. SETD4-defined qBCSCs had a high tumorigenesis potential and correlated with malignancy and chemotherapy resistance in clinical breast cancer patients. Taken together, the results from our previous study and current study on six cancer types reveal an evolutionarily conserved mechanism of cellular quiescence epigenetically controlled by SETD4. Our findings provide insights into the mechanism of tumorigenesis and relapse promoted by SETD4-defined quiescent CSCs and have broad implications for clinical therapies. SIGNIFICANCE: These findings advance our knowledge on the epigenetic determinants of quiescence in cancer stem cell populations and pave the way for future pharmacologic developments aimed at targeting drug-resistant quiescent stem cells.