The development and progression of solid tumors is dependent on cancer cell autonomous drivers and the tumor microenvironment (TME). Cancer-associated fibroblasts (CAFs) in the TME possess both tumor-promoting and tumor-restraining functions. In the current study, we interrogated the role of αSMA+ CAFs in a genetic mouse model of metastatic colorectal cancer (CRC). Selective depletion of αSMA+ CAFs resulted in increased tumor invasiveness, lymph node metastasis, and reduced overall survival. Depletion of αSMA+ CAFs reduced BMP4 and increased TGFβ1 secretion from stromal cells, and was associated with increased Lgr5+ cancer stem-like cells (CSCs) and the generation of an immunosuppressive TME with increased frequency of Foxp3+ regulatory T cells and suppression of CD8+ T cells. This study demonstrates that αSMA+ CAFs in CRC exert tumor-restraining functions via BMP4/TGFβ1 paracrine signaling that serves to suppress Lgr5+ CSCs and promote anti-tumor immunity, ultimately limiting CRC progression.