The major and essential objective of pre-implantation development is to establish embryonic and extra-embryonic cell fates. To address when and how this fundamental process is initiated in mammals, we characterize transcriptomes of all individual cells throughout mouse pre-implantation development. This identifies targets of master pluripotency regulators Oct4 and Sox2 as being highly heterogeneously expressed between blastomeres of the 4-cell embryo, with Sox21 showing one of the most heterogeneous expression profiles. Live-cell tracking demonstrates that cells with decreased Sox21 yield more extra-embryonic than pluripotent progeny. Consistently, decreasing Sox21 results in premature upregulation of the differentiation regulator Cdx2, suggesting that Sox21 helps safeguard pluripotency. Furthermore, Sox21 is elevated following increased expression of the histone H3R26-methylase CARM1 and is lowered following CARM1 inhibition, indicating the importance of epigenetic regulation. Therefore, our results indicate that heterogeneous gene expression, as early as the 4-cell stage, initiates cell-fate decisions by modulating the balance of pluripotency and differentiation.