Variation in gene expression is an important feature of mouse embryonic stem cells (ESCs). However, the mechanisms responsible for global gene expression variation in ESCs are not fully understood. We performed single-cell mRNA-seq analysis of mouse ESCs and uncovered significant heterogeneity in ESCs cultured in serum. We define highly variable gene clusters with distinct chromatin states and show that bivalent genes are prone to expression variation. At the same time, we identify an ESC-priming pathway that initiates the exit from the naive ESC state. Finally, we provide evidence that a large proportion of intracellular network variability is due to the extracellular culture environment. Serum-free culture reduces cellular heterogeneity and transcriptome variation in ESCs.