Testes contain two distinct Leydig cell populations during development: fetal and adult Leydig cells (FLCs and ALCs, respectively). ALCs are not derived from FLCs, and it is unknown whether these two populations share common progenitors. We discovered that hedgehog (Hh) signaling is responsible for transforming steroidogenic factor 1-positive (SF1(+)) progenitors into FLCs. However, not all SF1(+) progenitors become FLCs, and some remain undifferentiated through fetal development. We therefore hypothesized that if FLCs and ALCs share SF1(+) progenitors, increased Hh pathway activation in SF1(+) progenitor cells could change the dynamics and distribution of SF1(+) progenitors, FLCs, and ALCs. Using a genetic model involving constitutive activation of Hh pathway in SF1(+) cells, we observed reduced numbers of SF1(+) progenitor cells and increased FLCs. Conversely, increased Hh activation led to decreased ALC populations prepubertally, while adult ALC numbers were comparable to control testes. Hence, reduction in SF1(+) progenitors temporarily affects ALC numbers, suggesting that SF1(+) progenitors in fetal testes are a potential source of both FLCs and ALCs. Besides transient ALC defects, adult animals with Hh activation in SF1(+) progenitors had reduced testicular weight, oligospermia, and decreased sperm mobility. These defects highlight the importance of properly regulated Hh signaling in Leydig cell development and testicular functions.