Apolipoprotein E (APOE) plays a pivotal role in lipid including cholesterol metabolism. The APOE ε4 (APOE4) allele is a major genetic risk factor for Alzheimer's and cardiovascular diseases. Although APOE has recently been associated with increased susceptibility to infections of several viruses, whether and how APOE and its isoforms affect SARS-CoV-2 infection remains unclear. Here, we show that serum concentrations of APOE correlate inversely with levels of cytokine/chemokine in 73 COVID-19 patients. Utilizing multiple protein interaction assays, we demonstrate that APOE3 and APOE4 interact with the SARS-CoV-2 receptor ACE2; and APOE/ACE2 interactions require zinc metallopeptidase domain of ACE2, a key docking site for SARS-CoV-2 Spike protein. In addition, immuno-imaging assays using confocal, super-resolution, and transmission electron microscopies reveal that both APOE3 and APOE4 reduce ACE2/Spike-mediated viral entry into cells. Interestingly, while having a comparable binding affinity to ACE2, APOE4 inhibits viral entry to a lesser extent compared to APOE3, which is likely due to APOE4's more compact structure and smaller spatial obstacle to compete against Spike binding to ACE2. Furthermore, APOE ε4 carriers clinically correlate with increased SARS-CoV-2 infection and elevated serum inflammatory factors in 142 COVID-19 patients assessed. Our study suggests a regulatory mechanism underlying SARS-CoV-2 infection through APOE interactions with ACE2, which may explain in part increased COVID-19 infection and disease severity in APOE ε4 carriers.