Neuronal variability is a reflection of recurrent circuitry and cellular physiology, and its modulation is a reliable signature of cognitive and processing state. A pervasive yet puzzling feature of cortical circuits is that despite their complex wiring, population-wide shared spiking variability is low dimensional with all neurons fluctuating en masse. Previous model cortical networks are at loss to explain this variability, and rather produce either uncorrelated activity, high dimensional correlations, or pathologically network behavior. We show that if the spatial and temporal scales of inhibitory coupling match known physiology then model spiking neurons naturally generate low dimensional shared variability that captures in vivo population recordings along the visual pathway. Further, top-down modulation of inhibitory neurons provides a parsimonious mechanism for how attention modulates population-wide variability both within and between neuronal areas, in agreement with our experimental results. Our theory provides a critical and previously missing mechanistic link between cortical circuit structure and realistic population-wide shared neuronal variability.