Causes of secondary hypogonadism include4: Kallmann syndrome, a genetic problem that also diminishes the sense of smell Isolated hypogonadotropic hypogonadism, a genetic condition that only affects sexual development but not the sense of smell Prior radiation, trauma, surgery, or other injury to the brain or pituitary Tumors of the brain or pituitary