A gradual decline in testosterone is a normal part of aging among men, however, a higher-than-normal drop in testosterone levels may be related to certain medical conditions or other various underlying factors, including medication side effects, pituitary or thyroid problems, depression, and excessive alcohol use.