This retreat by investors, who until six months ago seemed to have a boundless appetite for risky mortgages, is forcing lenders to concentrate on more traditional types of loans that they can keep as investments or sell to Fannie Mae and Freddie Mac, the government-sponsored providers of mortgage funding.