Ruffin served on the board or was a director of such firms as Wachovia Bank and Trust Company, General Telephone of the South East, and the Durham and Southern Railway and was a member of the Durham board of Woodward, Baldwin, and Company of New York City.