James married twice: to Donna Lea Beach from 1948 to 1950, with whom he had one child, and to Laurie Harper, from 1951 until her death in 2015, with whom he had five children. He resided in Gladstone, Oregon, and died from complications of diabetes on April 15, 2017, aged 96.