Larsen was married three times. In 1953, he wed Actress Susan Cummings. They have one child. After their divorce, he married Actress Vera Miles on July 16, 1960. He was married to Vera Miles until 1971. Their son, Erik Larsen, was born in April 1961. After Larsen and Miles divorced, he married Trang Thu Nguyen in 1983. The couple had one child. The marriage ended when Larsen died, aged 82, in Santa Barbara, California.