In 1951, Diller married Helen Samuels (born 1928), the daughter of Jewish immigrants from Poland, and fellow University of California, Berkeley graduate who he met in college. Her Father was a clothing salesman and shop owner. They have three children: Brad Diller, Ron Diller, and Jackie Safier, President of Prometheus. The Dillers live in Woodside, California.