Swift was formerly married to Novelist Margaret Drabble (1960–75). He is the father of one daughter, Rebecca (who died in April 2017), known for running The Literary Consultancy in London, and two sons, Adam Swift, an academic, and Joe Swift, a TV gardener.