Kennedy was born in 1919 in Edinburgh, the son of a career Royal Navy officer, Edward Coverley Kennedy, and his wife, Rosalind Grant, daughter of Sir Ludovic Grant, 11th Baronet. His mother Rosalind was a cousin of the Conservative Politician Robert Boothby, later Lord Boothby. He had two younger sisters, Morar and Katherine. Morar married the Playwright Royce Ryton in 1954. Katherine married Major Ion Calvocoressi in 1947.