Terry Downes was born in Paddington, London. His father Richard worked as a mechanic, and his mother Hilda in a department store. Downes boxed as a junior for the Fisher ABC. He moved with his parents to the United States in 1952, while still a teenager, to live with his trapeze Artist sister Sylvia, who had lost an arm in a traffic accident, going on to serve in the US Marine Corps from 1954-56, being recruited after boxing against them for the YMCA. In the marines he won several amateur trophies, including the all-services championship and the Amateur Golden Gloves. He missed out on selection for the US Olympic team, being ruled ineligible on residence grounds, and after his term of Service, he returned to London and turned professional.