Setzer was born April 1959 in Massapequa, New York. He started on the euphonium and played in jazz bands when he was in school. He found a way to hear jazz at the Village Vanguard, though as he got older he got interested in rock, punk, and rockabilly. He was a member of the Bloodless Pharaohs and the Tomcats, which he began with his brother, Gary. The Tomcats became the Stray Cats when double Bassist Lee Rocker and Drummer Slim Jim Phantom joined. In 1980, thinking they might have more success in England than in America, they sold their instruments to pay for airplane tickets and flew to London.