Cassel was born in Detroit, Michigan, the son of Pancretia Ann (née Kearney), a performer, and Seymour Joseph Cassel, a nightclub owner. Cassel's early career was tied to fellow actor John Cassavetes, who is informally part of his clan of actors. He made his movie debut in Cassavetes' first film, Shadows, on which he also served as associate Producer. In 1961 he co-starred with Cassavetes in Too Late Blues and 1962's The Webster Boy. He also appeared in The Lloyd Bridges Show in the episode "A Pair of Boots" directed by his friend Cassavetes. Cassel appeared on such popular programs as Twelve O'Clock High, Combat! and The F.B.I. He also appeared as "Cancelled", one of Colonel Gumm's henchmen, in the 1960s Batman TV episode "A Piece of the Action", which also featured guest stars Van Williams and Bruce Lee as The Green Hornet and Kato, respectively.