Rubin was a graduate of the Juilliard School of Music in New York. He was a member of the Saturday Night Live Band, with whom he played at the Closing Ceremony of the 1996 Olympic Games. As a member of The Blues Brothers, he portrayed Mr. Fabulous in the 1980 film, the 1998 sequel and was a member of the touring band.