Taylor was born Ida Estelle Taylor to a Jewish family in Wilmington, Delaware. Her father, Harry D. Taylor (born 1871), was born in Delaware. Her mother, Ida LaBertha Barrett (1874–1965), was born in Easton, Pennsylvania, and later worked as a freelance makeup Artist. Harry and Ida divorced in 1903, and Ida later married vaudevillian Harry J. Boylan, Estelle's stepfather. Estelle's sister, Helen Taylor (1898–1990), was cast in supporting roles in a few silent films of the 1920s.