Speyer was born in Milwaukee, Wisconsin, the son of Germaine M. and Ernest A. Speyer. According to a 1998 profile in The New York Times, "[Speyer's] mother is Swiss, and his Father comes from one of the old Jewish families of Frankfurt" (however, there is only very distant connection to the Speyer banking family, if any); his Father, a shoe manufacturer, fled Germany in 1939, established a Business in Milwaukee, before moving to New York when Jerry was three months old. Speyer grew up in a cultured German-Jewish household on Riverside Drive. He graduated from the private Horace Mann School. At Columbia University, he majored in German literature and joined Zeta Beta Tau, a Jewish fraternity. "Speyer was one of those people who were solid, and even solemn, at an age when others are still flailing and unsure of themselves." Speyer graduated from Columbia College in 1962 and received an MBA from Columbia Business School in 1964.