In February 1935, Ray's father, known by the nickname Speedy, passed a bad cheque in Alton, Illinois, then moved to Ewing, Missouri, where the family had to change their name to Raynes to avoid law enforcement. Ray left school at the age of fifteen. He later joined the U.S. Army at the close of World War II and served in Germany, although Ray struggled to adapt to military life.