Castle was born to Erby G. Noblett, Sr. and Myrtle A. Noblett (née Brown) in Pampa, Texas. Her mother was one-sixteenth Quapaw Indian. The Nobletts moved to Fort Worth, Texas, then Phillips, subsequently a ghost town in Hutchinson County, Texas, prior to relocating to Long Beach, California. At the age of nine, Castle contracted pneumonia. Her brother, Erby Noblett, Jr. (1927–1992), taught her trick riding and later became a police officer in Long Beach.