Spence was born in Mel Rose, Scotland, as the fifth child in a family of eight. In 1839, following sudden financial difficulties, the family emigrated to South Australia. Arriving on 31 October 1839 (her 14th birthday), on the Palmyra, at a time when the colony had experienced several years of drought, the contrast to her native Scotland made her "inclined to go and cut my throat". Nevertheless, the family endured seven months "encampment", growing wheat on an eighty-acre (32 ha) selection before moving to Adelaide.