Joel Palmer was born in Elizabethtown, Ontario, Canada, on October 4, 1810. His parents, Hannah Phelps and Ephraim Palmer, were American Quakers. When he was two years old, they moved to New York's Catskill Mountains in response to the War of 1812. He received only three months of formal education in elementary school. In 1822, when Palmer was 12, his parents indentured him to the Haworth family for a period of four years. When he gained his freedom, he moved to Bucks County, Pennsylvania, to work on canals and bridges.