In 1728, he married Anne Forster, daughter of John Forster, Chief Justice of the Irish Common Pleas, and his first wife Rebecca Monck. He then went to America on a salary of £100 per annum. He landed near Newport, Rhode Island, where he bought a plantation at Middletown – the famous "Whitehall". It has been claimed that "he introduced Palladianism into America by borrowing a design from [William] Kent's Designs of Inigo Jones for the door-case of his house in Rhode Island, Whitehall." He also brought to New England John Smibert, the British Artist he "discovered" in Italy, who is generally regarded as the founding father of American portrait painting. Meanwhile, he drew up plans for the ideal city he planned to build on Bermuda. He lived at the plantation while he waited for funds for his college to arrive. The funds, however, were not forthcoming, and in 1732 he left America and returned to London. He and Anne had four children who survived infancy: Henry, George, william and Julia, and at least two other children who died in infancy. William's death in 1751 was a great cause of grief to his father.