Josiah and Lucy Caldwell bought this house on Elm Street in 1822. The Caldwells believed in the moral power of home and family, and their beliefs inspired a radical mission. The Caldwells were local leaders in the international struggle to end slavery.
The Caldwells had no children. In 1836, they brought their young niece Margaret into their home and later adopted her. They were a family of reformers. Josiah led the Ipswich Anti-Slavery Society. Lucy held meetings of the Ipswich Female Anti-Slavery Society in the parlor. Margaret attended the Ipswich Female Seminary, which prepared her for a career as a teacher.