The "Saxton House" was located at the main intersection in the village. It was built by John Saxton and nephew, James Waddington of oak lumber from the new sawmill. Previous structures were made of locally quarried limestone. Bob LaFollette lived in this house from 1862-1870 when his widowed mother married Saxton.
As noted by the Wisconsin Historical Marker placed on this site, "This is the surviving portion of Argyle's oldest existing house, built by John Z. Saxton and James S. Waddington in 1850." The house was moved from its orginal site in 1905 and continued to serve as a private residence; it was acquired by Historic Argyle in 2000, moved to a new foundation, and subsequently restored.