Asa R. Slocumb built Vermont House in the 1790s shortly after moving to Shelburne, Vermont. His descendants continued to live there until 1929 when they abandoned the house. According to legend, Slocumb's first home was a log cabin which probably resembled the museum's Settlers House. Over time he constructed the wood frame and clapboard exterior that would become Vermont House's original façade around and over the log cabin. After completing the new exterior, he dismantled the cabin and carried the logs away.
When the Shelburne Museum acquired Vermont House in 1950, the building's clapboard siding and interior walls had deteriorated beyond repair. While preserving its basic structure, the museum built a new façade with stone from a Shelburne Falls gristmill, randomly laying the stones using a technique known as scatter-stone, and reconstructed the interior with salvaged feather-edged boards from Vermont, New Hampshire, and Connecticut houses. Although the museum preserved the structure's four original fireplaces, it changed the floor plan so that it would include a study and a larger kitchen.
