William Smith and later his children had extensive holdings in Haverstraw dating from February 1734. The family owned the ridge now known as Treason Hill down to the narrow passage where the County Park is today. The manor house was known as The White House, built by Loyalist son William in 1774 and used often before hostilities.
William left Haverstraw in 1776 for Albany and never returned to live, the house being generally vacant for the duration. This is the house in which Benedict Arnold and Major John André planned their conspiracies, it being available to his brother the scoundrel Joshua Hett Smith in September 1780. No image of the White House has ever been found yet is often mentioned in correspondence of the day.
Just north was the home of brother Thomas Smith, who owned and occupied it during most of the hostilities, practicing law and teaching law students including Aaron Burr. This is the house that was torn down in 1929 forever being misidentified as the Treason House.
Just below were the home and farm buildings occupied by William Smith‘s sister Martha Smith Hay and her husband Colonel A Hawkes Hay. Twice her home and farm were burned by the British, payback for Colonel Hay’s rebellious nature.
The property and landing ran from now Route 9 W to just north of the iron bridge over the Minisceongo Creek. The Hay’s used the White House several times during the war when they were burned out, and occupied it full time soon after the capture and trial of John Andre.
The White House was Washington’s Headquarters twice during the war and was Hay’s Orange County Militia HQ when in residence. After the war, the Smith Family would petition the State and new Federal government for reparations following the confiscation of their property, and much of the family would emigrate to South Carolina at the turn of the century. Colonel A. Hawkes Hay is buried in the Old Presbyterian Churchyard on Calico Hill.
In 1803 a ten acre riverfront tract was developed by nephew William Smith bounded by today’s Main, South, West, and First Streets that would eventually be the Village of Warren. In 1809 the White House burned down, being replaced soon after with the Fraser-Hoyer House that stands today, this property being listed in the National Register of Historic Places.
An obscure Letter to the Editor in the Messenger in 1854 from Epsilon makes clear the White House is in fact the location of the deed and there was no one better than Rich Koke to figure it out.
Link below to the definitive story in South of the Mountains
