The museum is based in a building from 1723 which was designed by crown prince "Christian (VI). It served as residence for the estate manager (godsforvalteren) at Hørsholm crown estate which belonged under "Hirschholm Palace. It later served as residence for the county manager (Amtsforvalteren). Hirschholm Palace was torn down in about 1810 but a number of buildings associated with the palace. The museum's neighbouring building, known as Vognremisen, used to house the palace's carridges. It now contains the local archives. The current rectory on the opposite side of the street was originally a cow stable. Its neighbouring buildings, which now houses the "Danish Museum of Hunting and Forestry, were originally barns and stables.
In front of the museum stands the Stolberg Column which commemorates the agricultural reforms that took place at Hørsholm Estate between 1860 and 1761. They were introduced by count Christian G. Stolberg and were the first of their kind in Denmark. The memorial, a commission from Christian VI's widow, Queen Downer "Sophie Magdalene, was raised in 1766 to designs by the French sculptor "Nicolas-Henri Jardin.