The castle was built during the reign of Hakang V, who used it as his royal palace. The fire of 1527 destroyed the castle. When Christian IV was in office, Hans Steinwenko, a famous architect, expanded and decorated the castle on a large scale. By 1600 the castle had become a Renaissance-style palace, and around 1630 it had added walls. It was not until 1815 that its defensive function as a fortress was diluted. When members of the royal family died, they were buried in the castle.