By 25 March 1651 Jan van Riebeeck had received instructions from the Lords XVII (the directions of the Dutch East India Company) to look for a suitable site to build a fort on the Cape. It would be named Fort de Goede Hoop and the drawing of the Fort accompanied the instructions. The aim of the Fort was to protect the refreshments station from local and foreign attacks as well as to confirm and maintain the Dutch claim on Table Bay.

The very next day after he dropped anchor on 5th April 1652 Jan van Riebeeck chose the most favourable site for the fort. As previously planned by the Lords XVII, the chosen site has to contain a river which could be lead through the moat of the fort. This would form an integral part of its defense. After the fort have been measured out on the ground construction commenced on the 10th April 1652. The Fort was ready for defense by 31st December 1653 and was completed in full on 25th March 1656. The Fort was demolished in May 1674 and the castle has taken over its function. The five bastions names are Leerdam, Oranje, Nassau, Catzenellenbogen and Buuren.

The ground where the fort would be built was according to a report sent to the Lords XVII, suitable for mason work. It was realized that the walls to be erected from the sandy and stony ground would not be very permanent. By working branches and undergrowth in between the sods, it was hoped to improve the walls. There were doubts however even with this method, about preventing the rain from washing away the soil. The fort was not built without problems. Not only the rain in winter cause damage, but the southeasterly wind caused havoc.

As a result of the state of war between England and Netherlands (first English Dutch naval war 1652-1654), the fort was hastily built to safeguard the Dutch at the Cape against a British attack. The results of the hasty construction soon became evident. The clay walls could not withstand the rain and on 1 August 1654 one side of the bastion Walvisch collapsed, as well as sections of the ramparts. Fortunately, the peace was declared between England and the Netherlands and the fort never suffered an attack during its existence.

In 1683 Commander Simon van der Stel created a new entrance to the castle moving stonework from around the old sea entrance to the new portal and building a beautiful bell tower above it, using imported yellow dutch ‘klompies’ (yselsteen). The earlier entrance according to surveyor J. Wittebol was on the seaside of the castle, with stonework surround similar to that of the present entrance. The stones for the work were brought from the Robben Island but the stonemason remains unidentified.

Inside the split pediment is the symbol of the state's general. It consists of a lion holding seven arrows representing the united provinces of Netherlands and is surmounted by a crown. The six shields in the frieze represent the chambers of the VOC (Dutch East India company) Amsterdam, Rotterdam, Delft, Hoorn, Middelburg, and Enkhuizen. Deterioration of the stone and its attachment to the wall necessitated extensive restoration in 1999. A new wooden door was designed to replace the recent iron gate.

The castle gable was constructed together with van der Stel’s new entrance n 1683. The painted teak trophy is a copy of the original damaged one, which is now displayed in the castle museum. The two figures were sculpted to match those shown in the drawing done by Lady Anne Barnard in 1800. The iron rods, on which they have been fixed into the gable, were struck by lightning in 1984 and the gable was shattered. It was subsequently restored and strengthened.

Block A is about 75 m long, was built against the rampart between the bastions Leerdam and Buuren before 1682. It originally contained the main guard, clothing shops and the grain stores on the ground floor, and quarters for the officers on the first floor. A teak colonnade supported the first floor terrace in front of this building but these columns were replaced in the 19th century by the present cast iron pillars. These pillars together with the balcony which has deteriorated badly were removed and repaired during the 1990 restoration. The pillars were painted to simulate marble as indicated on the drawing by Lady Anne Bernard.

Block B. This, the first Castle Building was constructed between Oranjes and Leerdam bastions before 1682 and is the only one with exterior walls of stone. A central double staircase led to a church a council chamber and the “comptoir” or secretariat on the right. On either side of this, secondary stairs led to​ houses for the captain and highest ranking officials of the VOC. Governor van der Stel lived in the captain's house for a short period before his own quarters were constructed in the castle against the cross wall which later divided the courtyard into two.

A cross-wall completed by 1695 divides the castle courtyard into two. Block F- consists of a central stone wall with buildings against it on either side. The wall runs from the Catzenellbogen bastion in the east to the center of block B in the west. A central arched portal connects the front and back courtyards and inside it is the original castle well. 17th century sundial painted on the wall above the entrance indicated it a time for routine activities in the castle. The governor's quarters were built to the right of the portal to face the front courtyard and secondo’s the house to the left.

In 1695 governor Simon van der Stel completed the house for himself against the stone wall dividing the castle courtyard. The entrance of the top of the staircase led to stair hall with offices to the left and three dining rooms overlooking the back courtyard​ and connected to the front with the passage through the stone wall. On the first floor were three bedrooms at the back and further offices looking out over the front courtyard. These were changed to form one longer room in 1936 and became known as the Lady Anne’s Ballroom.

The 18th-century​ double stairway with its ornate was probably designed by the German sculptor Anton Anreith who was also responsible​ for the stucco work on the cornice. The balcony originally led to a new church and council chamber, but the entrance presently leads to the William Fehr Collection. The wrought iron balustrade is probably ​17th century but moved to its present position from an older balcony to the right. These balconies were important for official announcements​ and public auctions

The governor's kitchen wine and other storerooms were situated in the basement. Most of this wing was demolished and rebuilt​ in 1936 but wrongly proportioned windows had to be remade during the restoration in the 1980s. He also made provision for the extensive cellar in the basement to store his wine. The arch door to these cellars have been bricked up over the years and was opened and restored in the 1980s. When Governor van der Stel obtained permission to build a new bakery in 1705, he constructed this double storey building at right angles to and directly behind his quarters, thus creating a private governor’s garden.

Here he built a square pool with a central dolphin fountain, surrounded by a colonnade with officers for his clerks. In 1860 the British army pushed the balustrade into the water filed the pool with soil, and turned the clerks’ offices into stables for officers then living in the refurbished bakery. The buildings and pool were build on their original foundations in the 1980s and the balustrade reconstructed using the excavated original pieces as models. The designs for the fountain and planting pots were based on the 17th century examples, and the pots filled with plants known to the van der Stels.

The dungeon and adjoining torture chamber are both constructed of solid stone walls with ventilation duct instead of Windows. the torture instruments usually consisted of a rack, thumbscrews a brasier for heating branding irons and an iron ring in the ceiling for a rope with which the prisoner could be hold up and dropped again.

The prisoner was put to the question, “are you guilty”? and then tortured until he admitted his guilt or was released. The senior clerk sat at the table in this room to record the proceedings. During the restoration, the stone dividing wall between the two rooms which had been demolished was built on its original foundations.

Part of the castle was used as a prison during the Second Boer War. The former cells remain to this day. Previously home to South African Army in the Western Cape today the Castle houses a mechanized infantry unit, the Cape Town Highlanders Regiment.

In 1969 the Castle was declared national monument. It is the ​best preserved example of a Dutch East India Company fort.