Castle Ward has been the home of the Ward family since ca. 1570. Known originally as Carrick na Sheannagh and owned by the Earls of Kildare, it was bought by Bernard Ward, father of Sir Robert Ward, Surveyor-General of Ireland. The 850 acre walled demesne also dates from the 16th century. The Ward family built a succession of homes in their estate; Old Castle Ward, built about 1590 near to Strangford Lough, still survives, but a mansion built about 1720 by Judge Michael Ward was demolished about 1850, although some of the associated landscaping remains.
The architect of the current building, built during the early 1760s for Michael Ward's son Bernard Ward, 1st Viscount Bangor is unknown, although he may have come from the Bristol area, with which the Ward family had associations. It may have been James Bridges who practiced in Bristol between 1757 and 1763 and whose work there has some similarity to Castle Ward.
The property was inherited by a settlement made in 1748 by Bernard Ward's eldest son, Nicholas, 2nd Viscount Bangor, who was clearly insane. When his younger brother, Edward, died in 1812, leaving a young son, the youngest brother Robert took the opportunity to relocate the insane Nicholas into a smaller house in Downpatrick and strip Castle Ward of everything valuable. The house stood empty until the death of Nicholas in 1827, when it was inherited by Edward's son, now the 3rd Viscount. He and his descendants restored the building and its furnishings, but on the death of the 6th Viscount in 1950 the house and estate were given in lieu of death duties to the Government of Northern Ireland, who presented the house and its gardens to the National Trust in 1952.