Crickhowell Castle was built in the late eleventh/early twelfth century by Bernard de Neufmarche or one of his vassals - perhaps Robert Turberville. Bernard was the son of a minor Norman Lord and hailed from Rouen in Normandy. At the time of the Domesday Book (1086), he was not recorded as holding any lands but he certainly came to England shortly after for William II granted him territories in Herefordshire. From here he acquired 'Marcher Lord' status allowing him to act in a quasi-legal capacity in the fractious Welsh border region as well as capturing any further territory he could. This saw him invade the Welsh Kingdom of Brycheiniog which was eventually carved up into the Lordship of Brecon. Crickhowell would have fallen into his sphere of influence and its location, on the banks of the River Usk which was a key communication and trade artery into central Wales, made it a strategic point for control of the region.

The initial structure was a simple earth and timber motte-and-bailey castle built around one mile to the north-west near Maeselyn on a spur of land between the Cwmbeth Brook and the River Usk. This was rebuilt in the early twelfth century on the current site, by which time it was certainly in the hands of the de Turberville family. It remained in their possession for over 150 years and was substantially rebuilt in stone around the 1270s by the then owner, Hugh de Turberville. These upgrades saw the castle evolve into a substantial fortification - the motte was now crowned with a stone shell Keep, two substantial gatehouses protected by conjoined towers were added and an additional gatehouse (complete with double drum towers) controlled access to the motte.

Hugh de Turberville was the last member of the family to own the castle and, when he died in 1293 without leaving a male heir, it passed to Sir Grimbald Pauncefote who had married Hugh's daughter, Sybil. After him it passed into the ownership of Roger Mortimer, Earl of March who appointed Garard Alisby as Constable from whom the structure got its alternative name, Alisby's Castle.

The castle was returned to the Pauncefote family in 1402 as the political stability of Wales shook from the rebellion of Owain Glyn Dŵr. The then owner - Sir John Pauncefoot - was charged by the King with bringing the defences of Crickhowell back to strength but his efforts proved inadequate as the castle was overrun, sacked and left in ruins by the rebels in 1403. The rebellion was eventually defused but the castle never regained its former status. It passed through various owners including Sir William Herbert (from Raglan Castle) but drifted into ruin and eventually much of its stone was quarried to support building works in the town.

ref: http://www.castlesfortsbattles.co.uk/crickhowell_castle.html