Olaf is a fearless opponent. He is much better at offensive combat but causes severe destruction unless dealt with. His castle is very poorly constructed, and it's easy to damage his economy by destroying his Granary. This is always placed outside his castle and I recommend targeting this with Trebuchets or Catapults at the largest distance possible. He persistently sends over his endless swarms of Axe Throwers and Berserkers who will do serious damage to your own castle unless you prepare yourself for them.
The best way to deal with them is to keep your buildings protected behind Stone walls where they cannot be reached. I also strongly recommend placing towers periodically along your wall and stationing them with crossbowmen. Place all troops in towers only so that firstly they can reach further, and secondly, they cannot be attacked by Olaf's Axe Throwers. They can come very close to your walls and will attack your troops stationed on them so by placing your own troops on Towers this will prevent them being annihilated.
Olaf only uses two troop types: Berserkers and Axe Throwers. Sieges are certainly no exceptions, and in addition to his regular waves of attacks designed to cause terror in your castle, you can expect a large number of Berserkers and Axe Throwers to attack you in Sieges. Again, the best way to protect yourself against these is to place as many towers as possible on your wall, preventing ladders from being placed, but more importantly preventing those Axe Throwers from causing havoc on any troops stationed on the wall.
The best troops to deal with Olaf's invasions are Crossbowmen who can easily take care of his troops who are not armoured. He attempts to destroy your walls using Catapults, uses Fire Ballistae to kill troops, and then sends Laddermen in afterwards, although having a good number of Crossbowmen should prevent anything getting close enough to attack your walls. He will also attempt to set fire to your wooden buildings using burning carts, so beware of this and try to keep them close to your castle with stone wall.
Turning to his castle, this is very poorly constructed - whichever design he picks - although it is usually the first one. It's made entirely out of wood which means that your troops can destroy it themselves without the need for any actual siege equipment. There is however, one issue with doing so, and that is the Axe Throwers which are stationed upon the wall. The rest of his defence appears to rely solely on man traps, killing pits as well as war hounds.
First, you should attack his Axe Throwers stationed on the walls with Archers, keeping them far enough back so they are not in range - they will get decimated in a matter of seconds if they are within range. Once this is done, you can destroy the war hound cages from a distance with catapults. Finally, move in and start to destroy his wall. Move into the Keep, and kill Olaf himself!