Once through the gates immediately tag any approaching Overseer with a Sleep Dart then knock them out. There is one in front of you and one to the left on the back porch of the manor. Climb on up the nearby wall and deal with the Gunner playing lookout. Climb out onto the overhang of the front porch. Arm yourself with Firecrackers and lob them to the West. This will draw the guards away from de Fayet. The moment Adewale begins his approach the Governor will start running.
The Governor will be running for the pier from his mansion. Use Smoke Bombs to slow your pursuers. When he makes it to the fort by the beach, he will stop running. The problem is there are a lot of people here. If you are still after the Blunderbuss optional objective, this is a great place to pull it off.
To get at the Governor, it is important to save him for last. Use Smoke Bombs to thin out the rest of the foes. Once it is done, you can disarm him to take the branding iron and kill him with it. If you do not care about that optional objective then just go for a kill steak that involves him.