"Follow your hosts" is a mission objective in Sequence 2 / Memory 03: Mister Walpole, I Presume?
Walkthrough
Just stay close to Woodes Rogers and Julien du Casse
If you wander too far, you risk desynchronization.
Also, when both templars and the Governor are in the room looking at the map, try to pickpocket all three of them to practice pickpicketing and complete the Optional Objective
