To take over the ship you will need to kill a set number of the crew, the captain and 2 Officers. To make your life easier, focus on using the Swivel to kill the Officers or the Captain if possible. They are not too hard to handle in combat but it helps end the boarding action a lot sooner.
Once on board, focus on your targets. Start with the paired officers who are about the midship. Get up there and take them out. After that move onto the Captain. You can generally find him around the helm. Go and take him down next.