How to /testfor if a player is sitting in a boat

One thing you could do is make an invisible armour stand ride any boats in the world, this would stop players being able to get in them. (Don't necessarily need to be armour stands, but armour stands don't emit particles when invisible and cannot be interacted with.)

Another thing you can do is set a scoreboard for riding a boat as so:

/scoreboard objectives add Riding dummy

Clock:

/scoreboard players set @a[type=Player] Riding 1

/scoreboard players set @a[type=Player] Riding 0 {Riding:{id:Boat}}

/kill @a[score_Riding=0]

This will kill all Players with Riding score of 0, which will only happen if they are riding a boat. The commands that need to be clocked must be clocked in that order. They can be activated at once, but ensure that they run in that order. Command blocks that are activated on the same tick are executed in the order lowest x to highest x, lowest y to highest y, lowest z to highest z.

Note: In 1.9, the Riding tag was removed in favor of Passengers. For players, RootVehicle:{Entity:{id:Boat}} should work. You will have to clock the second command in the clock section 5 more times for each of the boat types. You can look up entity ID's for these on Google, you should find them.

Note 2:

/scoreboard players remove @a[type=Player] Riding 1

This also works for first clock command if you set last clock command to

/kill @a[score_Riding_min=0]