I used rubber (i think they were neoprene off ebay )washers and they worked really well. You may have to lube them a bit so they don't deform as they twist when you tighten the bolt head. Or PU sealer would work well.

That's what I'd try. Aim to make a smooth hole just big enough to get the bolt through (maybe a paper punch would do it?). You could also try covering the whole inner end (nut & washers) with hot melt glue from a glue gun (it's just low melting point plastic so fuses well to the boat without melting it significantly).

I use PTFE tape, same as plumbers use on sealing pipework; if its good enough for radiators under pressure and heat it's definitely good enough for a kayak. Less than 30 pence from your local screwfix;