These are almost certainly the mud cells of the Red Mason bee, Osmia bicornis. They look mostly vacated so will be from a previous year. They will make their aerial nest in any suitable hole/cavity often door locks entering through the keyhole, but I have also seen channels in double glazed window frames and Velux windows used.

We have had the same problem and for us leaving the bees alone was a mistake. It's a long story so I'll try to keep it as short as possible. We were delighted when the red mason bees nested in the Oxford Bee Co. nest boxes we bought for them many years ago. After a cold winter we were amused to find that the next spring some bees moved in under the tiles of a new extension we had just built, who could blame them. For several years bee numbers increased massively and two years ago we noticed that they were starting to nest in Velux windows. As every time we opened the window we got showered with mud, pollen, eggs and larvae we so we left well alone. We also noticed a grey dust appearing in the room on a table under the window. But the "dust" was mobile, presumably mites living on the bees or bee debris around the nests. But as the weather was so bad last year I didn't get around to investigating any further.

A week or so ago I thought I ought to see what was happening and found the gap between the openeing window and the frame was filled with nests for about two thirds of the way round. It became obvious that some damage had been done to the frame/window so reluctantly I cleared out all the nests, scrapped and then washed off the mud, treated the wood with a wood preservative before filling the holes and revarnising. Interestingly although some the bees were still nesting under the surrounding tiles they did not attempt to recolonise the wooden parts of the frame. In contrast in another window despite clearing out the nests a persistent bee has revisited the site on a daily basis for 5 days rebuilding nests each day. I will be trying a "repellent" (WD-40 has been suggested) rubbed onto the wood in an attempt to persuade the bee to nest elsewhere. Somewhere I read that adult bees tend to nest at the site from which they hatched, which might explain the persistent nature of this bee! ( I would have attached some photos but couldn't work out how to do it!)

Photos attached show nests on frame and some of the holes bored by something! Thanks to my daughter for showing me how to fix it. Pity about the spelling mistakes though. Nick