The black tiles of the Jerwood Gallery pay deference to the wooden boards of the surrounding buildings. Photograph: Ioana Marinescu

I have to declare an interest: the Stade, Hastings, is one of my favourite places on earth. A wedge of land between cliffs and sea, it is the base of the town's fishing fleet, and so its shingle is covered with beached boats and the apparatus of nets, and a town of shacks of to-the-point design. Better, it holds arrays of black wooden towers lined up at the base of the cliff like the apparatus of a medieval siege. Called "net shops", they are unique to Hastings and look mute and enigmatic, but were actually functional places to store nets.

The place smells of just-landed fish sold in little shops, or the fried and battered sort in some fine fish-and-chip restaurants. Ice cream and rock vendors are inserted into odd slivers of space, and somewhere above are the ruins of a Norman castle, reached by a funicular railway with pompous crenellated termini. Another piece of toy engineering, a tiny railway, weaves through the detritus on the beach. When I was aged about five, living near here, it was one of the most delightful rides imaginable.

It's a working place that actually works, unlike so many redundant and re-purposed industrial zones, with bits and pieces of seaside entertainment thrown in. It is improvable. So when I first heard that the Jerwood Foundation were building a gallery here to house their collection of art, I felt some of the doubts expressed by some vociferous local objectors, whose party-pooping posters – "NO Jerwood on the STADE" – feature prominently around the brand new building. Did the Stade need something that might accelerate its inevitable bijoufication and take it closer to being another example of Shoreditch-on-Sea? Worse, was it going to get a dose of "culturally led regeneration", the formula whereby blocks of overpriced flats huddle round a quasi-Guggenheim, with less-than-zero benefit to local people?

As it turns out, the architects of the gallery, Hana Loftus and Tom Grieve of HAT Projects, try hard to do none of these things. The building doesn't try to dominate, overwhelm or out-dazzle its surroundings, but nor does it patronise them. It has a black cladding material in deference to the wooden boards of the net shops, but it doesn't make the mistake of mimicking them – the surface of the gallery is of large glazed tiles with a rich oily sheen. It is horizontal and cuboid where the towers are vertical and pointed, but slips easily between them, making courts and yards similar to those already formed by the net shops. It is not embarrassed by the stuff and clobber around it, and does not embarrass them. It does not demand, as some cultural institutions do, that everything rise to the level of gloss set by the new building.

‘The building doesn't try to dominate.’ Photograph: Ioana Marinescu

In its plan, it observes a lack of ceremony about boundaries typical of the Stade, where the beginning of the shingle, or the zone of the boats, or the miniature railway lines, occur unannounced. They are just there. Similarly you enter the gallery on the level of the pavement, through a glass-walled lobby and, with a minimum of kerfuffle, straight into the temporary exhibition space.

The latter is not air-conditioned – daringly so, as museums usually have to achieve exacting standards of temperature and humidity control, without which major collections will not lend them their treasured works. These rules, understandable in the case of old master paintings, make less sense for contemporary works that might recently have been hanging in imperfectly conditioned commercial galleries, and survived. The Jerwood worked with Tate to make sure that they would not be endangering valuable artefacts, and now find that the Arts Council, formerly sticklers for the old rules, will lend them pieces.

If this sounds like a matter for heating-and-ventilation nerds, its effects can be directly felt. Apart from the benefits of sustainability, there is also no need for the sealed buffer zones that go with air conditioning, which would take away the immediacy of the space. And it's nice not to breathe plastic air.

Walking on terrazzo in warm pink, you pass a glass-walled courtyard on the left and reach a series of galleries housing the permanent collection, mostly of 20th-century British artists – Nicholson, Brangwyn, Sickert and others. It includes many who, at places like St Ives, worked among the scents of salt and fish that permeate the Stade. They are shown in a series of domestically scaled galleries, placed over two floors and linked by a staircase, with well-placed windows connecting you back to views of the beach, boats and net shops. The building is revealed through wandering, like a stroll on the beach taken indoors.

This is about it, apart from a cafe that faces both the horizon and an open space where bonfires, May Day festivals and other bacchanals are held. The Jerwood is a simple and straightforward place for viewing art which, after the hellish, art-phobic spaces of the country's last new gallery, Firstsite in Colchester, is a relief. In one or two places it doesn't quite hit the right notes – the stairs are on the clunky side and the courtyard, for now, is a little desolate – but mostly it's a building that honours its context and its content and creates new pleasures in the connections between the two.

Its simplicity owes something to the fact that the Jerwood Foundation is a privately funded charity dedicated to promoting the arts, and that the £4m project is paid for with its own money, plus the donation of free land – a former coach park – by Hastings borough council. There was nothing from the National Lottery, and there was therefore no need for the sort of gesticulating "world-class" architecture that is often thought necessary to help with fundraising from the lottery and other sources.

Fears that it would overwhelm the Stade, architecturally speaking, have proved misplaced, and it's fortunate that Hastings council now lacks the funds to tart up the surroundings. Rather, the Jerwood is a beautiful addition to the many juxtapositions – of cliff and sheds, work and distraction, ancient castle and seaside tat – that make this part of the coastline what it is. Its scale is small enough that it can co-exist rather than dominate. As to whether it will attract more urban trendies with funny facial hair, well yes it will, but that could be used as an argument against putting good art and architecture almost anywhere.