I've done a bit of this sort of thing, too, within anime and boardgamer scenes. Vincent, here's my conclusion about the proper marketing strategy for non-rpg outreach:

The hobby convention for whatever hobby is dominantly a local phenomenon. The rest of the visitors are hardcore. This means that you'll get the same people coming in every year. What you need to do is focus on outreach over sales until such a time as you've actually convinced people that they want to spend money on this new thing. Takes a few years, certainly, simply because people are hidebound by nature. Of course this only makes sense for a convention that's close by to you and you enjoy visiting over the long term; forget things where your travel expenses outstrip your initial profits.

Outreach looks like this: you have a game that can be played as a fun pastime for free during the convention. You have some free pamphlets that direct the interested people to your chosen Internet information sources on the hobby you're selling. You also sell product, but that's just for the repeat customers and the rare first-timers who don't get scared by the price point. Over a few years you'll find that the germinating process you've started convinces people to spend some money on this thing, too, now that they've seen that it's a thing.

Of course, doing outreach has its expenses, which you should attempt to externalize: let volunteers run your free entertainment games, and let the convention provide you with free space for your free entertainment; no sense in you paying for the space you use to introduce the rpg hobby to people. If you can manage to have free GMs/faciliators and free space for them to work in, then you can have a functional crowd-handling workflow: you can catch people, give them a pitch, turn them over to the gaming department, and if they find that they like it, they can come back to you and make some purchases or not. Of course the return per customer will be small, but the point is, you don't have to spend a lot of time on each individual, because you've externalized your generic "here I'll play this game with you for half an hour to show you how fun it can be" to your not-for-profit general outreach department. Note that the outreach department doesn't have to worry about sales, they're here just because they love gaming and want to show the mainstream how cool it is.

If, for a given non-rpg convention I want to visit, I have to choose between the outreach department or running a sales booth, I'll go with the former: it's more useful to everybody involved, and I'll have a shot at getting my travel expenses paid by the convention, too, because now I'm a culture activist and not a salesman. It's simply more sensible to try to get people to try the new hobby by enabling them to try it for free. They'll be capable of buying e.g. game texts from the Internet afterwards, but they'll only do that if they already had a fully immersive and enchanting experience of play in the convention itself.

Note that most conventions are starved for programming relevant to the interests of whatever crowd they're attracting. With a little forethought you can set yourself and your GM crew up as a desirable addition to the program. Conventions actually pay to have people come in and do content for their visitors, so positioning your outreach more as a cultural good for the event than a sales opportunity for yourself might make much more sense.