Last week, Facebook issued yet another denial that the company eavesdrops on its users to target them with adverts.

“Just not true,” said Rob Goldman, the company’s head of ad product, in response to an open query from podcast Reply All.

After writing about the denial, my inbox almost immediately began filling with missives from people insisting that Facebook must be lying.

One person wrote they never drink wine, but a friend of his mentioned a wine delivery service to his wife and he saw the ad the next day. Another insisted he had proved it to his friends once before. A third said she had met many users where eavesdropping was “the only option” for explaining why they had received the adverts they had.

What is it about this conspiracy theory that makes it the most persistent in tech?

Part of the explanation is that it’s really very hard for Facebook to prove that it isn’t doing what it’s accused of. The company’s apps will generally have permission to access the camera and microphone, because people film video and take pictures using Facebook, Instagram, WhatsApp and/or Messenger.

While you can tell from looking at the traffic to and from your phone – or even just from your mobile data usage – that Facebook isn’t uploading a livestream of everything you say and do, there are always other possibilities. Perhaps it’s doing the processing on-device? Or only listening for key phrases? Or uploading everything in a burst when you get on wifi?

Facebook doesn’t help itself. It has a long history of pushing the boundaries of what’s acceptable in order to increase ad revenue, from profiling the “ethnic affinity” of users (totally different from racial profiling, it swears) to merging user data from WhatsApp with its main service. With that history, it’s not a giant leap for many people to simply assume Facebook is lying in its denials.

But the sheer wealth of evidence many are able to summon to support the theory also demonstrates another element to its persistence: the true nature of algorithmic ad targeting is still not widely known.

The sheer volume of information the social network has about a typical user is difficult to comprehend. It logs every action of you and your friends, and a substantial proportion of browsing off-site thanks to its Facebook share button. It also has information provided by friends, such as that ex who uploaded her address book containing your phone number and your embarrassing teenage email address, allowing Facebook to work out that you know that high-school friend who you haven’t seen for a few years but who still has your older contact details.

And, apologies, but you aren’t that special. If you have had a conversation about a particular topic, then it’s unlikely you are the only one. While your conversation may have been held in person, a lot of others will have happened on Messenger or in Instagram comments. It’s not a stretch to assume that if a lot of London-based men aged 25 to 34, whose interests include stadium rock and panel shows on Dave, start talking about buying tickets to see the Foo Fighters, then other London-based men aged 25 to 34 are probably solid targets for the same advertising.

‘The sheer volume of information Facebook has about a typical user is difficult to comprehend.’Photograph: Lauren Hurley/PA

Part of the reason Facebook’s creepy levels of knowledge skirts below the radar is that it’s rarely used to its full extent with on-site adverts. There are enough advertisers out there with deep pockets and a desire to hit as many people as possible that the hyper-targeted adverts can be outbid. Facebook provides advertisers with the tools to get uncomfortably accurate targeting, but it’s up to the advertisers themselves to actually use them.

For a real picture of the extent of Facebook’s knowledge, the best place to turn is the section where it applies its vast banks of data in service of its own aims: the “people you may know” suggestions.

That section has outed sex workers, psychiatrists and family secrets, all using as much data as possible to find every single connection in your life and show you that they’re on Facebook. People you may know is also subject to its own, lesser, conspiracy theory: many who have been connected with people they would rather remain invisible to blame location tracking, a feature the company swears it doesn’t use for this purpose. Then there’s the possibility that Facebook shows you people who have been searching for you.

This is creepy on a personal level. My Facebook account has zero friends, manages one page with five likes, and follows one person – Mark Zuckerberg. Despite that, the site has still managed to link me with a bunch of fellow journalists, some friends of friends, someone I went to school with and the brother of an ex. As well as someone from the Household Cavalry Mounted Regiment, and a whole load of Turkish people, for some reason. It’s not perfect, but it’s still an alarming amount of insight to be gleaned from a site that I have been scrupulously careful to avoid telling anything of use.

And that’s the core of the problem. The unsettling ability of Facebook to make accurate guesses based on what feels like barely any information doesn’t match with what we think should be possible. But rather than updating our beliefs, the easier thing to do is turn to another thing that we know is possible: spying.

This will become an increasingly fraught issue as AI pervades more and more of our lives. Just think of the first time an AI security guard arrests someone for shoplifting in the changing room – leading to an accusation that the shop is spying on people getting dressed. Or Amazon sending you coupons for some shoes you broke in front of its Echo Show video screen, but based only on its expectations about how often someone like you breaks the heels on shoes like that. Will we resort to more conspiracy theories? Or will we confront the new issues head on?