Most jerks don’t know they’re being jerky. That’s what makes them so annoying.

If you find some of your co-workers displaying any of these five signs, you may need to wake up and come to terms with the fact that you’re driving them nuts.

You Never Hear Anything Negative

It seems counterintuitive, but if no one ever says anything bad to you, they may think you’re a jerk, says Alexandra Levit, a career and workplace speaker who writes for the Fast Track blog. “One subtle sign that your co-workers (or direct reports) think you’re a jerk is that they never say anything negative to you. People who treat you with kid gloves often do so because they are afraid of you, and if they are afraid of you, you are probably being a jerk.”

Examine your behavior and see if there’s something you need to do to make people feel comfortable enough to share feedback — both positive and negative — with you.

There’s a Stealth Campaign Against You

When people avoid you or you hear rumors about yourself, you know someone’s out to get you. But what if the signs are more subtle?

“Avoiding you would actually be better than what some co-workers do when they think you’re a jerk,” says Rosalie Robinson, a partner at Consilium Consulting Group. “The stealthy campaign to torpedo you can be damaging beyond words. The subtle sigh when someone new asks what it’s like working with you. The raised eyebrows when someone talks about your work followed by a chortle. The off-handed remark, ‘Well if you have to work with him, then remember to…’” All of these can mean people don’t like working with you for some reason — and you need to find out what it is.

You’re ‘Forgotten’

When you’re not included in internal communications and emails that may directly affect your work, it can be damaging to your career, Robinson says. “‘Forgetting’ to include you in a meeting about a topic or project directly related to your work” is also a problem.

Most serious, though? Something that, on the surface, looks like a good thing, but in reality is meant to push you aside: “The sign you probably have gone from ‘jerk’ to someone who has been voted off the island is reassignment to the ‘important’ high-level project that really isn’t destined to go anywhere but it does help move you out of the way while your replacement is groomed and put in place.”

Your Chickens Come Home to Roost

What goes around often comes around in the workplace. “Having a hard time getting time with the VP (or getting a meeting room booked) yet your colleagues seem to be able to with relative ease?” says motivational speaker and trainer Dave Fleming. “Guess what, you ticked off Betty, the awesome admin.”

People in administrative or helping positions deserve as much respect as anyone else — sometimes more — because they’re the ones who can help you get things done. “You may not have intentionally turned her into your personal errand runner, but you did in her mind,” Fleming says. “And you have not made an effort to get to know her on a personal level. You should have, she’s cool.”

You’re Attacked for Succeeding

“Everyone has a story about themselves and their career,” says Don Maruska. Some are “victim stories,” such as “I could be doing wonderful things, but [my boss won’t let me, we don’t have a budget for it, I got passed over for someone less qualified, etc.].” If you’re able to transcend these constraints, everyone else with a victim story starts to feel uneasy, Maruska says.

“When people resist challenges to the stories they hold, they attack the person whose example contradicts them,” he says. But there are ways you can use your success can help others. Reaching out and showing others how to work more efficiently can make the whole team stronger.