The frustration in the room was palpable on Capitol Hill on Monday afternoon where a group sexual assault victim advocates and law enforcement experts in sex crimes met to talk about how the police and college administrators could better together to handle campus sexual assault.

The group had assembled for a roundtable hosted by Senator Claire McCaskill (D-Missouri), one of a team of three senators working to draft legislation that would address growing concerns about campus safety. As TIME wrote in a recent cover story, criticism of how college campuses have dealt with sexual assault has risen this year with accusations that officials have been sweeping the problem under the rug. But the tension over universities’ mishandling of these issues begs the question of why college administrators are expected to deal with these cases in the first place. A passive observer might wonder, shouldn’t these serious crimes be dealt with by the police? The answer, it turns out, is that administrators and police will have to work together to address the problem.

Yet the difficulty of building more effective partnerships became clear as the conversation unfolded at the round table today. Victim advocates articulated fears about anything that would make the relationship between law enforcement and the schools overly formal. For the advocates, doing right by the victim often means respecting her or his wishes not to report the crime to the police and even telling the victim about the possible downsides of the criminal justice system– which can lead to a months-long process that might threaten a victim’s confidentiality. In response, law enforcement officers explained how difficult it can be to pursue criminal action when they don’t collect evidence from the victim early in the process, making it difficult for them to get repeat offenders off the streets.

The question of when and how to involve the police in campus sexual assault is a salient one for administrators and politicians as they work together to overhaul the system of reporting and preventing these incidents. Alexandra Brodsky, a student at Yale law school and an organizer at Know Your IX, a grassroots organization that educates sexual assault survivors about their civil rights in the college setting, illustrated the tension beautifully during the discussion when she said: “When I reported violence to my school, I was told not to go to police. But I never would have told [the school] if I knew I was going to be forced into that option.”

If colleges are going to do a better job of handling sexual assault, college administrators are going to have to work together with police chiefs. But that collaboration is difficult, particularly because victims (especially those in college), are reluctant to report their assaults to the police.

In a 2007 study funded by the U.S. Department of Justice that surveyed 5,446 undergraduate women and 1,375 undergraduate men at two large public universities in the South and the Midwest, just 2% of sexual assault victims incapacitated by drugs or alcohol and just 13% of “physically forced” victims reported the crimes to law enforcement (that’s why increased reporting at colleges is, counter-intuitively, a good sign).

So, why don’t victims go to the police? Every victim is different, but there are a few common themes that ran through the testimony at the hearing and through conversations with experts in the field.

1. They don’t want anyone to know. In the round table, confidentiality was the most often sighted goal of both victim’s advocates and police officers and prosecutors who work most closely with victims. Survey data backs them up. Contrary to Washington Post columnist George Will’s bizarre theory that reporting sexual assault could confer a “coveted status” for victims, research shows that college victims don’t report sexual assault to the police because they don’t want anyone to know. In the 2007 study, 42% of the “physically forced” victims who did not report the incident to the police said it was because they “did not want anyone to know.” Nearly half of the victims gave the same answer in an earlier survey (also funded by the National Institute of Justice) that randomly surveyed 4, 446 women attending two or four year colleges during 1997.

Victims, especially those in college, know that reporting rape comes with a social risk, especially when the perpetrator is someone they know. At a small or midsize college, the rapist is likely to be part of the victim’s social circle. “I’ve seen this in every single case. The victim lose friends or becomes a social pariah. If you report on a really small campus, its really difficult to re-integrate after you report,” says Bruno.

Interestingly, even as the attitude towards victims has improved over the last several years in the broader culture and by police, self-blame and shame has persisted among victims, leaving them just as unwilling to come forward. Years ago, says Scott Berkowitz, the founder and president of the Rape Abuse & Incest National Network (RAINN), the most common reason victims gave for not reporting was: “‘I think I won’t be believed. I think I will be blamed.’ We hear that less often. Now it is much more common to hear: ‘I want to keep this private. I don’t want people to know. I’m embarrassed.'”

2. They don’t understand what constitutes rape. The 2007 survey showed that just over 35% of victims said that they didn’t report to law enforcement because it was “unclear that it was a crime or that harm was intended” (44% gave that same answer in the earlier 1990’s study).

The victims’ confusion does not mean that all of these crimes fell somewhere in the gray. More likely, their confusion reflects shame, denial, and internalized misconceptions that rape is always perpetrated by a stranger and involves physical violence, when often, rape happens between acquaintances and involves alcohol, threats, or other kinds of coercion.”Victims don’t often identify it as a crime because they know the person, they trusted the person, sense of denial or disbelief that it happened,”says Colby Bruno, Senior Legal Counsel at the Victim Rights Law Center, who represents victims of sexual violence in civil matters, with particular expertise in representing college students.

3. They are afraid the police won’t believe them. In the more recent 2007 study, 21% of physically forced victims and 12% of incapacitated victims did not report because they didn’t think the police would take the crime seriously and 13% of forced victims and 24% of incapacitated victims feared the police would treat them poorly. Victims have also reported that their colleges discouraged them from reporting.

Victims aren’t wrong in their perception. According to research funded by the U.S. Department of Justice, only 18% of reported rapes result in a conviction.

4. They don’t know how much control they will have after they report to the police. Victims are afraid of going through a public rape trial because of how awful it can be for the victim. Media portrayals of rape trials show how often they are about the victim’s character and credibility. Given the low rate of conviction, victim’s naturally decide it isn’t worth the risk. Unfortunately, there is wide discrepancy between how prosecutors and police officers in various jurisdictions handle sex crimes. Some will give broad power and control to the victim, while others may pursue the case against the victim’s wishes. Predicting those outcomes are difficult for victims and the advocates who advise them (a theme reflected in today’s round table). According to Carrie Hull, a detective with the Ashland Police Department in Ashland Oregon who attended the round table, said reporting was up 106% from 2010 to 2013 after the implemented a program called “You Have Options,” designed to decrease barriers in reporting, which gives women three options when reporting to police – to give information only, to trigger a partial investigation, or to trigger a complete investigation that will be referred to the prosecutor.

Bruno says that prosecutors are more likely than they were a few years ago to follow the victim’s wishes to drop a case. Still, it is impossible to predict the outcome, and victims are rightly scared by what they know of the system.

As I’ve reported before, rapists are very often repeat offenders. The best way to ensure that more victims report is to continue to create flexibility in the system for victims and change the wider culture so that victims will feel supported. Being the victim of rape will never reach George Will’s imagined “coveted status,” but at the very least, we have to work together to ensure it’s not a shameful one.