The decisions by both International Monetary Fund director Christine Lagarde and Robert Birgeneau, former chancellor of the University of California, to withdraw as planned commencement speakers are only the latest in a rash of controversies this commencement season.

Other planned speakers, including former Secretary of State Condoleezza Rice, women’s rights advocate Ayaan Hirsi Ali, Attorney General Eric Holder, and Oscar-winning screenwriter Dustin Lance Black, have all either been disinvited from speaking or withdrew in the face of significant student protest.

The phenomenon isn’t new; it’s become such a rite of passage in the spring that some free-speech advocates have started calling the spring “disinvitation season.” But it’s a trend that some believe is growing, and that many observers worry is shifting college campuses away from being a free marketplace of ideas.

“It’s incredibly disappointing that commencement speakers get cancelled because of the controversial nature of their remarks,” says Ken Paulson, president of Vanderbilt University’s First Amendment Center and dean of the College of Mass Communication at Middle Tennessee State University. “We need more people to come in and raise hell on college campuses and provide challenging ideas and concepts.”

“By booking someone to speak at your commencement, you’re not issuing a blanket endorsement of who they are or what they do,” he adds. “Given the amount of boring commencement speakers we’ve all heard, you’d thing it would be refreshing to hear someone provocative.”

Ms. Lagarde, the IMF chief, was scheduled to speak at Smith College until a petition began circulating among students criticizing the IMF as “a primary culprit in the failed developmental policies implanted in some of the world’s poorest countries” and demanding that Lagarde be reconsidered as the commencement speaker.

On Monday, Lagarde withdrew.

On Tuesday, Mr. Birgeneau withdrew as commencement speaker for Haverford College in Pennsylvania. The controversy over Birgeneau started several weeks ago when more than 40 students and faculty wrote Birgeneau a letter, demanding that he meet nine “conditions” before speaking at the Haverford commencement. They cited the University of California’s decision to use police force on students participating in Occupy Wall Street protests in 2011. Those conditions included publicly apologizing and writing a letter to Haverford students explaining the events and “what you learned from them.”

Birgeneau responded tersely in a letter: “First, I have never and will never respond to lists of demands. Second, as a longtime civil rights activist and firm supporter of nonviolence, I do not respond to untruthful, violent verbal attacks.”

But on Tuesday he apparently changed his mind and backed out of the speaking engagement.

In both cases, the Smith and Haverford presidents expressed their disappointment with the decision but seemed to have little control over the outcome.

“Those who objected will be satisfied that their activism has had a desired effect. But at what cost to Smith College?” asked Smith President Kathleen McCartney in a statement. “I remain committed to leading a college where differing views can be heard and debated with respect.”

Haverford President Daniel Weiss had also pushed back against students’ demands, calling Birgeneau “one of the most influential and important higher education leaders in our generation” and admonishing students that their letter with demands “read more like a jury issuing a verdict than as an invitation to discussion.”

But despite his efforts to work with the protesting students, he was unable to come up with a solution.

In many instances, it’s a voluntary decision for speakers to back out, not a “disinvitation.” Last week, Ms. Rice also voluntarily dropped out after Rutgers protesters called her a “war criminal.” But many free-speech advocates say they amount to the same thing.

“People say, ‘These are people who withdrew,’ but I assume that when the pressure gets high, the first thing a university president would do is talk to the speaker and give them an opportunity to decline,” says Greg Lukianoff, president of the Foundation for Individual Rights in Education, which champions free speech on campus. “By the time you have students and faculties demanding that someone be disinvited, it’s already too late.”

Mr. Lukianoff cites the Ayaan Hirsi Ali case this year as an example in which someone refused to back down in the face of student protests, and Brandeis University ultimately caved to those protests instead by officially disinviting her. (She was to be a recipient of an honorary degree, not the primary commencement speaker.)

Protesters had criticized Ms. Hirsi Ali’s fierce condemnation of Islam, and Brandeis justified its decision by drawing a distinction between a speaker designed to provoke discussion and awarding an honorary degree, which might be viewed as a blanket approval of someone’s work. “We cannot overlook that certain of her past statements are inconsistent with Brandeis University’s core values,” the university said in a statement.

But Hirsi Ali issued a statement of her own condemning Brandeis’s decision. “Neither Brandeis nor my critics knew or even inquired as to what I might say,” she said in the statement. “They simply wanted me to be silenced.”

Lukianoff, like Mr. Paulson, acknowledges that the silencing of speakers is nothing new, but he says it may be getting worse. And he blames, in part, a culture on college campuses that so restricts free speech that students “think they have a right not to be offended.”

“We’ve been teaching a generation of students through speech codes, through censorship in some cases of really quite mild speech,… to think like censors,” says Lukianoff. “If you create an idea that people who have done or said things you dislike should never be allowed to speak on your campus, there’s kind of no bottom to that pit.”

Lukianoff says that the majority of speakers discouraged from campuses are social conservatives, but notes that that’s not always the case. Obama appointees like Eric Holder and Kathleen Sebelius have been protested, as have relatively apolitical people like actor James Franco.

In some cases, even when universities have insisted a speech be allowed to go on, protesters have succeeded in shutting it down in process. That was the case this past fall when students at Brown University managed to halt a speech (not a commencement address) by New York City Police Department Commissioner Raymond Kelly. Students angry about the NYPD’s “stop and frisk” policy essentially booed Commissioner Kelly off the stage, preventing a lengthy question-and-answer session that had been planned.

That “sent a really powerful message to university presidents and administrators that unless they really stand up to students and faculty members, they might figure out a way to shut down a speech no matter what,” says Lukianoff.

“I think we’re training people to seek out echo chambers,” Mr. Lukianoff worries. Instead, he says, students should be learning “that you should be skeptical when you’re too confident in what you believe, and should seek out smart people with whom you disagree.”