Returning to the Fleet Marine Force in 1985, he commanded rifle and weapons companies and served as the operations officer of 3rd Battalion, 4th Marines. During this period he received the Leftwich Leadership Trophy. In 1988, Allen reported to the U.S. Naval Academy, where he taught in the Political Science Department and also served as the jump officer and jump master of the academy. In 1990 he received the William P. Clements Award as military instructor of the year.

Allen reported to The Basic School as the director of the Infantry Officer Course from 1990 to 1992 and was subsequently selected by the Commandant of the Marine Corps Fellows Program, which provides "assignment to either a prominent national foreign policy/public policy research institution or to a national security studies program at a selected university".[5] Allen has served as a special assistant on the staffs of the 30th Commandant of the Marine Corps and the commanding general of Marine Corps Combat Development Command. In 1994, he served as the Division G-3 Operations Officer for the 2nd Marine Division and subsequently assumed command of 2nd Battalion, 4th Marines; re-designated as 2nd Battalion, 6th Marines. This unit served with JTF-160 in Operation SEA SIGNAL during Caribbean contingency operations in 1994, and as part of the Landing Force of the 6th Fleet in Operation JOINT ENDEAVOR during Balkans contingency operations in 1995–1996.

He commanded The Basic School from 1999 to 2001, when he was selected in April 2001 to return to the Naval Academy as the deputy commandant. Allen became the 79th Commandant of Midshipmen in January 2002, the first Marine Corps officer to serve in this position at the Naval Academy.[6] In January 2003 Allen was nominated for appointment to the rank of brigadier general.[7]

Allen's foreign awards include the Mongolian Meritorious Service Medal, First Class; the Polish Army Medal in Gold; the Taiwan Order of the Resplendent Banner; the French Legion of Honor; the Order of Merit of the Italian Republic; Honorary Officer of the Order of Australia; and the Ghazi Mir Bacha Khan Medal.

After leaving the military, Allen continued to work as an adviser to Secretary of State John Kerry and former Defense Secretary Chuck Hagel. In this capacity he worked closely with Israeli, Palestinian, and Jordanian interlocutors on the Israeli-Palestinian Peace Plan.

On September 11, 2014, the Obama Administration announced that Allen would coordinate international efforts against the Islamic State in Iraq and Syria.[25] He was named as the Special Presidential Envoy for the Global Coalition against ISIL, a position he would hold for 15 months. Allen's diplomatic efforts grew the coalition to 65 members.[26][27]

On October 23, 2015, the White House announced his departure from the post. He officially departed that role on November 12, 2015, and was succeeded by Brett H. McGurk.[28]

After government service, Allen joined the Brookings Institution as senior fellow and co-director of the Center for 21st Century Security and Intelligence.[29] On October 4, 2017, Allen was named the seventh president of Brookings, succeeding Strobe Talbott.[30]