(CNN) -- A police commissioner in a northern New England town says he won't apologize for calling President Barack Obama the n-word.

Robert Copeland, 82, sat with his arms crossed at a packed town meeting Thursday evening while a crowd of angry residents of predominantly white Wolfeboro, New Hampshire, called for his resignation.

Copeland acknowledged in an email to his fellow police commissioners that he used the racial slur to describe Obama, according to CNN affiliate WMUR.

Jane O'Toole, who moved to Wolfeboro four months ago, said she overheard Copeland use the word at a restaurant in March.

"It's not like I was eavesdropping. Mr. Copeland was being very loud," O'Toole told WMUR.

She wrote to the town manager about the incident, and Copeland replied with a letter to O'Toole.

"I believe I did use the 'N' word in reference to the current occupant of the Whitehouse (sic)," Copeland wrote. "For this, I do not apologize -- he meets and exceeds my criteria for such."

O'Toole said "the letter really took the wind out of my sails, really."

Copeland is one of three members of the police commission, which hires, fires and disciplines officers and sets their salaries. Two months ago he ran unopposed for re-election and secured another three-year term. The police commission plans to meet to discuss what action they should take against Copeland, but they didn't tell the audience Thursday where or when that meeting would take place.

CNN spoke with Wolfeboro Police Chief Stuart Chase, who said that Copeland can't be "fired" since he's an elected official. Chase told CNN that Copeland can either resign, or the board of commissioners can move to recall his election and hold a special election to replace him.

Since there was no resolution at the gathering, some of those attending the meeting later confronted Copeland outside.

"I admitted what I did. I made no bone about it," Copeland said to those who followed him as he walked to his car.

And he became angry when WMUR recorded the confrontation, referring to reporter Nick Spinetto as a "nosy individual,"

When Spinetto responded that he was a reporter doing his job, Copeland fired back, "I know what you are. You're a skunk. Goodbye,"

Wolfeboro, located in the scenic Lakes Region in the central part of New Hampshire, has around 20 African-American residents among its population of 6,300. The town was often a dateline in news reports during the 2012 presidential election, with GOP nominee Mitt Romney spending time with family and campaign aides at his vacation home on Lake Winnipesaukee.