View from the Vacay.ca Top 50 Restaurants Chair: The Lessons, The Challenges, The Results

Disclosure: I am the 2015 Vacay.ca Top 50 Restaurants in Canada Chair, which means it’s my responsibility to ensure that the voting guidelines are adhered to, help and manage our team of excellent regional chairs as they work with our judging team. I provide assistance to the Executive team in ensuring that this special project is completed within our deadline, and am available to the general population to address any questions or concerns relating to the selection process.

What is Vacay.ca’s Top 50 Restaurants in Canada list?

Since 2012, Vacay.ca has hosted the Canada’s Top 50 Restaurants list, an annual ranking of the nation’s restaurants, providing an unbiased source of places food experts consider are the top places to eat. The project became reality as a response to Canadian businesses, too often overlooked in the global culinary arena. Many of those involved were familiar with the scene, and believed there was enough great works in the country that should be applauded, and voluntarily put in the time and capital to produce something that our own country could relate to.

Aim

The purpose is simple: to acknowledge, celebrate, and promote the great work that hard-working chefs, restaurateurs and food entrepreneurs are doing in Canada. To promote culinary tourism within Canada (because there is just so much to discover, appreciate and enjoy in our vast and beautiful country).

The process is fair, and inclusive of any food serving business operating in Canada. There is no discrimination based on price, genre, location, or ambiance. Any restaurant is eligible and can receive votes.

Evolution of the judging panel

It’s been an ongoing evolution of our methodologies as we attempt to find the most ideal method to properly assess what’s happening across the country.

Now in its fourth year, and fourth incarnation, Vacay.ca’s Top 50 has seen our panel grow from select members of the culinary industry (chefs, restaurant owners, and purveyors), food and travel writers, food loving friends of the panel, to also including the general public’s voice (which included 10,000 additional votes). More recently, via feedback, we tailored it back towards food experts, and this year, to well traveled food experts that had to fit within a select subset of criteria in order to qualify as a judge. More below.

What we’ve learned from others

Full disclosure: I’m an academy member/panelist on a number of national and international restaurant recognition surveys. Without disclosing which one (something that’s superfluous to mention, save for the fact that I’ve personal experience with the different survey processes involved; also, “anonymity”), I’ve come to form an opinion on which methods I like and find effective, and what other ones can be improved upon.

Using some of the more prominent examples recognized by many diners, here’s what we’ve learned and how we’ve built our 2015 guidelines.

While crowd sourced data, like that compiled by Zagat is a good reflection of general popularity, it’s less reflective of what might come from an experienced dining base. Aka those restaurants deemed to be top or ‘best’ don’t tend to shine vs. what’s popular and accessible (factors that don’t necessarily make a place delicious).

Well traveled diners who have an established reputation are often great references make dining decisions from.

In contrast to guides like Michelin, we are a startup without the financial means to actually hire critics (or a critic that would travel across the country and eat at the 90,000 restaurants. Within the year). I find Michelin of value because it’s a standardized guide. While it might not be au courant to trends, etc., there is an expectation for those restaurants that receive recognition (even if there’s discrepancies between the restaurants that rank Europe and Japan vs. US (adjust by 1 star) and, Hong Kong (adjust by 2 stars)). This latter fact is an issue: although there is standardization within a country, it is not the case across the world, nor does it assess restaurants outside their regions of interest.

Similarly, well traveled, reputable diners or critics are a great reference to draw information from. Even if you don’t agree with the person’s taste, you at least have a benchmark to measure all those opinions shared from. Notice the emphasis on well traveled: that exposure and experience helps those individual make educated conclusions about where those restaurants lie on a scale of excellence.

EnRoute‘s Best New Restaurant in Canada receives its nominees from a panel of regional specialists, but it is the (enviable) task of one critic to dine at all those spaces and draw his/her conclusion of which rank the top. Not to dismiss the challenge of this process, but it is a smaller pool and less arduous (or expensive than trying to cover old and new restaurants all over the country).

There’s also the San Pellegrino sponsored World’s 50 Best – a source I also refer to as a guide indicating where members of industry directing most of the niche conversation, are flocking to right now. It is not reflective of the general palate, nor dining needs. Every year the list comes out, there seems to be as much anticipation for the results, as there is to tear it down.

Perhaps it’s how people are hung up on the word ‘best’ – subjective to the user – but I’ve always called the list a guide to “where people are going to right now.” It’s just a reflection of where the 900 or so judges have traveled in the last 18 months, and perhaps where people’s interest lie when it comes to dining choices. But it certainly is not a competition of skill or knowledge or other things that would make one ‘the best’ at their craft.

Which brings me to the Opinionated About Dining guide, my go to resource (outside of trusted palates, many who are fellow OAD voters). It’s a system that I feel is trustworthy. It’s also Nate Silver approved. *(For those who might not appreciate the significance, I would encourage following the link.)

It’s within this guide that the concept of “destination dining” first arised – people who travel specifically to eat well, or vice versa – to eat at some of the best places known, these diners would deliberately head out of their way to experience the anomalies.

A guide like OAD has the mission to help such diners make decisions not based on popular opinion, but by a trusted (or in this case, experience is weighted) set of diners who cast their votes, that are ranked (from “worthy of travel” to “not recommended”). The weighting is what distinguishes the contributors: those who have dined more, at what the founder has established as being the restaurants that matter, who have done so extensively, with good track record, get a bigger say. So despite the fact that diner A and diner B might have varying experiences a the same restaurant, the fact that diner A is a more trusted palate, that opinion weights a little more.

It makes sense. We all do the same when we ask a trusted source for recommendation (even if it’s a critic’s pick in the paper, if that’s your go to), vs. a general list.

From these established examples, we built our current constitution.

How our system works

When someone asks you to tell them about two things you like, but one is your most treasured and the other you simply like, how do you express this?

Verbally you might say, “A is good, but B is incredible, life changing/an epiphany/reminded me of my gran/made me cry.”

In our voting system, it is permissible via a quantitative means. That’s to say, that within the list of restaurants our judges nominate, they can distribute points to emphasize which restaurants they like, and which they really like. Simply ranking from one to 10 permits the assumption that the difference between two consecutive businesses is small, but what happens if you think #1 is vastly superior to #4, and #2 and #3 although your preference over #4, still paled in comparison to your chosen #1?

Our multi-layered system helps us suss out the absolute favourites of the listed lot due to the total points distributed from all the votes cast. Single votes for the restaurants also shows us the popularity of a restaurant.

Judges were also not permitted to vote stack their restaurant choices: a minimum of 5 restaurants were to be selected (maximum was 10. Not all judges name 10 restaurants). Each judge had 100 points to distribute, but no more than 20 points were permitted per a restaurant. Finally, only 80% of the votes cast per ballot was allowed to be within the judge’s region (aka if the judge wanted to vote for restaurant all outsides his/her region, that’s totally fine).

As the Chair of the 2015 list, it was my responsibility to organize, implement and manage the new judging system. This was made possible with the joint efforts of our esteemed regional chairs. So many thanks to AB, RC, RL, & JR!

Our 2015 Panel

The distribution of our judges was as follows based proportionally on region population and number of restaurants per region:

A table drafted to show the general distribution of judges in comparison to total area covered, population and number of restaurants per region.

We acknowledge there are limitations in the number of judges we were able to invite to judge. But we also had to face the reality that it is not easy to find voters, let alone diners, who have traveled across Canada the way we were expecting our quantifiable judges to have already done by the time of ballot casting.

Travel in Canada is not economical (thanks, Canadian airlines), which is what hampers most from doing it in the first place. It was disappointing to learn that many of our experienced and well qualified nominees (past judges and new recruits) were ineligible to participate because they hadn’t traveled (and dine) across the country within our 2 year time frame.*

Another is expense. And interest. We do not pay our judges. At least not yet (ultimately, we hope to compensate those who are contributing to our list eventually, until then, we rely on their generosity of sharing their experiences). Meals reviewed are not assigned, so the bank of restaurants that were experienced were not within our control (it is likely that some excellent businesses were not visited by enough judges to warrant placement).

We rely on transparency. That these same judges have discernment that they are voting fairly without bias. We do, however, request that judges consider the type of meal they’re experiencing. That it has to be reflective, at least in terms of the food served, of what the restaurant is like for the average diner. Any one off special dinners, collaborative events, or off-menu experiences can’t be considered unless the general population can have the same access on a consistent basis. This, too, is a new condition we incorporated into this year’s rules, and addresses discrepancies we’ve seen between those who vote high for special experiences that might showcase the talent in the kitchen, but is not what most people will encounter if they participated in a regular meal.

And by request by members of the industry, we were urged to eliminate peer reviewers (aka no chefs who would have a contending restaurant, or restaurant owners) from our panel. We ended up having a smaller pool of food experts to draw from, but also one that’s tailored towards the overall restaurant experience.

Results

We were pleased with our results. Although a few favourites seemed to have unfortunately lost their position on the list, all those who ended up ranking had great reason to be there. We were also happy to see there’s consistency within the votes: those that ranked high, surged well above their colleagues not only in number of votes but also the points they received. If the word unanimous could be used (though it’s not quite appropriate here), I would say it was unanimous across the board. Judges, Canada, these were certainly your favourites.

Chef Normand Laprise, whose restaurant Toqué! came in at #2, had a well said response to the results of the 2015 list.

Challenges & Reflections

The founder of both Vacay.ca and Vacay’s Top 50 Restaurants in Canada list had a well written response in reflection of the list. It’s an annual tradition that dates back to the start of the list, which I think is important, promoting both discussion and understanding, but also reflection and potential modifications towards that ideal judging system.

This year’s addresses some of the concerns raised not only by members of the public, but also within our executive team as we review the success of our latest judging criteria:

We have learned a lot about Canada’s dining scene since 2012, when the annual ranking of the nation’s best restaurants was launched. Each year, we celebrate the best in dining across the country. Each year, there are exalted debuts and happy restaurateurs, and there are also notable, high-quality restaurants that are left off the list, and these omissions concern their chefs, owners and supporters…

….We also know there will be grumbling from the restaurants that miss out and from their avid supporters. The question that could be asked of them is, “What restaurant would you remove to make room for your choice?” It’s a far more difficult judgment to make this year than in any other, because every member of the 2015 Top 50 Restaurants in Canada is quite exceptional. Ultimately, that’s a good thing. Our nation’s dining choices are getting better and better, just look at all the amazing restaurants we’ve ranked — and the names of some of the great ones we didn’t.

The response has been hugely positive, and my explanation (not excuse) is that it appears to be a numbers game. Given that we are a top 50 list, we could only list 50 businesses. No more. And unfortunately there are more than 50 excellent contenders in the Canadian market, which means, they weren’t mentioned.

I’ve had feedback from one judge who felt uneasy with the placement of restaurant X, feeling it deserved to be higher than Y and Z. In our communications, it didn’t become clear until the end that the judge realized it’s a hard argument to make when (s)he hadn’t personally cast a vote for restaurant X. What difference that could have made. Alas, next time.

That’s the thing. There’s only so much one can do if the support isn’t stepping up, or, in this case and given the limitation of guidelines, prioritized. Everyone on our panel starts with an equal say (votes and points to distribute). How they use it to influence the final results becomes their responsibility. It’s all our responsibilities.

As always, I’m happy to discuss and open to feedback. It’s my belief that we can reach our goal of becoming a definitive guide by achieving a methodology that all parties can agree is fair, reasonable, and can be delivered given the resources we have. After all, the only way towards improvement is to have active participation of all those involved – from the judges, restaurants, and of course, the readers.

Finally, to the men and women who feed us:

The cream rises to the top, and I’m very proud of what our team has achieved. I’m excited to share in the celebration of excellence with those who have placed, but also those who were recognized (but unfortunately due to the definition of being Top 50, not publicly announced). There were many fine businesses and hard working staff who have worked tirelessly to cater to a very demanding, and often fickle public. It’s not an enviable position to be in – where everything you do is subject to someone’s assessment (official or not), where livelihoods are made based on the dining dollar, but if you’re a restaurant, drawing in customers, rest assure you are doing something right. You don’t need a list to tell you so. But we’d love it if you join with us in congratulating your colleagues for their similar great work; and for some others, perhaps the motivation to keep pushing.

Many of our best memories happen in your restaurants, thank you for being a part of it.
Thanks for reading.

PS If there is anyone out there interested in helping sponsor the list without conflicts of interest, please do reach out.

*The last 18 months I made it a specific goal to explore the far corners of our beautiful country, just for this purpose. As a restaurant writer who doesn’t just look at Toronto (although I’m based in the city), it was almost embarrassing that I knew and had experienced the scene of other countries, on the opposite side of the world, more than 3 hour drive from my own home. I’d like to think that part of the reason why I was asked to Chair this year (although it was in late winter, so time constraints, but I digress) is that I was familiar with the culinary scene of the country as a whole. There’s still so much to explore, but I can proudly say that I can tell you what’s awesome in Saskatoon, that some of our most mature and well defined Canadian cooking is happening in lesser traveled Newfoundland (how appropriate that it’s in one of our oldest cities), and that living in the self named “center of the Canadian universe” (it’s a Toronto thing) doesn’t mean that everything is here.

Advertisements

Share this:

Like this:

LikeLoading...

Related

About Renée Suen

Renée Suen is a food loving freelance writer and photographer based in Toronto. Her insatiable appetite, curiosity and camera are often found travelling around the world in search of memorable tastes and the stories behind the plate. In another life, she is a PhD candidate in Cardiovascular Sciences.