I am currently researching the trial roulette method for my masters thesis as an elicitation technique. This is a graphical method that allows an expert to represent her subjective probability distribution for an uncertain quantity.

Experts are given counters (or what one can think of as casino chips) representing equal densities whose total would sum up to 1 - for example 20 chips of probability = 0.05 each. They are then instructed to arrange them on a pre-printed grid, with bins representing result intervals. Each column would represent their belief of the probability of getting the corresponding bin result.

Example: A student is asked to predict the mark in a future exam. The
figure below shows a completed grid for the elicitation of
a subjective probability distribution. The horizontal axis of the
grid shows the possible bins (or mark intervals) that the student was
asked to consider. The numbers in top row record the number of chips
per bin. The completed grid (using a total of 20 chips) shows that the
student believes there is a 30% chance that the mark will be between
60 and 64.9.

Some reasons in favour of using this technique are:

Many questions about the shape of the expert's subjective probability distribution can be answered without the need to pose a long series of questions to the expert - the statistician can simply read off density above or below any given point, or that between any two points.

During the elicitation process, the experts can move around the chips if unsatisfied with the way they placed them initially - thus they can be sure of the final result to be submitted.

It forces the expert to be coherent in the set of probabilities that are provided. If all the chips are used, the probabilities must sum to one.

@Harlan I couldn't track down her thesis, but the trial roulette method is described on p 18 of Eliciting Probability Distributions (a nice reference pointed out by @john-l-taylor)
–
AbeJun 28 '13 at 14:50

structuring and decomposition (typically deciding precisely what variables should
be elicited, and how to elicit joint distributions in the multivariate case);

the elicitation itself.

Of course, they also review how the elicitation results in information that may be fit to or otherwise define distributions (for instance, in the Bayesian context, Beta distributions), but quite importantly, they also address common pitfalls in modeling expert knowledge (anchoring, narrow and small-tailed distributions, etc.).

I'd recommend letting the subject expert specify the mean or mode of the prior but I'd feel free to adjust what they give as a scale. Most people are not very good at quantifying variance.

And I would definitely not let the expert determine the distribution family, in particular the tail thickness. For example, suppose you need a symmetric distribution for a prior. No one can specify their subjective belief so finely as to distinguish a normal distribution from, say, a Student-t distribution with 5 degrees of freedom. But in some contexts the t(5) prior is much more robust than the normal prior. In short, I think the choice of tail thickness is a technical statistical matter, not a matter of quantifying expert opinion.

excellent point about the tails, which I think is key. Another contrasting example would be to consider weibull and gamma as alternatives to the log-normal. In practice, these often provide more realistic fits to right-skewed positive variables.
–
AbeJun 28 '13 at 14:41