Now that I've got even more free time, here's a ridiculously long post on making an already-pretty-good game slightly better. Enjoy.

The recent/current threads on reducing packet-writing requirements, the future of collegiate quizbowl, packet submission etiquette, and various tournament set discussions got me thinking about the rigors of editing today's packet submission tournaments. It seems to me that there's a larger population of decent-to-good question-writers now than there has ever been, which is fantastic. It also seems to me that this hasn't yet translated into a significant reduction in the amount of time editors have to spend working on a tournament set to bring it in line with today's high standards, and that's not so fantastic. In general, I think we should try to find ways to shift more of the set production burden onto the writers. In particular, I think that most decent-to-good question-writers should start by working on their bonuses.

My impression is that there are lots of people that understand the tossup-related concepts of pyramidality; hard, unique lead-ins; and (subject-appropriate, if possible) giveaways. Once a writer has those things figured out, I think the next two things to work on are bonuses and middle clues in tossups. Of those two, I think it might be more productive to talk about bonuses, because I think good bonus-writing habits are easier to acquire than the skill of finding good middle clues to tossups.

There have been a number of pretty good tournament sets this year that left various players (including some high-level players) expressing variants of the sentiment "The questions were generally good, but the hard parts on the bonuses were too hard." I think overly-hard bonuses are detrimental to the cause of good quizbowl: specifically, I think they make it less likely that the most knowledgeable team will win most of the time. There will always be statistical fluctuations/upsets in a game played on a finite number of questions, but I think it's "good quizbowl practice" to try to write questions so that more knowledgeable teams have as many opportunities as possible to take advantage of their superior knowledge, and that means that super-hard bonus parts have to go.

I'll pause for a moment to note that there's been some debate over the relative merits of the "quizbowl as a game" and "quizbowl as an opportunity to learn cool stuff" viewpoints. I subscribe to both, but I think the game aspect has to trump the learning aspect, at least at the level of bonus-writing that I'm talking about. Also, I think it should be possible to work in some amount of cool new material as clues to stuff knowledgeable teams can answer without forcing it in as answers no one (or almost no one) can give. I would argue that this makes for a more satisfying playing experience and at least as satisfying a learning experience: if you bring in some cool new thing as an answer and you can't link it to anything people recognize, many people won't be interested; if you can link it to something people recognize, why not make the recognizable thing the answer and use the new thing as one of several clues?

Moving back into bonus theory: I'm advocating the idea that the hardest parts of bonuses should be less hard, in general, than most people make them. Let's talk numbers: suppose you have a tournament with 20 teams, on the large side (I think) for a typical invitational. Suppose you write/edit a bonus with a hard part that only 1 team can answer (or 5% of the field). Well, about half the time that bonus will see a 30-point conversion; half the time, it won't, when that team's opponent gets the bonus. This seems pretty lame to me, especially when you factor in that teams routinely fail to answer things they know--because they misunderstand the question, because they're too tired to pull the answer, or whatever. If no one at a decent-sized tournament answers a bonus part, that bonus part has completely failed to distinguish any teams on the basis of knowledge; I think this should be minimized. There's a similar argument against easy bonus parts that every team can convert in their sleep, but I don't think those are a problem in today's tournaments. I would argue that the hard part of a bonus should aim for 15-20% conversion by the field. In a 20-tournament field, this translates to 3 or 4 teams knowing the hard part.

Let me put this another way: suppose you only write three part bonuses where each part is worth 10 points. Then in some sense, you're sorting teams into 4 classes: teams that will earn 0 on the bonus, or 10, or 20, or 30. If your goal is to provide the best-possible ranking of teams based on knowledge, it makes sense to write bonuses so that all 4 of these classes are populated. It doesn't make sense to write a bonus so that everyone gets clumped into the "will earn 10" and "will earn 20" classes. To my mind, 1 team out of 20 is insufficient for the population of any class. Occasionally people write a bonus with a hard part that they know is extra-hard, then try to make up for it by making the easy/medium parts easier than normal. This seems suboptimal; there's always variation from one bonus to the next, but it seems to me the appropriate thing to do when you recognize that a hard part is too hard is to rework or replace it. Also, while making the easy/medium parts easier helps preserve the average bonus conversion rate, it actually does a worse job of sorting teams by knowledge level than leaving the easy/medium parts alone.

While I'm pulling numbers out of thin air, I'll go ahead and toss out 80-90% and 40-50% as proposed conversion rates for the easy and middle parts. Note that this gives an average conversion of about 15 ppb. The specifics of the easy and middle part conversion rates and the total average ppb don't interest me much right now, but if people want to talk about them (or anything else bonus-related), go for it.

I'll assume the high-powered analysis I presented above has convinced everyone that hard bonus parts should be written so that there's a decent conversion rate, and move on to bonus practice. There are lots of ways to make hard bonus parts gettable. First off, you have to forget about answers that the tournament audience doesn't know anything about--by definition, there's no way to make these gettable using acceptable clues. Sometimes people write this type of bonus part because they're writing in a subject they know little about; it's unfortunate, but it's understandable. I'm more concerned with people that should know better, but still submit bonuses with really hard parts in topics they know well (see example 1 below). For people writing bonuses in topics with which they're not comfortable, here's a suggestion: the hard part of a bonus doesn't have to have an answer that's not commonly known--you can write a hard bonus part on a well-known answer if you use hard clues. I think this type of hard bonus part is also easier for editors to work on and use than a hard bonus part on a marginal answer, which makes it more desirable.

Another source of poorly-constructed bonuses: sometimes people come up with an idea for a bonus, then find that they can't find a good third part and, rather than abandon the bonus theme, write a bad third part (usually a ridiculously hard part). I would suggest widening the scope of the bonus (see example 2 below [also example 1 again, I guess]) until you can find an acceptable third part.

Here are two examples, both written by me for this year's Cardinal Classic.

Example 1:

Seth Teitler wrote:Name these lazy bums of Norse myth that will survive Ragnarok, FTPE.
[10] This son of Rind and Odin refrains from washing his hands or combing his hair until he avenges Balder’s death. He accomplishes the task the day after he is born, after which he presumably can wash his hands and comb his hair all he likes.
ANSWER: Vali
[10] This long-legged god pissed off the Vanir by refusing to give them counsel after being sent as a hostage, so the Vanir naturally beheaded his wise companion Mimir and left him alone.
ANSWER: Honir or Hoenir
[10] This god is said to occupy the enviable position of second-strongest of the Aesir. When he’s not busy fulfilling his duties as the god of silence, he can usually be found putting together scraps of discarded leather to make the Thickmost Shoe, with which he will kill Fenrir at Ragnarok.
ANSWER: Vidarr

Example 2:

Seth Teitler wrote:It united with the Carlist party during the Spanish Civil War. FTPE:
[10] Name this nationalist movement which brought Franco to power.
ANSWER: Phalange or Falange Española Tradicionalista Y De Las Juntas De Ofensiva Nacional-sindicalista or Traditionalist Spanish Phalanx of the Juntas of the National Syndicalist Offensive
[10] The Falange was founded by the elder son of this man, who took control of the Spanish government for seven years as Prime Minister under Alfonso XIII after leading a 1923 coup.
ANSWER: Miguel Primo de Rivera y Orbaneja, Marqués de Estella
[10] Miguel’s daughter Pilar Primo de Rivera formed this organization, the women’s branch of the Falange.
ANSWER: Sección Femenina or Feminine Section

What's up with these bonuses?

In the first case, I decided to knock out a Norse myth bonus for "my choice" since I like Norse myth, then I started thinking about gods that survive Ragnarok, and then.. I picked three of the hardest possible answers. I'm not really sure why I did this. I think Hoenir might be too hard, even for a hard part answer. I think Vali and Vidarr should be okay, but certainly the three of them put together are not okay. I think at least one team at Cardinal Classic could 30 this bonus, maybe even two teams, but it has no distinguishable easy and medium parts. It's hard for me to imagine a team that could 10 this bonus. A couple days after submitting my team's packet, I looked back over my questions and decided the original version of the bonus was insane. I then rewrote the bonus and submitted the following version:

Seth Teitler wrote:Answer the following related to some lazy bums of Norse myth who will survive Ragnarok, FTPE.
[10] This long-legged god pissed off the Vanir by refusing to give them counsel after being sent as a hostage, so the Vanir naturally beheaded his wise companion Mimir and left him alone.
ANSWER: Honir or Hoenir
[10] When Vidar’s not busy doing whatever it is that a god of silence does, he works very slowly on the Thickmost Shoe, with which he will kill this son of Loki at Ragnarok. This gigantic wolf ate Tyr’s hand.
ANSWER: Fenrir or Fenrisulfr
[10] Vali gets to sleep in late, wash his hands and comb his hair all he likes after accomplishing his sole task in life one day after his birth, when he kills this blind god. This poor guy was duped into killing his brother Balder.
ANSWER: Hoder or Hodur or Hodr

This version's hard part (Hoenir) may still be too hard--I'm not really sure how many teams know about him--but I'm much more confident that Fenrir and Hoder make fine easy and medium parts than Vali and Vidarr. Note that I had to widen the scope of the bonus slightly--the answers aren't all "lazy gods that survive Ragnarok" any more.

Moving on to example number 2: this is me writing a bonus in a topic I don't know well. I found some stuff on José Primo de Rivera and decided to make a bonus on the Primo de Rivera family and the Phalange. Unfortunately, I don't think I found a good hard part. Perhaps multiple teams at Cardinal Classic know about Pilar Primo de Rivera's work with the Sección Femenina, but I doubt it. In this case, the editors did a fine job of widening the scope of the bonus and adding a better hard part: they replaced the third part with a part on Sanjurjo, if I remember correctly. I don't know for a fact that multiple teams at CC know about Sanjurjo, but it certainly seems more likely.

Moving on to the Cardinal Classic discussion thread: from what I read of the indie music discussion, it sounds like there's a subpopulation of quizbowlers that follows indie music in some depth, while the majority of quizbowlers know very little about it. Given that the proper structure for an indie music bonus is the same as for a bonus on any other topic, this suggests that the hard part should be somewhat challenging for the people that follow indie music, and the easy and medium parts should be significantly easier, since there will be a sizable number of teams that don't have an indie music expert. The easy part may even have to be non-indie, just to achieve an acceptable 10 point conversion rate.

Really long story short: people that have figured out the basics of writing tossups should be able to write/edit good bonuses, especially in areas they know well. I think the most common problem with non-good bonuses is that they are too hard to 30. Suggestions for avoiding this problem include widening the scope of bonuses (e.g., including 1 or 2 non-indie music parts with a harder indie music part, rather than 3 indie music parts that most teams can't get) and using an "easy" answer with hard clues as the hard part of a bonus.

This is an excellent post, and I think I will take up a few points for clarification/contention.

setht wrote:This seems pretty lame to me, especially when you factor in that teams routinely fail to answer things they know--because they misunderstand the question, because they're too tired to pull the answer, or whatever.

I think this is generally correct, but I am unsure how we could account for that. This comes down I think to a general distinction between the "orality and literacy" of the game. One, as a writer I am subjugated by the text, I have the opportunity to read and reread, as an editor (unless I play test them) this is similarly the case and so telling how something will "play out" seems complicated and furthermore, it can go either way, a bonus sounds harder than it reads and vice versa. Also, if one's packet is at the end of the day then mental exhaustion and confusion are factors to be sure, while the same exact bonus, if played at the beginning of the day will be pocketed for mad points. How does one account for this? For tossups, the giveaway is almost always a dead giveaway and thus even if it should have been answered before then, and would have had it been asked earlier, one always has a shot at redemption.

setht wrote:For people writing bonuses in topics with which they're not comfortable, here's a suggestion: the hard part of a bonus doesn't have to have an answer that's not commonly known--you can write a hard bonus part on a well-known answer if you use hard clues.

If I am uncomfortable with the subject matter, then I think I would be equally so with determining hard clues. The last thing I would want to do is rob someone of points, especially if they know the answer part, but failed to earn them because I wrote something that was not relevant to what one should know. If an overtaxed editor failed to recognize the error-since going through some 60 bonus part answers per packet might be difficult especially if the answer prima facie is "gettable," then the editor might just trust the fact that there are answerable clues in the bonus for such an answerable part. I have heard people complain vociferously, "why didn't the bonus include [obvious clue]?" I mean it is a bonus after all isn't it by design a reward for answering a tossup.

Or is this the wrong mentality to have? I believe that this mentality spurs hard answers rather than hard clues to answers in bonuses. The proverbial bone is thrown to reward the answering of the tossup and then the rest constitutes a bone being placed...somewhere less pleasant or appetizing. Thus in order to insure a tossup isn't worth 40 points (10+30) hard second and third parts are added to keep the game a "fair" balance between anticipation/recognition/recall speed and depth of knowledge.

Finally, though you argue teams shouldn't be lumped into 10, 20, 30 point calibers-isn't this true of a situation that already is, rather than an intention that is previously suggested? Of course no one averages 30 points per Bonus, but on any one bonus a team of the 30 point caliber variety can expect 30 points half the time and so on for each level of team.

Yeah, this is a good post, and I mostly agree with Seth...there is a tendency to write third parts that are pointlessly hard, which can be frustrating for teams.

I think what happens a lot is that people lock themselves into bonuses that don't have three parts of ideal difficulty. I've done that lots of times - I think of a fine idea for a bonus, but I just don't have that perfect "easy" part or I don't have the ideal medium part or I don't have an ideal hard part, whatever. So, I try to jam my triangle of a bonus into the circle slot and then just say "oh well, if people know something about this subject, they can get points here...it doesn't look too unfair, i.e. it's not a probable 0 or a gift 30, so it'll do." I think that's just the way it goes sometimes.

Plus, the fact is, that when you choose bonus parts, you have this "guessing problem" again. Unless it's a tested-and-true warhorse of a bonus, you really have to make a pretty iffy guess as to whether enough people are going to know each part. Although, of course, if it's that tested-and-true, teams will start knowing it. It's all just a big guess, and it's especially hard with better players to predict where the margins of their knowledge lay. It's even harder when you're writing on something that you're not familiar with. But, I agree with Seth that it's often all too tempting to just pick something off-the-wall for that third part, and that doesn't do much good, under either the "game" or the "learning" theory of qb.

Ryan Westbrook, no affiliation whatsoever.

I am pure energy...and as ancient as the cosmos. Feeble creatures, GO!

Left here since birth...forgotten in the river of time...I've had an eternity to...ponder the meaning of things...and now I have an answer!

SnookerUSF wrote:If I am uncomfortable with the subject matter, then I think I would be equally so with determining hard clues. The last thing I would want to do is rob someone of points, especially if they know the answer part, but failed to earn them because I wrote something that was not relevant to what one should know. If an overtaxed editor failed to recognize the error-since going through some 60 bonus part answers per packet might be difficult especially if the answer prima facie is "gettable," then the editor might just trust the fact that there are answerable clues in the bonus for such an answerable part. I have heard people complain vociferously, "why didn't the bonus include [obvious clue]?" I mean it is a bonus after all isn't it by design a reward for answering a tossup.

I think this goes hand in hand with the "I've heard of (clue x) or (answer x), so it's too easy to be put there" problem. I'm all for easy-ifying hard parts of bonuses, but I think that there's been such a backlash on "gimme 30s" that people are trying to find the most ridiculous hard parts they can come up with when they don't know a subject. I know I've been guilty of the "I have to find something that's outside my knowledge base since I shouldn't be 30'ing this bonus but I'd 20 it off these two already-written parts" thought process.

I think there's a dimension of what is predictable or "studyable" involved as well. You can write a hard bonus part on The Celestial Omnibus if two more reasonable things about EM Forster comprise the rest of the bonus, even though that's a very hard thing that may have never been an answer before, because anyone who is tugging on his lit abilities will want to know about an author who is important and who comes up as much as EM Forster. When you're writing an "English authors" bonus and want to start asking about Arnold Wesker, you can't rely on the elite players who study for quizbowl necessarily having heard of that answer like you can with a hard Forster title.

SnookerUSF wrote:This is an excellent post, and I think I will take up a few points for clarification/contention.

setht wrote:This seems pretty lame to me, especially when you factor in that teams routinely fail to answer things they know--because they misunderstand the question, because they're too tired to pull the answer, or whatever.

I think this is generally correct, but I am unsure how we could account for that. This comes down I think to a general distinction between the "orality and literacy" of the game. One, as a writer I am subjugated by the text, I have the opportunity to read and reread, as an editor (unless I play test them) this is similarly the case and so telling how something will "play out" seems complicated and furthermore, it can go either way, a bonus sounds harder than it reads and vice versa. Also, if one's packet is at the end of the day then mental exhaustion and confusion are factors to be sure, while the same exact bonus, if played at the beginning of the day will be pocketed for mad points. How does one account for this? For tossups, the giveaway is almost always a dead giveaway and thus even if it should have been answered before then, and would have had it been asked earlier, one always has a shot at redemption.

I'm saying that the (occasional) difficulty of figuring things out from hearing the clues once, possibly at the end of a long day, makes it all the more important that people not aim for a hard part conversion rate of, say, 5-10%. Aim for 15-20% instead. If you aim at having 4 out of 20 teams 30 every bonus, hopefully there will be few if any bonuses that have no 30s. Really, all I'm arguing for is that people that have a basic grasp of question-writing should keep certain bonus conversion goals in mind as they write/edit bonuses; obviously it's a subjective, good-faith sort of thing--you do your best, and sometimes you mess up. I think there are lots of people that could do a better, more consistent job with their bonuses if they just thought about this stuff while writing. In the absence of detailed bonus statistics (e.g., how many teams got 0/10/20/30 on each bonus in a packet), all we can ask is that writers shoot for a consistent, reasonable goal. If detailed stats become available, writers can go back and see how they did, then adjust for future writing.

SnookerUSF wrote:

setht wrote:For people writing bonuses in topics with which they're not comfortable, here's a suggestion: the hard part of a bonus doesn't have to have an answer that's not commonly known--you can write a hard bonus part on a well-known answer if you use hard clues.

If I am uncomfortable with the subject matter, then I think I would be equally so with determining hard clues. The last thing I would want to do is rob someone of points, especially if they know the answer part, but failed to earn them because I wrote something that was not relevant to what one should know. If an overtaxed editor failed to recognize the error-since going through some 60 bonus part answers per packet might be difficult especially if the answer prima facie is "gettable," then the editor might just trust the fact that there are answerable clues in the bonus for such an answerable part. I have heard people complain vociferously, "why didn't the bonus include [obvious clue]?" I mean it is a bonus after all isn't it by design a reward for answering a tossup.

Or is this the wrong mentality to have? I believe that this mentality spurs hard answers rather than hard clues to answers in bonuses. The proverbial bone is thrown to reward the answering of the tossup and then the rest constitutes a bone being placed...somewhere less pleasant or appetizing. Thus in order to insure a tossup isn't worth 40 points (10+30) hard second and third parts are added to keep the game a "fair" balance between anticipation/recognition/recall speed and depth of knowledge.

You're probably right that it can be hard for people writing in awkward categories to figure out how to write a good bonus part on an easy answer using hard clues, but I would argue that there are at least more resources for doing this than there are for writing a good bonus part on a hard answer. For one thing, if people have access to (good) old questions, they're going to find more examples of questions on easy answers than on hard answers, and correspondingly I think they'll find more hard clues for easy answers than they will easy clues for a particular hard answer. Similarly, I would guess that it's easier to find hard clues on "easy answers" in textbooks because "easy answers" tend to be things that are important in their fields and get more space in textbooks. Based on this line of argument, I think there's a better chance that the writer will do a good job in the first place, or that the editor will be able to fix things up if the writer misses the target, by writing hard bonus parts on easier answers with hard clues. Now, if the editor is overworked and can't devote the time to check over every bonus part, that's unfortunate, but I don't see why hard bonus parts on hard answers should hold up better under a lack of editing. Finally, I think many people understand the concept of bonus parts of graded difficulty, and I think that reserving clues on well-known answers to increase the difficulty of a bonus part is also a fairly well-known practice. As long as it's done right--that is, there really is enough clue material to give people with some target level of knowledge a decent shot at answering the bonus part--I don't think there's any problem, and I think disgruntled players that say, "Oh, I would have gotten that Mark Twain part if only you'd said 'Tom Sawyer'" will understand when you explain that the Mark Twain part was, in fact, not the easy part of the bonus.

I don't follow the argument in your second paragraph, but I'll take a stab at responding anyway. If any bonus is an automatic 0, or 10, or 20, or 30 for every team at a tournament, it serves no purpose in differentiating teams--in effect, it just sets how much the corresponding tossup is worth, and the teams only compete on tossups (and this is particularly bad if the bonus difficulty varies drastically--suppose the first 10 bonuses are automatic 0s and the next 10 are automatic 30s; then teams are primarily competing on the last 10 tossups). I think the goal in structuring bonus difficulty is somewhat analogous to the goal in writing a pyramidal tossup: at each stage, you want to "let in" a certain percentage of the teams playing the question by including material from which they can produce an answer. In the case of a good tossup, the structure includes much more than 4 stages, so the difference in difficulty from one stage to the next is smaller (e.g., you start with a lead-in that ~5% of the teams can buzz on, then add clues that increase the number of teams that can buzz by 5% per clue, until the 20th clue/giveaway). With a (traditional 3-part) bonus, you've only got 4 stages: 0, 10, 20 and 30 points. If someone can write a tossup on Mark Twain with a mid-level clue that 40% of a tournament field can buzz on, that same clue should be usable in a middle bonus part with a target goal of ~40-50% conversion.

If I'm not addressing your concerns, let me know.

SnookerUSF wrote:Finally, though you argue teams shouldn't be lumped into 10, 20, 30 point calibers-isn't this true of a situation that already is, rather than an intention that is previously suggested? Of course no one averages 30 points per Bonus, but on any one bonus a team of the 30 point caliber variety can expect 30 points half the time and so on for each level of team.

I must not have expressed myself clearly in the original post. I am arguing that writers/editors should lump teams into 0/10/20/30 point bins when thinking about any particular bonus. Which teams are in which bins changes from one bonus to the next, since every team varies in strength over different categories, but that doesn't change anything: you look at one bonus, and you think to yourself, "Do I believe that ~15-20% of the tournament field will 30 this bonus if they get it? Do I believe 40-50% will 20 it? Do I believe 80-90% will 10 it? Do I believe 10-20% will 0 it?" If the answer to any of those questions is "no," go work on the bonus to change that. I don't think there's anything useful (from a question writing/editing perspective) to be gained from trying to classify teams in 0/10/20/30 ppb bins based on average conversion over all the bonuses in a packet or a tournament: it's absolutely fine if a team that winds up averaging 24 ppb gets the occasional 0 or 10, just as it's absolutely fine if a team that winds up averaging 6 ppb gets the occasional 30--in fact, this should happen, since teams do vary significantly in their strength from one category to the next. What's not good is if one bonus has a conversion distribution which looks significantly different from other bonuses in the packet, or which doesn't have a decent population in each bin (since then it doesn't do as good a job as it could in differentiating teams). Trying to predict conversion rates while writing bonuses (and reworking bonuses so the predictions look the way you want them to look) won't produce perfect bonuses with identical, ideal conversion distributions, but I think it will produce better bonuses than we're currently getting, with higher consistency and better conversion distributions.

Ryan Westbrook wrote:I think what happens a lot is that people lock themselves into bonuses that don't have three parts of ideal difficulty. I've done that lots of times - I think of a fine idea for a bonus, but I just don't have that perfect "easy" part or I don't have the ideal medium part or I don't have an ideal hard part, whatever. So, I try to jam my triangle of a bonus into the circle slot and then just say "oh well, if people know something about this subject, they can get points here...it doesn't look too unfair, i.e. it's not a probable 0 or a gift 30, so it'll do." I think that's just the way it goes sometimes.

Like Ryan, I think people are sometimes staying attached to their bonus concepts to the detriment of their bonus structure, and as I noted in my first post, I am against this. If a writer/editor doesn't realize or doesn't notice that a bonus has bad structure, that's a shame but it's understandable; however, if someone looks at a bonus and does realize that there's a problem, I think they should throw the linking concept out the window and rewrite/edit to make a bonus with good structure. There's going to be some variance, some bonuses will have a somewhat harder middle part or a somewhat easier hard part than others, and that's fine; my point is that when people realize there's a noticeable problem with a bonus, they should work to bring it back within the acceptable bounds of that variance. If that means scrapping a supposedly great idea for a bonus on lazy bums that survive Ragnarok in favor of a bonus on "name these characters from Norse myth," so be it.

Ryan Westbrook wrote:Plus, the fact is, that when you choose bonus parts, you have this "guessing problem" again. Unless it's a tested-and-true warhorse of a bonus, you really have to make a pretty iffy guess as to whether enough people are going to know each part. Although, of course, if it's that tested-and-true, teams will start knowing it. It's all just a big guess, and it's especially hard with better players to predict where the margins of their knowledge lay. It's even harder when you're writing on something that you're not familiar with. But, I agree with Seth that it's often all too tempting to just pick something off-the-wall for that third part, and that doesn't do much good, under either the "game" or the "learning" theory of qb.

Again, I'm not calling for writers to get everything perfect before submitting their questions--that's just not reasonable. If you're writing a bonus and you really don't know how hard various things are in the topic you're writing on, you'll make some mistakes, and that's okay. Also, I realize there's no clear way to know for a fact that precisely 3 teams out of a given set of 20 will know any particular bonus part. So I agree with Ryan that "It's all just a big guess" and that "It's even harder when you're writing on something that you're not familiar with." My point is that I think many writers and editors aren't even taking a stab at that "big guess"--they're just slapping hard bonus parts onto bonuses without stopping and thinking about how many teams might have a shot at it--and I think bonus consistency and quality will improve without killing writers or editors if people do take a moment to think about this. Also, if you adopt the practice of writing hard bonus parts on well-known answers, the "big guess" boils down to pretty much the exact same issue faced by people writing/editing tossup lead-ins/early clues. In fact, it should be a bit easier, since you don't need to guess at pyramidal ordering for multi-clue bonus prompts. My impression is that tossup quality is generally higher than bonus quality at most events, so I think this scheme should help things.

cvdwightw wrote:there's been such a backlash on "gimme 30s" that people are trying to find the most ridiculous hard parts they can come up with when they don't know a subject. I know I've been guilty of the "I have to find something that's outside my knowledge base since I shouldn't be 30'ing this bonus but I'd 20 it off these two already-written parts" thought process.

I think the main objection people have to "gimme 30s" is that they're surrounded by bonuses that have a very different structure (namely, "struggle to get 30"). The appropriate response is to try to bring all bonuses to one consistent, happy norm, which (I think) doesn't require people "to find the most ridiculous hard parts they can come up with when they don't know a subject." If you're writing outside your knowledge base, so you feel that you shouldn't write a bonus you could 30 (without having written the bonus), why not look up some old tossups on a related easy answer, then look up material related to some of the lead-ins/early clues and write hard clues to an easy answer? If you're writing outside your knowledge base, presumably you don't buzz regularly on the first clues of tossups in that area, so doing this serves the same purpose ("I shouldn't 30 this bonus") as writing a hard bonus part on a hard answer, and it's much more likely that an editor will be able to put the bonus part to good use.

Matt Weiner wrote:I think there's a dimension of what is predictable or "studyable" involved as well. You can write a hard bonus part on The Celestial Omnibus if two more reasonable things about EM Forster comprise the rest of the bonus, even though that's a very hard thing that may have never been an answer before, because anyone who is tugging on his lit abilities will want to know about an author who is important and who comes up as much as EM Forster. When you're writing an "English authors" bonus and want to start asking about Arnold Wesker, you can't rely on the elite players who study for quizbowl necessarily having heard of that answer like you can with a hard Forster title.

I suppose this is true, but I think it really depends on the tournament field (to clarify: I suppose Matt's right that there are some tournaments where a hard part on The Celestial Omnibus would be fine, even if it hasn't shown up before as a bonus part or early tossup clue; I certainly agree with Matt that Arnold Wesker is a terrible choice for a hard part on an English authors bonus). In most situations, I would imagine that writing a hard part with the answer "EM Forster," using The Celestial Omnibus and other not-so-well-known works as clues, would be preferable to writing a hard part with the answer "The Celestial Omnibus," and this is the approach I advocate for anyone that doesn't feel confident in their abilities while writing a bonus.

I want to note here that I think it's perfectly fine for writers and editors that do feel confident in their knowledge base for a particular bonus to write bonus parts on whatever answers they want, as long as they feel they're maintaining a good bonus structure consistent with the other bonuses in the packet. Some of my earlier remarks (on principles of bonus structure, asking people to think about these things, and asking people to prioritize structure ahead of tight focus/answer linking) are meant for everyone, whether they're writing/editing in an area of strength or not; a good chunk of what I've said (the suggestions for improving bonus structure by giving hard clues for well-known answers) are meant primarily for people writing/editing in an area of weakness.

So, if you're confident in your EM Forster/Celestial Omnibus knowledge, and you believe ~15-20% of your target audience can answer a part on The Celestial Omnibus, go for it. If you're not confident in your knowledge of all things EM Forster, go with an answer of EM Forster and try to put in some hard clues (the same way you would in writing the beginning the opening of an EM Forster tossup--actually, probably more like the second sentence of an EM Forster tossup, since your target conversion rate should be ~15-20%, which is higher than the target conversion rate for the opening of a tossup). If you've already used EM Forster as the easy part, and you're stuck trying to figure out a good hard EM Forster-related part, with no confidence in your EM Forster knowledge.. take a step back, abandon the EM Forster focus of your bonus, and put in a hard part on some other English author or work of English literature--perhaps someone EM Forster interacted with, a work that inspired him, or a work that he inspired. I think your success rate in finding good hard parts to finish off your bonuses will be higher than if you just take a random stab at picking a hard EM Forster work.

setht wrote:Moving on to the Cardinal Classic discussion thread: from what I read of the indie music discussion, it sounds like there's a subpopulation of quizbowlers that follows indie music in some depth, while the majority of quizbowlers know very little about it. Given that the proper structure for an indie music bonus is the same as for a bonus on any other topic, this suggests that the hard part should be somewhat challenging for the people that follow indie music, and the easy and medium parts should be significantly easier, since there will be a sizable number of teams that don't have an indie music expert. The easy part may even have to be non-indie, just to achieve an acceptable 10 point conversion rate.

Really long story short: people that have figured out the basics of writing tossups should be able to write/edit good bonuses, especially in areas they know well. I think the most common problem with non-good bonuses is that they are too hard to 30. Suggestions for avoiding this problem include widening the scope of bonuses (e.g., including 1 or 2 non-indie music parts with a harder indie music part, rather than 3 indie music parts that most teams can't get) and using an "easy" answer with hard clues as the hard part of a bonus.

First, INDIE MUSIC DISCUSSION OMGESUS

Second, my feeling is that in general bonus theory, thoughts on difficulty should go in two ways. First, one should ask if the general subset of a topic (i.e. Literature-English Lit or Trash-Indie Music) is too difficult in the first place for a tournament. If it's determined to not be, then the focus on difficulty should be on whether it is difficult in comparison to the subject. I don't see a problem with having an indie bonus go not obscure first part (Death Cab) then slightly more obscure (Rilo Kiley) then even more obscure (Sufjan Stevens and the like). The common complaint with bonuses like these are that people haven't heard of the bands. This may be true of the people that zero the bonus, but keep in mind that there are people that have never heard of, say, John Milton. A Paradise Lost-Milton-Random Milton poem bonus is perfectly acceptable in many levels of QB, if not too easy. Bonuses are not directed at the people that zero them, and if they do get a zero it's because they don't know it. The same standards for trash should be held up for any other category.

Colin O'Donnell -- ex-Eden Prairie High School (man, that feels nice to say), eventually University of Minnesota

theattachment wrote:
I don't see a problem with having an indie bonus go not obscure first part (Death Cab) then slightly more obscure (Rilo Kiley) then even more obscure (Sufjan Stevens and the like).

I consider myself to be in the "clueless about indie music" group. Oddly enough, I know nothing specific about Death Cab, have never heard of Rilo Kiley, and could get Sufjan Stevens off of several song titles.

Greg Peterson

Northwestern University '18
Lawrence University '11
Maine South HS '07

theattachment wrote:
I don't see a problem with having an indie bonus go not obscure first part (Death Cab) then slightly more obscure (Rilo Kiley) then even more obscure (Sufjan Stevens and the like).

I consider myself to be in the "clueless about indie music" group. Oddly enough, I know nothing specific about Death Cab, have never heard of Rilo Kiley, and could get Sufjan Stevens off of several song titles.

I consider myself to be in the "clueless about most things" group, so we even each other out. I don't advocate for a player to be put on a team simply to get the one indie question that will ever happen outside of a trash tournament, but I do advocate for the player that devotes his or her energy to trash to at least get a cursory knowledge of most things in the topic.

Colin O'Donnell -- ex-Eden Prairie High School (man, that feels nice to say), eventually University of Minnesota

setht wrote:
I don't follow the argument in your second paragraph, but I'll take a stab at responding anyway. If any bonus is an automatic 0, or 10, or 20, or 30 for every team at a tournament, it serves no purpose in differentiating teams--in effect, it just sets how much the corresponding tossup is worth, and the teams only compete on tossups (and this is particularly bad if the bonus difficulty varies drastically--suppose the first 10 bonuses are automatic 0s and the next 10 are automatic 30s; then teams are primarily competing on the last 10 tossups)

Yeah, I really don't understand my argument either, I guess I am saying that bonuses because they are worth three times more, at a maximum, than a tossup and because they aren't answered under the same kind of pressures as tossups, this is the opportunity to introduce new material into the canon. BUT, I totally agree with a previous point you made, new material can be introduced without having to resort to using them as bonus answers, but including them in the clue. I can definitely make that change.

setht wrote:Now, if the editor is overworked and can't devote the time to check over every bonus part, that's unfortunate, but I don't see why hard bonus parts on hard answers should hold up better under a lack of editing.

Precisely because they hard bonus parts and in a quick scan of the bonus parts they will arise as unusual because of their difficulty and prompt further inquiry. An easier part is more likely to be less rigorously edited, because it doesn't appear as unusual in the editing "cognitive map" to re-appropriate a Maginism. Therefore, if there are some wildly inaccessible clues to "Einstein" as a bonus part, those are less likely to be excised or altered than if there were some totally canonical clues to "Viktor Shklovsky" everybody's favorite Russian Formalist.

Also, I have to say unlike perhaps some of the sentiment expressed so far in this topic, I really appreciate a well constructed bonus theme-even if it includes an insanely hard third part, because those themes are really memorable and give you a trajectory to work with for future learning. It also shows, concern and creativity on the part of the writer who takes the time to research such a theme. Maybe its more of my writing hubris that cringes at the thought of unused bonuses, but nonetheless I dig it.

Ahmad Ragab, itinerant moderator at the New School for Social Research

ACF Nationals 2011:"Too real for the streets"
-Auroni Gupta

"Can 40,000 redacted topic Tossups be wrong?"

"With my gnomes I'm highlighting the danger of political opportunism and right-wing ideology. I get the feeling that this gnome has reopened an old wound."
-Ottomar Hoerl

setht wrote:Moving on to the Cardinal Classic discussion thread: from what I read of the indie music discussion, it sounds like there's a subpopulation of quizbowlers that follows indie music in some depth, while the majority of quizbowlers know very little about it. Given that the proper structure for an indie music bonus is the same as for a bonus on any other topic, this suggests that the hard part should be somewhat challenging for the people that follow indie music, and the easy and medium parts should be significantly easier, since there will be a sizable number of teams that don't have an indie music expert. The easy part may even have to be non-indie, just to achieve an acceptable 10 point conversion rate.

Really long story short: people that have figured out the basics of writing tossups should be able to write/edit good bonuses, especially in areas they know well. I think the most common problem with non-good bonuses is that they are too hard to 30. Suggestions for avoiding this problem include widening the scope of bonuses (e.g., including 1 or 2 non-indie music parts with a harder indie music part, rather than 3 indie music parts that most teams can't get) and using an "easy" answer with hard clues as the hard part of a bonus.

First, INDIE MUSIC DISCUSSION OMGESUS

Second, my feeling is that in general bonus theory, thoughts on difficulty should go in two ways. First, one should ask if the general subset of a topic (i.e. Literature-English Lit or Trash-Indie Music) is too difficult in the first place for a tournament. If it's determined to not be, then the focus on difficulty should be on whether it is difficult in comparison to the subject. I don't see a problem with having an indie bonus go not obscure first part (Death Cab) then slightly more obscure (Rilo Kiley) then even more obscure (Sufjan Stevens and the like). The common complaint with bonuses like these are that people haven't heard of the bands. This may be true of the people that zero the bonus, but keep in mind that there are people that have never heard of, say, John Milton. A Paradise Lost-Milton-Random Milton poem bonus is perfectly acceptable in many levels of QB, if not too easy. Bonuses are not directed at the people that zero them, and if they do get a zero it's because they don't know it. The same standards for trash should be held up for any other category.

I don't think this paradigm will work. For one thing, it has nothing built into it to distinguish between writing questions for, say, ACF Fall and ACF Nationals. I really prefer having people try to structure their bonuses based on some guess at the tournament field (and I don't mean the top 4 teams at the tournament, I mean the whole field). We disagree on how people should think about the terms "not obscure," "slightly more obscure," and "even more obscure"--I'm saying those terms should be defined relative to the perceived knowledge levels of the teams that are playing the tournament, rather than a definition "in comparison to the subject."

Jumping ahead in your post, I agree with you that the same standards should apply to trash as to any other category, but I draw a different conclusion from this: I think that the conversion distribution on a trash bonus should look as much like the conversion distribution on bonuses in other categories as possible. If the writer/editor is shooting for a ratio of 0s/10s/20s/30s on non-trash bonuses that looks something like 10:45:30:15, I think they should make an effort to shoot for the same ratio on their trash bonuses. Based only on what other people said about indie music bonuses in the Cardinal Classic discussion thread, it sounds like this means people writing indie music bonuses may need to include at least one non-indie part, or an easy part on one of the most famous indie music things they can find, and a hard part that's pretty hard, since there are lots of people that know very little about indie music and a decent number of people that know a lot. If the conversion distribution for the indie music bonus winds up being 60:10:10:20, I don't think that's good.

I'll admit right now that I know pretty much nothing about indie music, so I'll stop using that as an example and try something else: earth science. I think earth science is a perfectly reasonable topic for a bonus, but in my opinion almost no one in collegiate quizbowl knows almost anything about it. If I follow your paradigm and write an earth science bonus with one part well-known in the field of earth science, one part less-well-known, and finally one part that's somewhat obscure for people that study earth science... I'm guessing at most 1 or 2 teams in the country could 30 the bonus, perhaps as many as 3 could 20 it, and then there'd be tons of teams getting 0 or 10. This depends a bit on the particular subtopic within earth science and which answers I choose, but there are just large swathes of earth science that almost no one knows. If I want to write a good earth science bonus--that is, a bonus question that is good within the context of quizbowl as a competition and a learning process--the fact is that I have to write something more like easy/easy-to-medium/easy-to-medium, if I judge difficulty "in comparison to the subject."

SnookerUSF wrote:

setht wrote:I don't follow the argument in your second paragraph, but I'll take a stab at responding anyway. If any bonus is an automatic 0, or 10, or 20, or 30 for every team at a tournament, it serves no purpose in differentiating teams--in effect, it just sets how much the corresponding tossup is worth, and the teams only compete on tossups (and this is particularly bad if the bonus difficulty varies drastically--suppose the first 10 bonuses are automatic 0s and the next 10 are automatic 30s; then teams are primarily competing on the last 10 tossups)

Yeah, I really don't understand my argument either, I guess I am saying that bonuses because they are worth three times more, at a maximum, than a tossup and because they aren't answered under the same kind of pressures as tossups, this is the opportunity to introduce new material into the canon. BUT, I totally agree with a previous point you made, new material can be introduced without having to resort to using them as bonus answers, but including them in the clue. I can definitely make that change.

Again, I have no objection to people putting in hard/new bonus answers on things they feel confident will see acceptable conversion--I'm only arguing against doing this when you look back over a bonus and think something along the lines of, "Hmm, I guess almost no one/no one will get this part on Meili, the brother of Thor who only rates a passing mention in the Harbarthsljoth and never appears anywhere else." At that point, I feel the writer/editor should supply a new bonus part; if they can come up with something new that will see appropriate conversion, that's fine, but the main thing is to find a part that the writer/editor thinks will provide good structure.

SnookerUSF wrote:

setht wrote:Now, if the editor is overworked and can't devote the time to check over every bonus part, that's unfortunate, but I don't see why hard bonus parts on hard answers should hold up better under a lack of editing.

Precisely because they hard bonus parts and in a quick scan of the bonus parts they will arise as unusual because of their difficulty and prompt further inquiry. An easier part is more likely to be less rigorously edited, because it doesn't appear as unusual in the editing "cognitive map" to re-appropriate a Maginism. Therefore, if there are some wildly inaccessible clues to "Einstein" as a bonus part, those are less likely to be excised or altered than if there were some totally canonical clues to "Viktor Shklovsky" everybody's favorite Russian Formalist.

Also, I have to say unlike perhaps some of the sentiment expressed so far in this topic, I really appreciate a well constructed bonus theme-even if it includes an insanely hard third part, because those themes are really memorable and give you a trajectory to work with for future learning. It also shows, concern and creativity on the part of the writer who takes the time to research such a theme. Maybe its more of my writing hubris that cringes at the thought of unused bonuses, but nonetheless I dig it.

I see what you're saying about editor scanning. You may be right that hard bonus parts are more likely to attract editorial attention, but I think a bonus with three "easy" answers also has a good chance of being picked up in a quick scan--I think that should be just as unusual in the editing "cognitive map" as some new, crazy answer. To reiterate an earlier point, I think that if someone that doesn't know physics tries to write a hard bonus part on Einstein they will have more resources to do a decent job with that than they will with trying to write a decent hard bonus part on, say, de Sitter, and the editors will similarly have an easier time working on the bonus part on Einstein (if they notice it). I think the benefit in trying from the start to write on a well-known answer and setting the difficulty by clue selection (when the writer is writing in an area of weakness) is greater than the possible loss of the editor not working on the bonus part. Also, if people start doing this when writing questions, that should save the editors time: instead of scanning each packet and then trying to work on some large number of dubious hard answers that may or may not be salvageable, the editor can scan each packet, realize there are very few dubious hard answers, and then maybe go back to check that the clue selection for the easy answer hard parts looks reasonable. It seems weird to advocate hard answers because they will get flagged by editors and use up their time, if we're assuming that editors routinely run out of time.

I also appreciate well-constructed bonuses with interesting links. Sometimes they're interesting enough that I don't mind a bonus structure that's rather off, but those are rare. Based on people's comments in the discussion threads for MCMNT, ACF Regionals, Cardinal Classic, and lots of earlier tournaments, I think people would be happy to see more generic "name these things from Spanish history"-type bonuses with good structure at the cost of the occasional cool bonus concept with bad structure. I think good bonus structures also show concern and creativity. People will still produce some bonuses with both good structure and cool links, but the majority of bonuses won't reach that pinnacle, just as they don't right now. For those bonuses, I think structure should trump linking.

This discussion is illustrative of why it's often not a good idea to begin a bonus "For 10 points each, name these three things associated with X." From the standpoint of a writer, it is much better to start out like this:

After a visit to the Marabar Caves, Adela Quested levels an accusation at Dr. Aziz. For 10 points each:

1. Name this novel.

The charm of starting a question this way, as a writer, is that it allows you a lot of freedom to pitch the remainder of the bonus to the estimated knowledge level of the tournament's field. If you're writing for an easy high school tournament, the second part could easily be "Name the author of A Passage to India," and the third part could ask for another well-known Forster work like A Room with a View. If you're writing for a higher-level college tournament, then this first part (asking for Passage from well-known clues) can be the easy part, and you can next ask for less well-known Forster works, or two other characters from Passage, or a character from Passage and another Forster title, etc.

As a practical matter, it's usually a good idea to start a bonus with a specific part like the one above instead of the more open-ended "Name the following three things," because you're less likely to get bogged down trying to dredge up a third appropriate "thing" in that category.

As many have said already, packets (especially in college events) are often skewed so that almost no one gets a 30 (and a few are skewed so that lots of people get 0). Make an effort not to skew packets like that; the 80/20 rule above is a good rule of thumb.

As many have said already, packets (especially in college events) are often skewed so that almost no one gets a 30 (and a few are skewed so that lots of people get 0). Make an effort not to skew packets like that; the 80/20 rule above is a good rule of thumb.

Post of the year t/f

Edit: Are you actually advocating that up to 79% of teams should get 30 points on each given bonus?

Chris Ray
OSU
University of Chicago, 2016
University of Maryland, 2014
ACF, PACE

As many have said already, packets (especially in college events) are often skewed so that almost no one gets a 30 (and a few are skewed so that lots of people get 0). Make an effort not to skew packets like that; the 80/20 rule above is a good rule of thumb.

What conversion percentage do you think I should aim for when writing questions about spurious lawsuit threats in my "packets"? I tend to include a lot of multiple-choice questions on how to pretend that an open bidding process is a sealed-bid contract and intimidate people into ceasing their criticism of me, but the players have trouble getting them right because they are so whiny and can't find their asses with both hands.

As many have said already, packets (especially in college events) are often skewed so that almost no one gets a 30 (and a few are skewed so that lots of people get 0). Make an effort not to skew packets like that; the 80/20 rule above is a good rule of thumb.

So, Mr. Avery, when will you be writing collegiate packets that follow this "80/20" rule?