Arbitrator Quiz to ensure they understand Instructions and Updates before arbitrating that Project

Adding the Arbitration Agreement/My Results was a big step in making us aware of the mistakes that were being made. Is it possible to also implement a system wherein an arbitrator had to pass a proficiency quiz of sorts to demonstrate they have read and understand the Project Instructions and Project Updates before they can download a Project for arbing? It just seems like now that the indexers can see some of their mistakes, it would behoove the accuracy of the index to make sure arbitrators are following the same rules that indexers must follow. At the very least, maybe add a place for indexers to note why they captured what they did when clicking Feedback for review, as well as a place for Arbitrators to indicate whey they 'ruled' the way they did on the arbitration field in question?

I'd just like to chime in here on this VERY important topic. As relatively new (but trying to make up for time) indexers, there are three adults in our house who are about to go back to Find A Grave with our free time. We're indexing 17th-century UK Parish Registers, and we take this difficult work very seriously. About half of the Arbs have been fine, but the other half. Phewwwww. Many of them either can't see well or can't read old handwriting. Many of them haven't read the rules (e.g., babies' or wives' surnames). Many of them have no clue how to add or remove a record in the middle of a batch. Heaven forbid that an indexer makes an error near the beginning. All of a sudden, you've got this Arb retyping another 60+ records and marking yours wrong -- all because they're now one record off in number. There should definitely be a test for reading different kinds of old handwriting and the various abbrevations and symbols used by the churches back then. What do you do with an Arb who can't read "ye" as "the" or "his wife"? It often takes us a long time to get used to a certain recorder's handwriting. Some of those capital letters can cause you to knock your head against the wall. Some Arb comes along and, in five minutes, marks down inanely stupid "corrections".

For now, our answer to this problem is that we're going to stop looking at the Arbitration Results. This feels like an important thing to do right now, but we've got other volunteer projects clamouring for our attention, too. PLEASE, oh, please, give your indexers a break and improve the quality of your checkers.

Thanks for letting me say this. It isn't fun around here like it used to be . . .

I think we've already got the basic framework in place. We have indexers, we have arbitrators, we have a way for indexers to flag arbitration mistakes. (Is this ever checked?) These ideas don't require changes to the user interface*, only to the back end:
*(though it might help)

(1) When an indexer flags a mistake, it should go back to arbitration, with a different arbitrator. Only fields that the indexer has flagged for consideration should appear as different, permitting the new arbitrator to jump right to the issue.

(2) Arbitrators can be given tests in the style of "secret shoppers". In other words, you pass them the occasional batch that has been test proofed, and see how they do. Bad arbitrators could then be given feedback, or reported to the local director. (Surely these directors can't teach their helpers - or pull them - if they don't know there's a problem.) As a side benefit, after a while you wouldn't need to create test proofs - You just pass a batch to a couple of known great arbitrators, and see what they can agree on! Just turn it back around and now you've got a test batch.

What's the worst thing that could happen if arbitrators were tested and certified before cutting them loose? You know, I can't really think of anything negative. I guess it could slow down getting a project completed by a few days or so. Can anybody else think of any reason why we should not have better quality control and accountability?

I am an arbitrator and personally I want some sort of testing, although I agree self testing won't help or the "problem arbitrators" would have asked themselves a LONG time ago why they disagree with so many indexers. When I continually mark the same fields incorrect I always go back and check the project updates to see if something has changed that week. I review the field instructions and every project update at least weekly, often daily! I WISH there were clearer instructions for each project, but even similar projects will often have field helps worded in enough different ways to make total confusion from project to project. The basic rules for indexing should stay the same and EVERY approved arbitrator needs to know those rules, backwards and forwards before they even begin arbitrating.

Not a day goes by that I haven't wished for some contact with someone at Family Search to tell me if my arbitration is okay. But I have never ever been contacted by anyone, all my training is self taught and the best training comes from reading Family Search forums at least weekly to check the latest posts.

What protects the integrity of the Indexing program? I want to know that I'm making correct choices while arbitrating and positively aiding this great work, but where does that affirmation come from? I honestly don't know! As a long time family history researcher I know these concerns are valid and need to be addressed. Just one omitted name can create a road block and be the cause of great frustration for our beginner researchers.

First, I wish there was some way for FSI to test arbitrators on how to arbitrate, i.e. beginning with record matching, which is essential on some projects. Many arbitrators either don't know how or don't want to bother with record matching, even though the tutorial stresses it and it's the first window that comes up after opening an arbitration batch. There's no excuse for an arbitrator to mark an indexer's work all wrong on say, lines 5- 40, because above line 5 somewhere that indexer just forgot one line or the arbitrator erroneously added a line. That happens because the arbitrator did not record match.

Second, they should have a mandatory test on some of the Basic Indexing Guidelines, i.e. punctuation, how to enter names, dates, ages, etc. They should show they know where to find the Basic Indexing Guidelines for reference.

Third, I agree with Dellory Matthews that each project should have a test on the PIs and PUs that arbitrators must pass to show that they read those instructions. I don't think that all indexers need a test...they'll learn through Arbitration Results eventually if the arbitrators are marking them correctly. But arbitrators, no matter how well-intentioned, are the quality assurance people here and need to know what they're doing.

Well, those that know me know I have complained openly about several ARBS and while yes they are human and make mistakes, as handwritting is perception... I feel there should be a quiz given by Family Search for each project a person wishes to ARB.. Some just have BIG HEADS and its not about the info its how many DINGS they can get and then go gossip about it on other groups.... If you cant index it...you have ABSOLUTELY no business Arbing it and even then I think all rules should be gone over with others if u question yourself at all.... Its not a contest, its not a game and its not who is the best, its about the researcher and our ancestors and now that I have seen some of the arbs work, I understand why Ill never find my family.... Please, no matter how many "LIKES" this gets, it needs to be addressed as myself and some friends have considered giving up this effort several times due to lack of consideration on the part of FSI... But we have stuck it out and think you need to do the same for us... If this was a paid job, someone out there would sure make sure its done right, but in my opinion, volunteers deserve more respect as WE dont have to be here, we choose to be here.... HELP US , HELP YOU!!

I love the idea of a required assessment for arbitrators! So many times, pages that I have indexed have been arb'd incorrectly by someone who doesn't seem to know what the instructions are. For me, my score isn't what I am concerned about (although nobody like a low score.) My main concern as an indexer, and a researcher, is to make records accessible to the people looking for them.

I think we need more than self assessment. Too many arbs think they are beyond needing to read rules and think because they have been doing it forever that the rules only apply to others. No amount of "self assessment" would help there.

I too have been totally frustrated with the way some of my batches have been arbitrated. When I began indexing I thought the arbs were tested and had to know all the rules for every project they worked on. I diligently read all instructions before beginning a new project and refer to them often so am very upset when an arb obviously hasn't. For the success of the records please change the procedures which allow anyone to qualify to become an arbitrator.

I do hope the self-assessment does not consist of general questions which assess a person's attitudes. Successful arbitration is something which cannot be presumed based on a self-assessment. It can only be demonstrated through an actual arbitration test for each and every project.

Indexers are extremely frustrated with the poor quality of work by those who are finalizing the indexing work. There is no excuse for this situation.

For many months, people have been suggesting that arbitrators PROVE they are capable. It's going to take more than a "self-assessment."

Self assessment for many (or most) of these Arbs will be meaningless; in my time of Indexing, I have never seen such POOR and unqualified assessments of Indexed batches. These people do not bother to read the PI (look at the 1940 census as a MAJOR example -- the place /location columns where people lived in 1935 and how many Arbs added "same place or same house because it was written by the census taker, yet NOT to be included when indexing).

There needs to be a testing or cleaning of house to remove these lackluster, unintelligent individuals who are doing nothing but making good Indexers want to quit in sheer frustration, as has been talked about on other forums. That, is the real shame.

I too have been totally frustrated with the way some of my batches have been arbitrated. When I began indexing I thought the arbs were tested and had to know all the rules for every project they worked on. I diligently read all instructions before beginning a new project and refer to them often so am very upset when an arb obviously hasn't. For the success of the records please change the procedures which allow anyone to qualify to become an arbitrator.

I think we need more than self assessment. Too many arbs think they are beyond needing to read rules and think because they have been doing it forever that the rules only apply to others. No amount of "self assessment" would help there.

I had a personal experience with this.I was looking for my mom in law in the 1940 cencus and could not find her.Her name was Maudine King.After searching for several days I accidently came across her.They had her as Mandino King.On top of that on the same batch they had her mom Dannie Lee as Donnie Lee.I looked at that image and that was clearly an "a" in Dannie Lee.I had a lot of low scores because some of the arbs were not going by the same rules I am required to.

I absolutely agree! Inaccurate arbitration can ruin hours of good indexing and since there is not a mechanism in place yet to allow for altering a published index, we need to take every step possible to ensure the accuracy of the arbitration.

I love the idea of a required assessment for arbitrators! So many times, pages that I have indexed have been arb'd incorrectly by someone who doesn't seem to know what the instructions are. For me, my score isn't what I am concerned about (although nobody like a low score.) My main concern as an indexer, and a researcher, is to make records accessible to the people looking for them.

Well, those that know me know I have complained openly about several ARBS and while yes they are human and make mistakes, as handwritting is perception... I feel there should be a quiz given by Family Search for each project a person wishes to ARB.. Some just have BIG HEADS and its not about the info its how many DINGS they can get and then go gossip about it on other groups.... If you cant index it...you have ABSOLUTELY no business Arbing it and even then I think all rules should be gone over with others if u question yourself at all.... Its not a contest, its not a game and its not who is the best, its about the researcher and our ancestors and now that I have seen some of the arbs work, I understand why Ill never find my family.... Please, no matter how many "LIKES" this gets, it needs to be addressed as myself and some friends have considered giving up this effort several times due to lack of consideration on the part of FSI... But we have stuck it out and think you need to do the same for us... If this was a paid job, someone out there would sure make sure its done right, but in my opinion, volunteers deserve more respect as WE dont have to be here, we choose to be here.... HELP US , HELP YOU!!

First, I wish there was some way for FSI to test arbitrators on how to arbitrate, i.e. beginning with record matching, which is essential on some projects. Many arbitrators either don't know how or don't want to bother with record matching, even though the tutorial stresses it and it's the first window that comes up after opening an arbitration batch. There's no excuse for an arbitrator to mark an indexer's work all wrong on say, lines 5- 40, because above line 5 somewhere that indexer just forgot one line or the arbitrator erroneously added a line. That happens because the arbitrator did not record match.

Second, they should have a mandatory test on some of the Basic Indexing Guidelines, i.e. punctuation, how to enter names, dates, ages, etc. They should show they know where to find the Basic Indexing Guidelines for reference.

Third, I agree with Dellory Matthews that each project should have a test on the PIs and PUs that arbitrators must pass to show that they read those instructions. I don't think that all indexers need a test...they'll learn through Arbitration Results eventually if the arbitrators are marking them correctly. But arbitrators, no matter how well-intentioned, are the quality assurance people here and need to know what they're doing.

I am an arbitrator and personally I want some sort of testing, although I agree self testing won't help or the "problem arbitrators" would have asked themselves a LONG time ago why they disagree with so many indexers. When I continually mark the same fields incorrect I always go back and check the project updates to see if something has changed that week. I review the field instructions and every project update at least weekly, often daily! I WISH there were clearer instructions for each project, but even similar projects will often have field helps worded in enough different ways to make total confusion from project to project. The basic rules for indexing should stay the same and EVERY approved arbitrator needs to know those rules, backwards and forwards before they even begin arbitrating.

Not a day goes by that I haven't wished for some contact with someone at Family Search to tell me if my arbitration is okay. But I have never ever been contacted by anyone, all my training is self taught and the best training comes from reading Family Search forums at least weekly to check the latest posts.

What protects the integrity of the Indexing program? I want to know that I'm making correct choices while arbitrating and positively aiding this great work, but where does that affirmation come from? I honestly don't know! As a long time family history researcher I know these concerns are valid and need to be addressed. Just one omitted name can create a road block and be the cause of great frustration for our beginner researchers.

That's the word, Deanne. Even a volunteer needs to have some accountability. When I volunteer at the church cannery, you can be sure that the fellow in charge is watching and making sure things are done correctly. I realize that with that, the USDA is inspecting and everything has to be just so because you're dealing with food, but just because we don't have inspectors that will shut down our operation if procedure isn't followed doesn't mean rules should not be expected to be followed by the volunteers at FS.

What's the worst thing that could happen if arbitrators were tested and certified before cutting them loose? You know, I can't really think of anything negative. I guess it could slow down getting a project completed by a few days or so. Can anybody else think of any reason why we should not have better quality control and accountability?

A most recent example, too, of the sheer bufoonery of some of these Arbs:

Recently, I have been indexing batches of US soldier enlistments.

On one batch I was Indexing, there is a column on the indexing form to fill in the place (city or town) where the soldier enlisted or signed up. The batch I was doing was for soldiers who joined the army between 1815-1828.

For one individual, it indicated that he signed up at the Union court house, in South Carolina. Anyone with half a brain can see that this recruitment went to the court house in the town of Union and enlisted at that building. So, in the place column, I typed Union, then in the state column, I added South Carolina.

I checked the arbitrated batch later, and once again, some half baked Arb dinged this entry and changed the place name (town) to Union Court House.

Really???

On no map for South Carolina, is there a city or town that goes by the name of Union Court House, SC.

This is nothing but a moronic, lack of paying attention by these Arbs.

What is with these individuals?? Are they truly that stupid? Or do they just not care, and sit there, like a bunch of robotic monkeys, simply adding and changing information simply because something is indicated on the original document, even though the PI indicates otherwise?

Something, on some level, needs to be done. Good ans solid work done by Indexers is being butchered and mauled by these people and that is very much a disservice to the FamilySearch site, as a whole.

I think we've already got the basic framework in place. We have indexers, we have arbitrators, we have a way for indexers to flag arbitration mistakes. (Is this ever checked?) These ideas don't require changes to the user interface*, only to the back end:
*(though it might help)

(1) When an indexer flags a mistake, it should go back to arbitration, with a different arbitrator. Only fields that the indexer has flagged for consideration should appear as different, permitting the new arbitrator to jump right to the issue.

(2) Arbitrators can be given tests in the style of "secret shoppers". In other words, you pass them the occasional batch that has been test proofed, and see how they do. Bad arbitrators could then be given feedback, or reported to the local director. (Surely these directors can't teach their helpers - or pull them - if they don't know there's a problem.) As a side benefit, after a while you wouldn't need to create test proofs - You just pass a batch to a couple of known great arbitrators, and see what they can agree on! Just turn it back around and now you've got a test batch.

I'd just like to chime in here on this VERY important topic. As relatively new (but trying to make up for time) indexers, there are three adults in our house who are about to go back to Find A Grave with our free time. We're indexing 17th-century UK Parish Registers, and we take this difficult work very seriously. About half of the Arbs have been fine, but the other half. Phewwwww. Many of them either can't see well or can't read old handwriting. Many of them haven't read the rules (e.g., babies' or wives' surnames). Many of them have no clue how to add or remove a record in the middle of a batch. Heaven forbid that an indexer makes an error near the beginning. All of a sudden, you've got this Arb retyping another 60+ records and marking yours wrong -- all because they're now one record off in number. There should definitely be a test for reading different kinds of old handwriting and the various abbrevations and symbols used by the churches back then. What do you do with an Arb who can't read "ye" as "the" or "his wife"? It often takes us a long time to get used to a certain recorder's handwriting. Some of those capital letters can cause you to knock your head against the wall. Some Arb comes along and, in five minutes, marks down inanely stupid "corrections".

For now, our answer to this problem is that we're going to stop looking at the Arbitration Results. This feels like an important thing to do right now, but we've got other volunteer projects clamouring for our attention, too. PLEASE, oh, please, give your indexers a break and improve the quality of your checkers.

Thanks for letting me say this. It isn't fun around here like it used to be . . .

My arbitration results score just dropped like crazy. I went to find out why, and discovered that I had made a very irritating mistake. I typed the entire page as written, and got the given name and surname box mixed up.

I went through the results anyway, to see if there was anything I could learn. There were about a dozen discrepancies between what I put, and what was accepted. At least half of those, I was clearly right. The other half were debatable. But I can't even select "Feedback" on these, because I was clearly "wrong" too. (Not that "Feedback" does anything whatsoever)

There really needs to be a better way to ensure competent arbitration.