Incorporating Item Response Theory (IRT) into NLP tasks can provide valuable information about model performance and behavior. Traditionally, IRT models are learned using human response pattern data, presenting a significant bottleneck for large data sets like those required for training deep neural networks. In this I will introduce an approach to learning IRT models using response patterns generated from artificial crowds of neural network models. I will demonstrate the effectiveness of learning IRT models using machine-generated data through quantitative and qualitative analyses for tasks in NLP and vision. I will demonstrate a use-case for latent difficulty item parameters, namely training set filtering, and show that using difficulty to sample training data outperforms baseline methods.

I'll conclude by introducing Dynamic Data selection for Curriculum Learning via Ability Estimation (DDaCLAE), a curriculum learning strategy that leverages these learned IRT models to probe model ability at each training epoch to select the best training examples at that point in time. DDaCLAE adds data at a rate commensurate with the model's capability, in contrast to scheduled curricula that add data at a predetermined rate. Experimental results demonstrate that DDaCLAE is more efficient and effective than existing curriculum learning methods, improving test set accuracy while reducing training set size by up to 88%.

Seminar Speaker:

John P. Lalor

University of Notre Dame

John is an Instructor in the IT, Analytics, and Operations Department in the Mendoza College of Business at the University of Notre Dame and an ABD PhD candidate at the University of Massachusetts Amherst in the College of Information and Computer Science. At UMass John worked with Dr. Hong Yu in the Bio-NLP lab. Prior to UMass, John worked as a software developer at Eze Software in Chicago and as an IT Audit Associate for KPMG. John received his MS in Computer Science at DePaul University, and his bachelor's degree in IT Management with a minor in Irish Language & Literature from the University of Notre Dame.

Accreditation Statement

The bachelor’s degree programs in aerospace, chemical, civil, computer, electrical, environmental, and mechanical engineering are accredited by the Engineering Accreditation Commission of ABET, Inc. The bachelor’s degree program in computer science is accredited by the Computing Accreditation Commission of ABET, Inc.