README.md

Snorkel MeTaL

v0.4.1

ANNOUNCEMENT (3/20): We are thrilled to have achieved a new state-of-the-art score on the GLUE Benchmark and four of its component tasks using Snorkel MeTaL.
The code we used to accomplish this is part of a significant restructuring of multi-task end models in Snorkel MeTaL to make it as easy as possible to perform Massive Multi-Task Learning (MMTL) with supervision at varying levels of granularity and over an arbitrarily large number of tasks.
That code is being prepared for release and will be a part of Snorkel MeTaL v0.5, coming out in April!
Stay tuned on this and other Snorkel developments at our project landing page: snorkel.stanford.edu.

Getting Started

Motivation

This project builds on Snorkel in an attempt to understand how massively multi-task supervision and learning changes the way people program.
Multitask learning (MTL) is an established technique that effectively pools samples by sharing representations across related tasks, leading to better performance with less training data (for a great primer of recent advances, see this survey).
However, most existing multi-task systems rely on two or three fixed, hand-labeled training sets.
Instead, weak supervision opens the floodgates, allowing users to add arbitrarily many weakly-supervised tasks.
We call this setting massively multitask learning, and envision models with tens or hundreds of tasks with supervision of widely varying quality.
Our goal with the Snorkel MeTaL project is to understand this new regime, and the programming model it entails.

More concretely, Snorkel MeTaL is a framework for using multi-task weak supervision (MTS), provided by users in the form of labeling functions applied over unlabeled data, to train multi-task models.
Snorkel MeTaL can use the output of labeling functions developed and executed in Snorkel, or take in arbitrary label matrices representing weak supervision from multiple sources of unknown quality, and then use this to train auto-compiled MTL networks.

Snorkel MeTaL uses a new matrix approximation approach to learn the accuracies of diverse sources with unknown accuracies, arbitrary dependency structures, and structured multi-task outputs.
This makes it significantly more scalable than our previous approaches.

Note for Snorkel users: Snorkel MeTaL, even in the single-task case, learns a slightly different label model than Snorkel does (e.g. here we learn class-conditional accuracies for each LF, etc.)---so expect slightly different (hopefully better!) results.

Release Notes

Major changes in v0.4:

Upgrade to pytorch v1.0

Improved control over logging/checkpointing/validation

More modular code, separate Logger, Checkpointer, LogWriter classes

Support for user-defined metrics for validation/checkpointing

Logging frequency can now be based on seconds, examples, batches, or epochs

After running make dev to install the necessary tools, you can run make check to see if any changes you've made violate the repo standards and make fix to fix any related to isort/black. Fixes for flake8 violations will need to be made manually.

GPU Usage

MeTaL supports GPU usage, but does not include this in automatically-run tests; to run these tests, first install the requirements in tests/gpu/requirements.txt, then run: