The goal of this homework is to begin to assess the extent to which RNNs can learn to simulate compositional semantics: the way the meanings of words and phrases combine to form more complex meanings. We're going to do this with simulated data so that we have clear learning targets and so we can track the extent to which the models are truly generalizing in the desired ways.

/Applications/anaconda/envs/nlu/lib/python3.6/site-packages/h5py/__init__.py:36: FutureWarning: Conversion of the second argument of issubdtype from `float` to `np.floating` is deprecated. In future, it will be treated as `np.float64 == np.dtype(float).type`.
from ._conv import register_converters as _register_converters

defread_base_dataset(base_data_filename):"""Read in the dataset and return it in a format that lets us define it as a set. """withopen(base_data_filename,'rt')asf:base={((tuple(x),tuple(y)),z)for(x,y),zinjson.load(f)}returnbase

In [5]:

base=read_base_dataset(base_data_filename)

This is a set of triples, where the first two members are tuples (premise and hypothesis) and the third member is a label:

Complete the function sim_experiment so that it trains a TfRNNClassifier on a dataset in the format of base, prints out a classification_report and returns the trained model. Make sure all of the keyword arguments to sim_experiment are respected!

To submit:

Your completed version of sim_experiment and any supporting functions it uses.

In [12]:

defsim_experiment(train_dataset,test_dataset,embed_dim=50,hidden_dim=50,eta=0.01,max_iter=10,cell_class=tf.nn.rnn_cell.LSTMCell,hidden_activation=tf.nn.tanh):# To be completed: # Process `train_dataset` into an (X, y) pair# that is suitable for the `fit` methd of # `TfRNNClassifier`. # Train a `TfRNNClassifier` on `train_dataset`,# using all the keyword arguments given above.# Test the trained model on `test_dataset`;# assumes `test_dataset` is processed for use# with `predict` and the `classification_report`# below.# Specified printing and return value, feel free# to change the variable names if you wish:print(classification_report(y_test,predictions))returnmodel

Experiment with sim_experiment until you've found a setting where sim_experiment(base, base) yields perfect performance on all classes. (If it's a little off, that's okay.)

To submit:

Your function call to sim_experiment showing the values of all the parameters.

Tips: Definitely explore different values of cell_class and hidden_activation. You might also pick high embed_dim and hidden_dim to ensure that you have sufficient representational power. These settings in turn demand a large number of iterations.

Note: There is value in finding the smallest, or most conservative, models that will achieve this memorization, but you needn't engage in such search. Go big if you want to get this done fast!

Now that we have some indication that the model works, we want to start making the data more complex. To do this, we'll simply negate one or both arguments and assign them the relation determined by their original label and the logic of negation. For instance, the training instance

If you don't want to worry about the details, that's okay – you can treat negate_dataset as a black-box. Just think of it as implementing the theory of negation.

In [15]:

defnegate_dataset(dataset):"""Map `dataset` to a new dataset that has been thoroughly negated. Parameters ---------- dataset : set of pairs ((p, h), label) Where `p` and `h` are tuples of str. Returns ------- set Same format as `dataset`, and disjoint from it. """new_dataset=set()for(p,q),relindataset:neg_p=tuple(["not"]+list(p))neg_q=tuple(["not"]+list(q))new_dataset.add(((neg_p,neg_p),'equal'))new_dataset.add(((neg_q,neg_q),'equal'))combos=[(neg_p,neg_q),(p,neg_q),(neg_p,q)]ifrel=="disjoint":new_rels=("neutral","subset","superset")elifrel=="equal":new_rels=("equal","disjoint","disjoint")elifrel=="neutral":new_rels=("neutral","neutral","neutral")elifrel=="subset":new_rels=("superset","disjoint","neutral")elifrel=="superset":new_rels=("subset","neutral","disjoint")new_dataset|=set(zip(combos,new_rels))returnnew_dataset

Using negate_dataset, we can map the base dataset to a singly negated one:

So you got reasonably good results in the previous question. Has your model truly learned negation? To really address this question, we should see how it does on sequences of a length it hasn't seen before.

Your task:

Use your sim_experiment to train a network on the union of base and neg1, and evaluate it on the doubly negated dataset. By design, this means that your model will be evaluated on examples that are longer than those it was trained on. Use all the same keyword arguments to sim_experiment that you used for the previous question.

To submit:

The printed classification report from your run (you can just paste it in).

A note on performance: our mean F1 dropped a lot, and we expect it to drop for you too. You will not be evaluated based on the numbers you achieve, but rather only on whether you successfully run the required experiment.

(If you did really well, go a step further, by testing on the triply negated version!)