{
"cells": [
{
"cell_type": "markdown",
"metadata": {},
"source": [
"##Collating for real with CollateX\n",
"\n",
"Okay, let's do some serious hands-on collation.\n",
"\n",
"First of all we want to make sure that you have the latest version of CollateX. That's why we do…"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"collapsed": false
},
"outputs": [],
"source": [
"!pip install --pre --upgrade collatex"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"You don't need to do this every time, but make sure you do it regularly.\n",
"\n",
"Next we need to tell Python that we will be needing the Python library that holds the code for CollateX…"
]
},
{
"cell_type": "code",
"execution_count": 2,
"metadata": {
"collapsed": true
},
"outputs": [],
"source": [
"from collatex import *"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"Now we're ready to make a collation object. We do this with the slightly hermetic line of code: \n",
"\n",
" collation = Collation()\n",
" \n",
"Here the small caps `collation` is the arbitrary named variable that refers to a copy (officially it is called an *instance*) of the CollateX collation engine. We simply tell the collation library to create a new instance by saying `Collation()`."
]
},
{
"cell_type": "code",
"execution_count": 3,
"metadata": {
"collapsed": true
},
"outputs": [],
"source": [
"collation = Collation()"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"Now we add some witnesses. Each witness gets a letter or name that will identify them, and for each we add the literal text of the witness to the collation object, like so…"
]
},
{
"cell_type": "code",
"execution_count": 4,
"metadata": {
"collapsed": true
},
"outputs": [],
"source": [
"collation.add_plain_witness( \"A\", \"The quick brown fox jumped over the lazy dog.\")\n",
"collation.add_plain_witness( \"B\", \"The brown fox jumped over the dog.\" )\n",
"collation.add_plain_witness( \"C\", \"The bad fox jumped over the lazy dog.\" )"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"And now we can let CollateX do its work of collating these witnesses and sit back for about 0.001 seconds. The result will be an alignment table, so we'll refer to the result with a variable named `alignment_table`."
]
},
{
"cell_type": "code",
"execution_count": 5,
"metadata": {
"collapsed": true
},
"outputs": [],
"source": [
"alignment_table = collate(collation, layout='vertical', segmentation=False )"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"Well, that worked nicely it seems. But there's no printout, no visualization. That's okay, we can come up with a printout of the alignment table too:"
]
},
{
"cell_type": "code",
"execution_count": 6,
"metadata": {
"collapsed": false
},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"+--------+--------+--------+\n",
"| A | B | C |\n",
"+--------+--------+--------+\n",
"| The | The | The |\n",
"+--------+--------+--------+\n",
"| quick | - | bad |\n",
"+--------+--------+--------+\n",
"| brown | brown | - |\n",
"+--------+--------+--------+\n",
"| fox | fox | fox |\n",
"+--------+--------+--------+\n",
"| jumped | jumped | jumped |\n",
"+--------+--------+--------+\n",
"| over | over | over |\n",
"+--------+--------+--------+\n",
"| the | the | the |\n",
"+--------+--------+--------+\n",
"| lazy | - | lazy |\n",
"+--------+--------+--------+\n",
"| dog | dog | dog |\n",
"+--------+--------+--------+\n",
"| . | . | . |\n",
"+--------+--------+--------+\n"
]
}
],
"source": [
"print( alignment_table )"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"Usually you will want those segments that run parallel to be collected and displayed together. Since this is what most people seem to want, CollateX does that by default. We switched this option off in the example above for a moment because the result then shows you more clearly what the underlying primary structure is that CollateX returns. But for all practicle purposes you will probabably lose that `segmentation=False` option. So, let's get rid of that, and collate again…"
]
},
{
"cell_type": "code",
"execution_count": 7,
"metadata": {
"collapsed": true
},
"outputs": [],
"source": [
"alignment_table = collate(collation, layout='vertical' )"
]
},
{
"cell_type": "code",
"execution_count": 8,
"metadata": {
"collapsed": false
},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"+---------------------+---------------------+---------------------+\n",
"| A | B | C |\n",
"+---------------------+---------------------+---------------------+\n",
"| The | The | The |\n",
"+---------------------+---------------------+---------------------+\n",
"| quick | - | bad |\n",
"+---------------------+---------------------+---------------------+\n",
"| brown | brown | - |\n",
"+---------------------+---------------------+---------------------+\n",
"| fox jumped over the | fox jumped over the | fox jumped over the |\n",
"+---------------------+---------------------+---------------------+\n",
"| lazy | - | lazy |\n",
"+---------------------+---------------------+---------------------+\n",
"| dog. | dog. | dog. |\n",
"+---------------------+---------------------+---------------------+\n"
]
}
],
"source": [
"print( alignment_table )"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"The aligment table visualization is CollateX's default way of rendering a collation result. There are various ways in which one can depict collated results of course. The output in alignment table form can be a good basis for further visualizations. CollateX can also format your collation as a variant graph. This is a visualization that lets you trace from left to right through a directed network, to follow which witness carries what readings."
]
},
{
"cell_type": "code",
"execution_count": 9,
"metadata": {
"collapsed": false
},
"outputs": [
{
"data": {
"image/svg+xml": [
"\n",
"\n",
"%3\n",
"\n",
"\n",
"0\n",
"\n",
"\n",
"\n",
"2\n",
"\n",
"The\n",
"\n",
"\n",
"0->2\n",
"\n",
"\n",
"A, B, C\n",
"\n",
"\n",
"1\n",
"\n",
"\n",
"\n",
"3\n",
"\n",
"quick\n",
"\n",
"\n",
"2->3\n",
"\n",
"\n",
"A\n",
"\n",
"\n",
"4\n",
"\n",
"brown\n",
"\n",
"\n",
"2->4\n",
"\n",
"\n",
"B\n",
"\n",
"\n",
"12\n",
"\n",
"bad\n",
"\n",
"\n",
"2->12\n",
"\n",
"\n",
"C\n",
"\n",
"\n",
"3->4\n",
"\n",
"\n",
"A\n",
"\n",
"\n",
"5\n",
"\n",
"fox jumped over the\n",
"\n",
"\n",
"4->5\n",
"\n",
"\n",
"A, B\n",
"\n",
"\n",
"9\n",
"\n",
"lazy\n",
"\n",
"\n",
"5->9\n",
"\n",
"\n",
"A, C\n",
"\n",
"\n",
"10\n",
"\n",
"dog.\n",
"\n",
"\n",
"5->10\n",
"\n",
"\n",
"B\n",
"\n",
"\n",
"9->10\n",
"\n",
"\n",
"A, C\n",
"\n",
"\n",
"10->1\n",
"\n",
"\n",
"A, B, C\n",
"\n",
"\n",
"12->5\n",
"\n",
"\n",
"C\n",
"\n",
"\n",
""
],
"text/plain": [
""
]
},
"metadata": {},
"output_type": "display_data"
}
],
"source": [
"graph = collate( collation, output=\"svg\" )"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"Okay, that's all good and nice, but that's just tiny fragments—we want decent chunks of text to collate! Well, we can do that too, although is requires a little more work. Specifically for reading in text files from the file system. If we didn't do it that way, we would have to key in all characters of each witness, and that's just a lot of unnecessary work if we have done those texts already in a file. The below code uses the `open` command to open each text file and appoint the contents to a variable with an appropriately chosen name.\n",
"\n",
"The `encoding=\"utf-8\"` bit is needed because you should always tell Python which encoding your data uses. This is probably the only place and time where you will use that encoding directive: when you open a (text) file."
]
},
{
"cell_type": "code",
"execution_count": 11,
"metadata": {
"collapsed": false
},
"outputs": [],
"source": [
"collation = Collation()\n",
"witness_1859 = open( \"./fixtures/Darwin/txt/darwin1859_par1.txt\", encoding='utf-8' ).read()\n",
"witness_1860 = open( \"./fixtures/Darwin/txt/darwin1860_par1.txt\", encoding='utf-8' ).read()\n",
"witness_1861 = open( \"./fixtures/Darwin/txt/darwin1861_par1.txt\", encoding='utf-8' ).read()\n",
"witness_1866 = open( \"./fixtures/Darwin/txt/darwin1866_par1.txt\", encoding='utf-8' ).read()\n",
"witness_1869 = open( \"./fixtures/Darwin/txt/darwin1869_par1.txt\", encoding='utf-8' ).read()\n",
"witness_1872 = open( \"./fixtures/Darwin/txt/darwin1872_par1.txt\", encoding='utf-8' ).read()\n",
"collation.add_plain_witness( \"1859\", witness_1859 )\n",
"collation.add_plain_witness( \"1860\", witness_1860 )\n",
"collation.add_plain_witness( \"1861\", witness_1861 )\n",
"collation.add_plain_witness( \"1866\", witness_1866 )\n",
"collation.add_plain_witness( \"1869\", witness_1869 )\n",
"collation.add_plain_witness( \"1872\", witness_1872 )"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"Now let's check if these witnesses actually contain some text."
]
},
{
"cell_type": "code",
"execution_count": 12,
"metadata": {
"collapsed": false
},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"WHEN we look to the individuals of the same variety or sub-variety of our older cultivated plants and animals, one of the first points which strikes us, is, that they generally differ much more from each other, than do the individuals of any one species or variety in a state of nature. When we reflect on the vast diversity of the plants and animals which have been cultivated, and which have varied during all ages under the most different climates and treatment, I think we are driven to conclude that this greater variability is simply due to our domestic productions having been raised under conditions of life not so uniform as, and somewhat different from, those to which the parent-species have been exposed under nature. There is, also, I think, some probability in the view propounded by Andrew Knight, that this variability may be partly connected with excess of food. It seems pretty clear that organic beings must be exposed during several generations to the new conditions of life to cause any appreciable amount of variation; and that when the organisation has once begun to vary, it generally continues to vary for many generations. No case is on record of a variable being ceasing to be variable under cultivation. Our oldest cultivated plants, such as wheat, still often yield new varieties: our oldest domesticated animals are still capable of rapid improvement or modification.\n",
"\n"
]
}
],
"source": [
"print( witness_1859 )"
]
},
{
"cell_type": "code",
"execution_count": 13,
"metadata": {
"collapsed": false
},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"WHEN we look to the individuals of the same variety or sub-variety of our older cultivated plants and animals, one of the first points which strikes us, is, that they generally differ more from each other than do the individuals of any one species or variety in a state of nature. When we reflect on the vast diversity of the plants and animals which have been cultivated, and which have varied during all ages under the most different climates and treatment, I think we are driven to conclude that this great variability is simply due to our domestic productions having been raised under conditions of life not so uniform as, and somewhat different from, those to which the parent-species have been exposed under nature. There is also, I think, some probability in the view propounded by Andrew Knight, that this variability may be partly connected with excess of food. It seems pretty clear that organic beings must be exposed during several generations to the new conditions of life to cause any appreciable amount of variation; and that when the organisation has once begun to vary, it generally continues to vary for many generations. No case is on record of a variable being ceasing to be variable under cultivation. Our oldest cultivated plants, such as wheat, still often yield new varieties: our oldest domesticated animals are still capable of rapid improvement or modification.\n",
"\n"
]
}
],
"source": [
"print( witness_1860 )"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"And now let's collate those witnesses and let's put the result up as an HTML-formatted alignment table…"
]
},
{
"cell_type": "code",
"execution_count": 14,
"metadata": {
"collapsed": false
},
"outputs": [
{
"data": {
"text/html": [
"

\n",
"

\n",
"

1859

\n",
"

1860

\n",
"

1861

\n",
"

1866

\n",
"

1869

\n",
"

1872

\n",
"

\n",
"

\n",
"

-

\n",
"

-

\n",
"

-

\n",
"

Causes ofVariability.

\n",
"

Causes ofVariability.

\n",
"

Causes ofVariability.

\n",
"

\n",
"

\n",
"

WHEN we

\n",
"

WHEN we

\n",
"

WHEN we

\n",
"

WHEN we

\n",
"

WHEN we

\n",
"

WHEN we

\n",
"

\n",
"

\n",
"

look to

\n",
"

look to

\n",
"

look to

\n",
"

look to

\n",
"

compare

\n",
"

compare

\n",
"

\n",
"

\n",
"

the individuals ofthe same variety orsub- variety of ourolder cultivatedplants and animals,one of the firstpoints which strikesus

\n",
"

the individuals ofthe same variety orsub- variety of ourolder cultivatedplants and animals,one of the firstpoints which strikesus

\n",
"

the individuals ofthe same variety orsub- variety of ourolder cultivatedplants and animals,one of the firstpoints which strikesus

\n",
"

the individuals ofthe same variety orsub- variety of ourolder cultivatedplants and animals,one of the firstpoints which strikesus

\n",
"

the individuals ofthe same variety orsub- variety of ourolder cultivatedplants and animals,one of the firstpoints which strikesus

\n",
"

the individuals ofthe same variety orsub- variety of ourolder cultivatedplants and animals,one of the firstpoints which strikesus

\n",
"

\n",
"

\n",
"

,

\n",
"

,

\n",
"

,

\n",
"

,

\n",
"

-

\n",
"

-

\n",
"

\n",
"

\n",
"

is, that theygenerally differ

\n",
"

is, that theygenerally differ

\n",
"

is, that theygenerally differ

\n",
"

is, that theygenerally differ

\n",
"

is, that theygenerally differ

\n",
"

is, that theygenerally differ

\n",
"

\n",
"

\n",
"

much

\n",
"

-

\n",
"

-

\n",
"

-

\n",
"

-

\n",
"

-

\n",
"

\n",
"

\n",
"

more

\n",
"

more

\n",
"

more

\n",
"

more

\n",
"

-

\n",
"

more

\n",
"

\n",
"

\n",
"

from each other

\n",
"

from each other

\n",
"

from each other

\n",
"

from each other

\n",
"

from each other

\n",
"

from each other

\n",
"

\n",
"

\n",
"

,

\n",
"

-

\n",
"

-

\n",
"

-

\n",
"

more

\n",
"

-

\n",
"

\n",
"

\n",
"

than do theindividuals of anyone species orvariety in a stateof nature.

\n",
"

than do theindividuals of anyone species orvariety in a stateof nature.

\n",
"

than do theindividuals of anyone species orvariety in a stateof nature.

\n",
"

than do theindividuals of anyone species orvariety in a stateof nature.

\n",
"

than do theindividuals of anyone species orvariety in a stateof nature.

\n",
"

than do theindividuals of anyone species orvariety in a stateof nature.

\n",
"

\n",
"

\n",
"

When

\n",
"

When

\n",
"

When

\n",
"

When

\n",
"

And if

\n",
"

And if

\n",
"

\n",
"

\n",
"

we reflect on thevast diversity ofthe plants andanimals which havebeen cultivated, andwhich have variedduring all agesunder the mostdifferent climatesand treatment,

\n",
"

we reflect on thevast diversity ofthe plants andanimals which havebeen cultivated, andwhich have variedduring all agesunder the mostdifferent climatesand treatment,

\n",
"

we reflect on thevast diversity ofthe plants andanimals which havebeen cultivated, andwhich have variedduring all agesunder the mostdifferent climatesand treatment,

\n",
"

we reflect on thevast diversity ofthe plants andanimals which havebeen cultivated, andwhich have variedduring all agesunder the mostdifferent climatesand treatment,

\n",
"

we reflect on thevast diversity ofthe plants andanimals which havebeen cultivated, andwhich have variedduring all agesunder the mostdifferent climatesand treatment,

\n",
"

we reflect on thevast diversity ofthe plants andanimals which havebeen cultivated, andwhich have variedduring all agesunder the mostdifferent climatesand treatment,

\n",
"

\n",
"

\n",
"

I think

\n",
"

I think

\n",
"

I think

\n",
"

I think

\n",
"

-

\n",
"

-

\n",
"

\n",
"

\n",
"

we are driven toconclude that this

\n",
"

we are driven toconclude that this

\n",
"

we are driven toconclude that this

\n",
"

we are driven toconclude that this

\n",
"

we are driven toconclude that this

\n",
"

we are driven toconclude that this

\n",
"

\n",
"

\n",
"

greater

\n",
"

great

\n",
"

great

\n",
"

great

\n",
"

great

\n",
"

great

\n",
"

\n",
"

\n",
"

variability is

\n",
"

variability is

\n",
"

variability is

\n",
"

variability is

\n",
"

variability is

\n",
"

variability is

\n",
"

\n",
"

\n",
"

simply

\n",
"

simply

\n",
"

simply

\n",
"

simply

\n",
"

-

\n",
"

-

\n",
"

\n",
"

\n",
"

due to our domesticproductions havingbeen raised underconditions of lifenot so uniform as,and somewhatdifferent from,those to which theparent- species

\n",
"

due to our domesticproductions havingbeen raised underconditions of lifenot so uniform as,and somewhatdifferent from,those to which theparent- species

\n",
"

due to our domesticproductions havingbeen raised underconditions of lifenot so uniform as,and somewhatdifferent from,those to which theparent- species

\n",
"

due to our domesticproductions havingbeen raised underconditions of lifenot so uniform as,and somewhatdifferent from,those to which theparent- species

\n",
"

due to our domesticproductions havingbeen raised underconditions of lifenot so uniform as,and somewhatdifferent from,those to which theparent- species

\n",
"

due to our domesticproductions havingbeen raised underconditions of lifenot so uniform as,and somewhatdifferent from,those to which theparent- species

\n",
"

\n",
"

\n",
"

have

\n",
"

have

\n",
"

have

\n",
"

have

\n",
"

had

\n",
"

had

\n",
"

\n",
"

\n",
"

been exposed undernature. There is

\n",
"

been exposed undernature. There is

\n",
"

been exposed undernature. There is

\n",
"

been exposed undernature. There is

\n",
"

been exposed undernature. There is

\n",
"

been exposed undernature. There is

\n",
"

\n",
"

\n",
"

,

\n",
"

-

\n",
"

-

\n",
"

-

\n",
"

-

\n",
"

,

\n",
"

\n",
"

\n",
"

also

\n",
"

also

\n",
"

also

\n",
"

also

\n",
"

also

\n",
"

also

\n",
"

\n",
"

\n",
"

, I think

\n",
"

, I think

\n",
"

, I think

\n",
"

, I think

\n",
"

, I think

\n",
"

-

\n",
"

\n",
"

\n",
"

, some probabilityin the viewpropounded by AndrewKnight, that thisvariability may bepartly connectedwith excess of food.It seems

\n",
"

, some probabilityin the viewpropounded by AndrewKnight, that thisvariability may bepartly connectedwith excess of food.It seems

\n",
"

, some probabilityin the viewpropounded by AndrewKnight, that thisvariability may bepartly connectedwith excess of food.It seems

\n",
"

, some probabilityin the viewpropounded by AndrewKnight, that thisvariability may bepartly connectedwith excess of food.It seems

\n",
"

, some probabilityin the viewpropounded by AndrewKnight, that thisvariability may bepartly connectedwith excess of food.It seems

\n",
"

, some probabilityin the viewpropounded by AndrewKnight, that thisvariability may bepartly connectedwith excess of food.It seems