Main menu

At the end of the first century, Quintilian asked “Is it not sufficient to model our every utterance on Cicero? For my own part, I should consider it sufficient, if I could always imitate him successfully. But what harm is there in occasionally borrowing the vigour of Caesar, the vehemence of Caelius, the precision of Pollio or the sound judgment of Calvus?”

As philologists of the 21st century, we might ask “How often did Roman authors actually borrow phrases from Caesar as opposed to Cicero?”

Caitlin Diddams and I recently published an article in Digital Scholarship in the Humanities which lays out the best practices for determining:

Which phrases shared between two authors did not come from a second possible source

How to measure the relative strength of an “intertextual signal”

How to compare the relative influence of multiple authors on a cross-section of literature

As a test-case, we compared the influence of Cicero and Caesar during the early imperial and late imperial periods.

The methodology we outline in this article can be used on any number of source and target authors, regardless of language. Our formula for calculating the strength of an intertextual signal can be used with any tool for detecting intertextuality (not just Tesserae).

Relative influence in our methodology is compared according to the ‘rate of intertextuality,’ which is a normalized representation of the number of results you get in a Tesserae search. Normalization is necessary because the length of a work influences the number of results obtained. Previous methods of normalization assumed that Tesserae’s scoring algorithm would perform consistently across various authors and genres of literature. We propose that best practice should avoid such assumptions wherever possible.

Our normalization method in brief (the following is excerpted from a pre-print copy of the article):

The number of results of two searches cannot be meaningfully compared until we consider how many results each search could have produced. The number of search results depends on two factors: the level of engagement between the authors and the length of the texts being compared. Longer texts create more sentence-by-sentence comparisons. There are more opportunities for unique intertexts to occur. The number which can be meaningfully compared is not the number of unique results of a Tesserae search, but the ratio of the results found to the results that could have been found. We normalize the number of results according to the following formula:

We define the rate of intertextuality as the number of connected phrases per pair of phrases considered. This is derived by dividing the absolute value of the set of results by the absolute value of the cross-product of the sets of sentences in source and target texts. This cross-multiplication is necessary because Tesserae compares every sentence in a source text to all of the sentences in a target text.6 Therefore the number of possible results in a comparison of any source and target is the product of the number of sentences in the source and the number of sentences in the target.

Allusion was a powerful tool in the hands of Late Antique authors. Tesserae now offers a wider selection of popular Late Antique texts. Authors such as Augustine, Ambrose, Cyprian, Tertullian, Orosius, Libanius, and many others are available for Latin/Greek or multi-text searches. For a full list of available texts, click here for Latin and here for Greek.

Measuring the co-occurrence patterns of words with pointwise mutual information (PMI) can help identify bigram word-pairs that are unusually represented in the work of a given author. By comparing the PMI values of the Latin epic corpus to the PMI values of Vergil, for example, scholars can discover which word pairings are particularly Vergilian. Many of these Vergilian phrases will be obvious, such as pius Aeneas and puer Ascanius. Others, however, invite further investigation. Some word pairings are so unexpected that they may be sufficiently marked for quotation and imitation.

Tesserae is in the process of incorporating PMI data as an option for scoring search results. Tesserae scores currently rate rare words shared between two texts as more likely to constitute an allusion. This is problematic for capturing allusions from Vergil, who is known for combining common words in uncommon ways, in what ancient critics called a new form or affectation (cacozelia). The incorporation of comparative bigram frequencies can more accurately score allusions to Vergilian bigrams, which would otherwise be erroneously demoted. For example, if a search result is particularly indicative of the source author, but not of the corpus or target author, this might indicate that the target author is quoting a recognizable phrase. In this case, the Tesserae score should be increased. If the match is indicative of a target author’s shared language, but not of the corpus or the source author, it is less likely that the target author is trying to evoke the source author. In this case, the Tesserae score should be decreased.

Many studies from the 1990’s on have shown the efficacy of analyzing word co-occurrence patterns in English. In 2000, Rydberg-Cox adapted existing methods for ancient Greek as a basis for philological research. PMI values represent a ratio of “actual” versus “expected” frequency with which two words appear near each.

The actual frequency of a bigram is a measurement of the frequency with which a combination of words x and y occurs. The expected bigram frequency is a measurement of the frequency with which words x and y might have occurred as a bigram based on the frequencies of its constituent unigrams. This represents the bigram frequencies we would see if the distribution of each word were independent of the distribution of the other. In reality, contextual and syntactic relationships change the likelihood that word y will follow word x, and so the actual and expected frequency values diverge. Finally, because PMI overemphasizes low-frequency collocations, it is standard practice to cut off extremely rare words and to log and normalize the results.

Results from the Aeneid and the corpus are then normalized so that PMI values can be meaningfully compared. Normalization translates the scale of the PMI values from Vergil and from Latin epic authors to a range from -1 to 1. Positive PMI values indicate that once you read one word in Vergil, the uncertainty of the next words dramatically shrinks. Negative PMI values indicate that the presence of one word in Latin epic negligibly affects the possibility pool for the next word.

Consider the following example of a high PMI value from Vergil’s Aeneid 12.338: fumantis sudore quatit, miserabile caesis.Fumantis sudore describes horses frothing with sweat, and has a normalized PMI score of 0.684. Since PMI values for Vergil indicate that fumo usually occurs with incense, altars, food and homes, and sudor usually occurs with people, blood, and labor, fumantis sudore is “marked” or unusual phrase in Vergil. Since fumantis sudore is not a high ranking result in the PMI values of the Latin epic corpus, it is further likely to be an example of particularly “Vergilian” language.

The following graph shows the PMI values for Vergilian bigrams that also exist in the epic corpus. Most of the PMIs are positive, indicating strong associations between words. The data with the highest PMI values represents the strongest word associations in Vergil.

The next graph shows the PMI values for bigrams in the epic corpus that also exist in Vergil. Here, the PMIs are mostly negative. This indicates that in epic as a whole, word association is more flexible than in Vergil alone.

These graphs indicate that Vergil’s word associations as different from those in the epic corpus generally. For example, Vergil’s normalized PMI for aequore~toto is about 0.5, occurring 6 times. In the rest of Latin epic, aequore~toto appears 9 times and has a normalized PMI of 0.03. The difference is that in Vergil, aequore expects toto, whereas is in epic generally, aequore does not prime the reader to expect toto.

The data does not tell us which author differs from the corpus more dramatically – other normalization factors will have to be put into place before we can compare, for example, Vergil’s distance from the corpus to Lucan’s distance from the corpus. Beyond its applications for Tesserae, co-occurrence patterns can improve our understanding of what phrases are more striking or marked than others, and of what constitutes the recognizability of an ancient author’s hand.

The multitext search cross-checks your results against all other texts in the corpus. This will allow you to see whether a particular parallel is unique to your two selected works, or whether there is a broader precedent for the repeated expression.

This appendix contains the intertextual parallels that inform the paper “Measuring the Presence of Roman Rhetoric: An Intertextual Analysis of Augustine’s De Doctrina Christiana IV” published in Mouseion Vol. 14 No. 3, Open Digital Corpora of Greek and Latin. The search parameters for these comparisons are listed at the beginning of each file. Please direct any questions to Caitlin Diddams at acstaab@buffalo.edu or James Gawley at jamesgaw@buffalo.edu.

This paper examines the intertextual relationship between Augustine’s De Doctrina Christiana IV and Cicero’s Orator. We use quantitative methods to compare Augustine’s level of engagement with Orator against his engagement with other handbooks of classical Latin rhetoric. Our results inform a close reading of the text as body metaphor in DDC 4.13. Augustine incorporates Ciceronian colometry into his presentation of the epistles to demonstrate Paul’s eloquence. We argue that Augustine’s comparatively heavy use of Cicero is an attempt to justify the use of rhetoric in Christian teaching while adapting that rhetoric to Christian purposes.

All the major works of Lactantius, known as “the Christian Cicero,” are now available for Tesserae searches: Carmen de Passione Domini, De Ave Phoenice, De Ira Dei, De Mortibus Persecutorum, De Opificio Dei, Divinarum Institutionum, and the Epitome Divinarum Institutionum.

When Augustine quotes portions of Paul’s epistles in De Doctrina Christiana 4, he records versions that are not attested by the Vulgate tradition or the Old Latin (Versio Antiqua) tradition. Since Augustine quotes closely from the Vulgate, (and sometimes from the Versio Antiqua), for Gospel and Old Testament passages in DDC 4, why not for Paul’s epistles? My data suggests that these Pauline passages, which appear as examples of style rather than content, are in fact rendered in a more Ciceronian style than alternative translations.

The Pauline passages are a mosaic of Vulgate and Versio Antiqua renderings, mixed with variations that have no authority in either manuscript tradition. Variants are sometimes semantic changes, and at other times simply change the rhythm of the prose – an element of style Augustine is very concerned with (DDC 4.41). The following are examples taken from the first extended Pauline quotation (2nd Corinthians 11:16-31) in DDC 4.12:

DDC: Toleratis enim si quis vos in servitutem redigit

Vul: Sustinetis enim si quis vos in servitutem redigit

VA: Suffertis enim si quis vos in servitutem redigit

DDC: Si gloriari oportet in iis quae infirmitatis meae sunt

Vul: Si gloriari oportet quae infirmitatis meae sunt

VA: Si gloriari oportet quae infirmitatis meae sunt

Below are links to visual comparisons powered by Juxta Commons of Paul’s language as it appears in De Doctrina Christiana, and Sabatier’s Vulgate and Versio Antiqua, and exemplary passages from Psalms and Matthew. There are four options to visualize the differences: a heat map with hyperlinked variants, a side-by-side comparison, a histogram, and a VM model where all three versions can be viewed side-by-side (click “new version” after clicking on the VM button).

While it is immediately clear that the Pauline passages show more variations than non-Pauline passages, it is difficult to get a sense of the degree of difference because the passages differ so severely in length. Juxta Commons, however, provides an quantitative measurement of distance from a base text to other versions. The graph below shows Juxta’s measurement of distance between the above passages as they appear in DDC and in the Vulgate / Versio Antiqua. The baseline drawn at 0.05 is the distance Juxta measures between Sabatier’s Vulgate and the Vulgate available on Perseus. It provides a sense of what degree of difference we might expect between manuscripts.

This degree of difference demands explanation. The Pauline passages appear as block quotations, discouraging the interpretation that Augustine is weaving in his own language, as is often the case in his other works, in an extemporaneous style. These passages are not tied together thematically, but appear as distinct units with little or no connective language. It is also notable that Pauline diction appears in De Doctrina Christiana as models of style. Augustine argues at length that even though Paul was untaught in the classical rules of rhetoric, his eloquence displays the qualities of classical rhetoric such as climax, scala, and well-balance membra and caesa (DDC 4.11). He follows his quotation of many Pauline passages with extended colometric analyses, drawing on Cicero’s rhetorical metaphor of the body established in Orator, and continuing in DDC 4 to outline the officia oratoris and the genera dicendi according to Cicero’s model.

Given the extreme level engagement with Cicero in DDC 4, noted consistently in scholarship, might Augustine have chosen to deviate from the Vulgate and Versio Antiqua versions in order to present a more Ciceronian version of Paul in Latin? Whether Augustine quotes this passage from a manuscript no longer extant, or makes editorial choices of his own, Tesserae can lend an objective measurement to traditional stylistic analysis. The following Tesserae results are from my stylistic experiment to compare the three versions of 2nd Corinthians – Augustine’s closest rendition of Paul to the Vulgate / Versio Antiqua tradition – to the entire Ciceronian corpus. Notice that Augustine’s presentation of Pauline diction finds more instances of shared language that that of the Vulgate or Old Latin version. (For a full explanation of the search parameters that target style, rather than allusion, see below at “Explanation of Search Parameters.”)

*NB Tesserae expects to display results from one source text, but Cicero’s corpus includes many texts in one file. Tesserae only displays the location information for Cicero according to its initial processing. To find the location of a Ciceronian match, simply search for the exact string of characters in from the above results in cicero.corpus_1.tesscicero.corpus_2.tess. (The file is split into two parts so as not to exceed upload capacity; copy and paste cicero.corpus_2 into cicero.corpus_1 for the full .tess file.)

Examples of matches:

DDC: quoniamquidem multi gloriantur

Cicero Tusc. 3.66: quoniamquidem res in nostra potestate est

DDC: Iterum dico ne quis me existimet

Cicero Ver. 2.5.9: ne quis emeret nisi in demortui locum

These results are not particularly meaningful by themselves, but the composite of all the possible results like this is key component of style.

This graph displays the total number of times Tesserae found a set of match-words in Cicero’s corpus shared by each version of 2nd Corinthians. Thus the first column shows the total number of times Tesserae found one to four word chunks in the same position relative to each other in DDC and Cicero’s corpus. No similar pattern emerges from the same three passages tested against the corpora of Caesar, Tacitus, or even against another rhetorical work, Quintilian’s Institutio Oratoria. (Compare here: CaesC DDCCaesC VACaesC Vul; Q DDCQ VAQ Vul; TC DDCTC VATC Vul.) This suggests that Augustine’s translation does not intertext more frequently with classical works generally, and suggests that Augustine’s presentation of Paul’s language is indeed more “Ciceronian” than other Latin translations.

Support for this method of stylistic analysis:

For centuries, Lactantius has been called “the Christian Cicero.” Jerome ascribes to Lactantius “Tullian eloquence” (Ep. 58.10), developed by Pico della Mirandola as “Ciceronem sed Christianum.” Even in the present day authors argue that Lactantius deserves this title due to the form and elegance of his language. Tertullian and Justin, other early late antique prose authors, receive no such recognition – in fact, their styles are often disparaged even though their works were extremely popular. Tesserae is able to capture this difference in style. Lactantius’ De Mortibus Persecutorum intertexts with Cicero’s corpus at a much higher rate that similarly sized selections from Tertullian’s Apologeticum and Justin’s Epitome:

(This graph displays the rate of intertextuality rather than number of intertexts since these large selections must be normalized by number of phrases. Rate of intertextuality = {number of Tesserae results / (Number of source text phrases * Number of target text phrases)}

This stylistic measurement remains consistent even when very small chunks of texts are compared to Cicero’s corpus. Following are Tesserae comparisions of small selections chosen at random from Lactantius, Tertullian, and Justin, each about the same size as 2nd Corinthians 11:16-31.

Averaging these results, we find that Lactantian language even at a small scales intertexts with Cicero’s corpus much more than that of Tertullian or Justin:

This method supports the observations of ancients and scholars to this day who call the style of Lactantius Ciceronian. Such stylistic analysis can be used on other texts, like Augustine’s translations of Pauline diction, with greater confidence.

Future goals for this project include testing hundreds small sections of Lactantius, Tertullian, and Justin (and other authors) against Cicero’s corpus to get a more stable average. This will allow me to add meaningful error bars to the above graph and be more confident in the significance of an author matching Cicero “twice as much” as another.

Explanation of Search Parameters:

The search parameters used in these Tesserae comparisons are very different from the pre-set options geared towards finding possible instances of allusion. My searches use the following parameters:

–unit phrase This makes a Tesserae search divide units of speech by phrase, rather than by line. This option should be chosen for all prose text comparisons.

–dist 4 This constricts a Tesserae match to words that have two or fewer intervening words. The normal parameter is 10, allowing words to match across lines of poetry or across long clauses in prose. It’s important for a stylistic search to look only for words that appear very close to each other, rather than words that might constitute a element of intertextuality across a long distance. For comparisons with much smaller corpora, such as the texts of 2nd Corinthians compared to Caesar’s corpus, I widened the dist metric to 6. This includes a few more matches and provides a more stable measurement for otherwise very sparse results.

–stop 0 This allows Tesserae to search for matches that include every word in the corpus. Usually this parameter is set to 10, which excludes the top 10 most common words in search texts from results. This is helpful for eliminating results in an allusion search, where the user probably won’t want to examine results including qui, sum, et, in, etc. For a stylistic search, however, it’s extremely important to include all words, developing a composite of all the combinations of words that make up an author’s style rather than unusual or salient instances that might be allusions.

–feature word This instructs Tesserae to search for exact word matches rather than matches based on shared stems, or lemmata. Whereas an allusion might contain shared lemmata in different forms depending of local grammatical restraints, exact word matching allows Tesserae to capture elements of style like a tendency towards accusative + infinitive, a preference for a particular tense of a verb, etc. This kind of matching is much more precise for stylistic analysis.

A note on phrase-based searching:

While many other stylistic measurement exist, Tesserae is unique in measuring shared language in the context of an author’s phrase (delimited by periods and colons). Tesserae also looks not only at word frequency, but at the relative position of sets of words to each other in phrases. This helps Tesserae capture style in ways that corpus word frequency measurements do not.