I'm working on a program that essentially removes all duplicates from a list, i.e. reducing the list to its unique components. Is there a single word or small compound word that conveys this transformation?

3 Answers
3

Deduplication is the standard term for removing extra duplicate copies from a data set, such that remaining items appear once each.

Much of the usage of deduplication before about 1980, as shown by ngrams, corresponds to its meaning as a biological term. (E.g., 1847: "This is regarded as a deduplication of the original organ ... this is regarded as a collateral deduplication of three staminal organs, ...").

Regarding its meaning as applied to data files, I believe deduplication entered common use in connection with Z39.50 efforts during the late 1970's or early 1980's.

A company I formerly worked for had an extensive deduplication process for culling repetitive documents from hard drives. The word was also frequently shortened to dedupe or deduping when working on the documents.
–
saritoninNov 12 '11 at 3:40

As an aside, nowadays, some filesystems (e.g. ZFS) do that for you. Just think how much easier that would have been...
–
scottishwildcatNov 12 '11 at 13:02

If you don’t mind its novel application to computer science, one possibility is exemplify. In the literal sense of providing a single example to represent a class of items, your list of distinct elements is set of such examples.

It sounds vaguely jargony, too. “First we obtain our list from memory, then exemplify it by reducing with function f.”