A Connectionist Corpus-Based Approach to the Building of Word Representations (1994)

Rupert L. Tang

A system for building distributed representations for words using a
sliding window is described here which takes a corpora of sentences
from either artificial data or natural data. The system is able to
develop representations that capture the regularity of word
occurrences underlying a given body of text. The paradigm presented
here is based on a FGREP module with some variations on the
architecture and the training methodology. Experiments are done to
show that the system is capable of capturing the surface regularity of
words.