Family structure, particularly the dialectic between generations, fascinated Leo Tolstoy. In his novels ''War and Peace'' and ''Anna Karenina,'' families - peppered with disinherited offspring and crackpot great-aunts - are the center around which the universe revolves. For Tolstoy, family background and history determined an individual's personality and fate. Independence was illusory. Even in rebellion, his characters are bound by cords of blood not only to their mothers and fathers, but to their great-great-grandparents.

No doubt his own remarkable family history helped lead Leo Tolstoy (1828-1910) to this fatalism. To be born a Tolstoy was to enter one of the most celebrated, talented and resilient families in Russian history and to take one's place in a line of artists, writers, diplomats, scoundrels and eccentrics that stretched back to the 14th century. Few families in any country have succeeded, century after century, in maintaining such a pre-eminent position in political, artistic and social life. Even the 1917 Revolution, which theoretically annihilated the old world of aristocratic privilege to which the Tolstoys had always belonged, failed to end their conspicuous participation in the highest spheres of Russian culture. But, then, adaptability has alway been a strong Tolstoy trait. Alexei Nikolaevich Tolstoy (1882-1945), Leo's distant relative and Stalin's favorite novelist, demonstrated it to a fault. Most of the Tolstoys, however, abandoned their ancestral homeland shortly after the Revolution. Among them were the parents of Nikolai Tolstoy, an author born in England in 1935; he continues to live and write there today. In ''The Tolstoys'' he tells the fascinating and complicated story of his family's glorious and bloody past and, for the most part, he tells it with authority, grace and good humor. The reader will find no startling new revelations on the career or personality of Leo Tolstoy. It is his less-known but no less colorful relatives who occupy the center of attention.

The Tolstoys arrived in Russia from the west in 1353. ''A man of distinguished ancestry,'' as the historical records say, named Indris, emigrated from the Lithuanian empire to the city of Chernigov, where he served as military governor. About 100 years later, his great-grandson moved to Moscow, which was then emerging as the new center of Russian wealth and political power. The Muscovite Grand Prince Vasily the Blind rewarded him handsomely for his services to the ruler and, as a special mark of royal favor, bestowed upon him and his descendants the affectionate nickname ''tolstoy'' - ''fat.''

From that time the Tolstoys were never far from the throne. Their military talent and nearly legendary physical strength and endurance won them respect in the harsh world of medieval Russia. And, as Moscow slowly outgrew the primitive brutality that characterized its early political life, the Tolstoys began to use their minds as well as their muscles. The election of the first Romanov Czar, Michael, in 1613, finally restored a semblance of order to a weak Russia. But the Czar's power was severely limited by the old aristocratic families who schemed for influence and wealth behind the throne.

FEW families proved more adept at this game than the Tolstoys. Vasily Kharp Ivanovich Tolstoy shrewdly married his son Andrei into the well-placed Miloslavsky family in 1642, only six years before the reigning Czar, Alexei, married another Miloslavsky. For the next 50 years, the Miloslavskys enjoyed nearly unlimited prestige and influence. The Tolstoys' star soared with them. But the Miloslavskys' abuse of power later disgusted the progressive Peter the Great who eventually crushed the family. Only fast talking and fancy footwork saved the Tolstoys, including Andrei's sons, Ivan and Peter, from a similar fate. But they thrived and, as the 18th century dawned, the Tolstoy family was well situated to profit from Russia's belated entry into the modern world.

Nikolai Tolstoy is considerably more comfortable in relating these more recent adventures of his ancestors; the several turgid and confusing chapters about events before 1700 are the book's weakest. Overloaded with serpentine Russian names and excessive detail, they fail to guide the general reader through the Byzantine labyrinth of Muscovite political history. As it turns to the brighter, more civilized 18th century, however, the narrative suddenly relaxes, opens out and begins to flow.

Eighteenth-century Russian monarchs were in desperate need of educated and sophisticated men who could deal effectively with Europeans both at home and abroad. Peter the Great recognized one in Peter Andreevich Tolstoy (1645-1729). Peter Andreevich served brilliantly in the nearly impossible position of Russian Ambassador to the Ottoman court at Constantinople, enduring periodic imprisonment in dank dungeons while managing to defend his country's interests. Later, he turned his diplomatic skills to more sinister purposes. Ever eager to fulfill his Czar's commands, Peter Andreevich was responsible for the torture and eventual murder of Peter the Great's disobedient son Alexei. For this and other services, Peter Andreevich was named a Count of the Russian Empire, a title handed down to his descendants.

MORE than 100 years went by before the family produced another personality with Peter Andreevich's ambition, tact and energy. But in the 19th century the family fortunes reached their zenith. Gifted Tolstoys appeared in rapid succession. Alexander Ivanovich Osterman- Tolstoy (1770-1857) commanded bravely in Russia's victory in the Napoleonic Wars; Feodor Petrovich Tolstoy (1783-1873) achieved renown as a painter and officer of the Imperial Academy of Arts; Dmitri Andreevich Tolstoy (1823-89) served as a reactionary Minister of Education under Czar Alexander III. Nor can one forget the family's literati, the gentle poet and playwright Alexei Constantinovich Tolstoy (1817- 75) and the prolific and celebrated Leo.

Perhaps the most amusing, even bizarre, story in ''The Tolstoys'' concerns Feodor Ivanovich Tolstoy (1782-1846). In a family noted for refined intellectual and artistic achievement, his eccentrically mischievous, sometimes violent temperament clearly marked him the black sheep. As a member of an imperial emissary's suite on a diplomatic cruise bound for Japan in 1803-4, Feodor so irritated the captain with his wild and mutinous behavior that he was left behind - along with his pet ape, from which he was inseparable - to fend for himself on a desolate island off the Alaskan coast. Never at a loss, he made friends with the natives. Eventually he managed to make his way back to St. Petersburg by boat and on foot, crossing Siberia on horseback. For his exploits he earned the nickname ''The American'' and was immortalized in verse by many poets of the day, including his close friend Alexander Pushkin.

TODAY, however, the world remembers the Tolstoy family for the novels, stories, poems and plays produced by the three Tolstoy writers. Himself a writer and historian, the author of ''The Tolstoys'' is particularly drawn to them. His fascinating account of the sadly twisted and corrupt career of the last of them, the phenomenally popular Soviet historical novelist, Alexei Nikolaevich Tolstoy, is a sobering reminder of the sinister uses to which a glorious family name can be put. One of the most talented apologists for Stalinist oppression, Alexei Nikolaevich was allegedly directly responsible for the arrest and eventual death of the poet Osip Mandelstam.

Since Alexei Nikolaevich died in 1945, overfed and degraded, the Tolstoys have disappeared from the top ranks of literary and political society in their homeland. Even the legendary Tolstoy adaptability had limits, and most members of the next generation had sought refuge abroad, hoping to return one day to a kinder and nobler Russia. Now it is only the Tolstoys in exile who can maintain the family's traditions of intellectual breadth, rigor and honesty.