Tolliver was born one of 12 children to sharecroppers Ike and Laney Tolliver[1] in Pike Road, near Montgomery, Alabama. His exact year of birth is unknown, though it is known he was born on the Fourth of July.[1] He married his childhood friend, Willie Mae Thomas, in the 1940s and had 13 children, 11 of whom survived to adulthood.[1]

During the late 1960s, after a severe injury (his legs were crushed when a load of marble shifted and fell from a forklift as he was sweeping in the furniture factory), he turned to painting to combat boredom and long hours of idle time. Tolliver was self-taught and signed his work, "Mose T"[2] with a backward "s".[1] He regularly worked with "pure house paint" on plywood,[1] creating whimsical and sometimes erotic pictures of animals, humans, and flora. A "Quail Bird" may glide over a cotton field, or a spread-leg "Diana" or "Moose Lady" may be straddled over an exercise bicycle rack. Never able to walk well following his injury, he painted many self portraits with crutches. Watermelons and birds were also familiar themes. Tolliver's themes were drawn from his own experience.[2]

Tolliver was likely dyslexic, which may have encouraged his artistic efforts by limiting his reading and writing abilities. He would often turn his paintings upside-down and paint the picture of perhaps an animal and landscape positioned from various directions. The composition of Tolliver titles are wildly divergent; "Smoke Charlies", "Scopper Bugs" or "Jick Jack Suzy Satisfying her own Self".