Its very common to only have two of the books as you gets those early on in the area called The Forest of Inshriach. Either book 4 or 2 is found in the "The Skyhenge" area The last book which is book 3 is found on a table in Castle Langskaal. Which is much later in the story and you need to have found the Kings Key to open the one of the last doors to get to the kings room which has the 5 bells that open the secret door to fight Logath Zanta. You will go past this room to a large room that has a painted scene on it with ravens which is also the clue needed to ring the bells in the kings bedroom. On the the reverse side of that room is where the book 3 is found. Book 1 is found somewhere in Sewers of Castle Langskaal.

As a side note for those that might not know you need to complete this quest in order to potentially gain access Harkyn's Castle otherwise you wont be able to get in at all. Completing this quest will give you the item needed to access that dungeon. Posting below the link about Harkyn's Castle.