I would like to know why we use the following construction:
"seldom had I ever seen such fear"> Why is the sequence adverb, verb, pronoun...is there some kind of rule? We cannot do this with "always"so does this only apply to certain adverbs of time? Why not simply say " I had seldom seen such fear">

This syntax is correct. This sentence structure with the adverb at the head of it is common when the interlocutor wants to EMPHASISE (put stress on) the adverb rather than on the verb.
Look at other utterances in which the word order is at variance with the standard S-V-O:
"I really want to have a drink NOW..."
'So DO I..."

It is fine to put "seldom" at the beginning of the sentence, for emphasis. However, you do not need "ever" because it emphasises in a different way, for example "It was the worst shock I had ever had". Either "Seldom had I seen such fear" or "I had hardly ever seen such fear", but not a mixture.