in "你看过完那部小说没有?" (had you read finish that novel?), '看过'(had read) already indicated you finished reading the book in the past, '完' is redundant here.

~

in "你看完过那部小说没有？" (had you ever finish reading the novel?) '看完'(finish reading) already indicated you had finish reading it, '过' is also redundant
here. This implies 'you had tried to read that novel before, but had never been successful in finish it.'

This question only make sense if someone pick up a novel and only read half of it, then few weeks later pick it up again, read it from the start and stop at the middle again. Kind of weird, right?

Disclaim: the following are totally my personal idea. Maybe I am incorrect, but I think these are valid.

Neither of them is right to my ear. I would say

你（以前）看过那部小说没有？//// Have you read that novel?

and

你看完那部小说没有？////
Have you finished that novel?

过 just mean you did something, while 完 indicates that you finished that thing. Fortunately your 2nd choice could be understood by natives, which is equivalent to my 2nd sentence, i.e. the 过 is automatically omitted when people listen to you.