Finally solved this by completely rebuilding the way the FlipView works. If the FlipView is initialized with a really large "virtual" set (i.e. no content), then all I have to do when scrolling is update the content, not messing with the FlipView's index or item count.

Hope it helps anyone else.

EDIT:

I made a code snippet from the implementation. However, looking back it just begs to also use a recyclable pattern, to prevent massive GC when scrolling a lot. The concept of the large virtual list that updates still stands. I am using general objects because my view switched the type of custom control each page had (week page, month page, etc). Hope it helps you guys, happy coding.

On the control side, we have a FlipView with just the Loaded event subscribed.