We were close. The commit that introduces the regression is a8137cd10a1ab06650a337c85b001cb4dba3a56d "ARM: pxa: append options of tauros2 L2 controller"

It even introduces the regression before we've explicitly turned it on on the .config (f9d35acbe063de67747826ddba2bcaa692757e1b).

Reverted those 2 commits for the time being, because I know Jon has other priorities. We could consider a fix for this (that doesn't involve reverting the speedup) if it arrives within the current release milestone, i.e. before Nov 15th.