We can't see anything unusual with ceph pg query, the scrub timestamps are all current, the output of ceph pg query is attached at the end of this postOther PGs on the same OSD also have a significantly older timestamp (beyond the max time between scrubs), yet they are completely ignored.

Restarting the affected OSDs fixes the problem.

Unfortunately, it is not reproducible and this only happened twice in three months.It seems to be related to the creation of new pools/PGs as it appeared twice within 24 hours of the creation of a new pool.The PGs being scrubbed all belong to the new pool.

We first encountered it with 9.2.2 a few month ago and today with 10.2.2.

Two logs show that OSD::sched_scrub() is entered and the PG returned by service.first_scrub_stamp() is always the same PG. It appears that the PG is never removed from the ScrubJob set because the sched_time isn't advancing which is impossible if it were removed and re-added.

As far as I can tell unreg_next_scrub()and then reg_next_scrub() must be called in scrub_finish(). I see the log message from _scrub_finish() that it calls.

One explanation that would cause this is for is_primary() to be returning false. This would cause unreg_next_scrub() and reg_next_scrub() to do nothing. However, PG::sched_scrub() won't proceed if !is_primary(), so that would be impossible.