As part of the Anti-Harassment Tools team's continuing work on blocking tools we are looking to measure how often blocked users attempt (and fail) to edit pages.

To start this project we'd like to add tracking to the mobile block drawer you helped us build a few months ago (this ticket.) We would appreciate any advice you'd have on where to start — happy to discuss here or have a quick meeting to get started.

One thing to note about Graphite, especially for Trevor maybe, is that over time the completeness and and correctness of the data it has degrades. Depending on its configuration, Graphite does some roll-up averages (probably not the actual technical phrase) over time. So, if you look at the numbers for this week and want to compare them to a particular week 4 months ago, the scale and accuracy might be hard to reconcile. Since it's a time-series database and not a store like elasticsearch or hadoop or something else, it's focus is on trending over time and not being able to tell you that March of 2017 had 42 blocks while last week had 13.

We ran into this at Rackspace a lot because our POs were trying to use Graphite data to compare time periods across months and the data is just never comparable. We built an ElasticSearch/Kibana setup to track user/application analytics that need solid comparisons over longer timeframes. I'm not suggesting that here but just noting a potential shortfall of the Graphite solution.

I don't know anything about our logging. :) but it does look like you can use Graphite, or you can also use the schema (db?) logging from the client. So we should be golden either way. It also looks like MobileFrontend is using mw.track (Graphite) for most of it's logging (as far as I can tell).

One thing to note about Graphite [..] is that over time the completeness and and correctness of the data it has degrades. [..] trying to use Graphite data to compare time periods across months and the data is just never comparable. [..]

Graphite can be quite confusing, and for the better part of 5 years, this bothered me as well. However, while I do recommend Prometheus over Graphite any day, I have come to understand Graphite much better and no longer believe the correctness degrades. It comes down to three things:

Configuration (For the system admin): Graphite's backend must be configured to not discard low-resolutions of data (it discards <0.1XFF by default), and to appropriately aggregate your metrics (it uses "mean average" otherwise). For example, a metric foo_bar.sum that counts "events occurred" should be aggregated using sum aggregation instead. At WMF, some metrics were misconfigured before 2013, but have been correct since.

Query range (For the user): Contrary to popular belief, Graphite does not aggregate over time. Instead, it has (for example) a separate "last 7 days" and "last 3 years" database. Recent data is in both. This means as long as you use the same source, both recent and old will be comparable. Querying "1 day this week" and then "1 day sometime last month" is a problem. Querying the whole month and, within that, looking at a day this week and a day last month is fine.

Resolution, or interval (For the user): Shorter-term databases have higher resolutions (shorter interval) than longer-term databases. For example, a metric like foo_bar.sum that is correctly aggregated and consistently receives 1 increment every second, will show "60" for recent data (resolution = 1 minute), but might show "900" for the same time period in the longer-term database (resolution = 15 minutes). The resolution can be observed by comparing the timestamps in Grafana's tooltips. To avoid this problem, our practice is generally to use "rate" instead of "sum", which we aggregate as average and remains a "rate per second" even for older data.

@Krinkle Thanks for the details! I appreciate knowing a bit more about how it actually works. I always had a sense there was a way to configure it to be better in this regard but using other tools for additional reasons is the path I always took before.

Due to my limited familiarity with MobileFrontend I'm looking for some feedback/help on this before I submit the patch for review.

I've created a patch[1] with what I think is the right implementation. All we need is a simple counter every time the BlockMessage drawer is displayed. As per the docs[2], calling mw.track() with a counter prefix should do the trick.

Any help getting this change reporting to statsd locally (I'm using our vagrant setup) will be very much appreciated. I'm currently able to see data coming into statsd from php but not from js using mw.track.

Just pointing out that the main question that the ticket lists is: "to measure how often blocked users attempt (and fail) to edit pages."

As far as I can see you are not sending any data "per user" so will not be able to answer the question, correct? (maybe this is planned for a second instrumentation phase) You are going to get a counter on how many times a blocker notice displays per:

This will not tell you (please correct me if i am wrong) wether is the SAME user (or bot) trying to edit over and over or many different users hitting that block message. This is an important distinction cause you could have 1000 users trying to edit receiving the block message or 1 user trying to edit a 1000 times.

This was intentional. We did not want to capture any PII just for simplicity of execution. We wanted to capture some extremely basic information as a starting point — if we find that these messages are rarely/casually/frequently displayed, then we can form some hypotheses and dig in further.

We don't plan to make any important decisions with this data, it's mostly illuminatory. 💡