Description

Clicks to items in the main menu are logged via the MobileWebMainMenuClickTracking schema for 50% of all session IDs. For AMC users we want to log all clicks and we need a way to distinguish these users from normal users.

Acceptance criteria

Update the schema to have an AMC flag

Update the schema revision number in MobileFrontend and move to Minerva codebase

Sampling rate should be 100% if the user is AMC, otherwise use the configurable sampling rate

Sign off steps

Document the oversampling behaviour for AMC on the schema's talk page so Tilman can understand it

QA steps

This can be tested on the beta cluster using an incognito window.
50% of clicks in the main menu are tracked via the MobileWebMainMenuClickTracking schema. Please verify that you can see every scenario

Test Result

Status: Need More InfoOS: macOS MojaveBrowser: ChromeDevice: MBP

@nray I tried 1a, 1b, 3a, 3b about 10 times each and only saw amc:false once. Additionally when the instructions say "no event shows up" does this mean no event with the amc:false parameter, or no event at all. I get 3 beacon events every time for 1a and 1b, and two beacon events for 3a and 3b.

In the likely event I'm doing it wrong, after I click on the hamburger menu, I right click on Random and open that in a new window. I have Chrome to open dev tools automatically so it captures the traffic in the newly opened window. Given the randomness of 1a, 1b, 3a, and 3b, what sample size would you expect to see a 50/50 breakdown?

Test Result

Status: Need More InfoOS: macOS MojaveBrowser: ChromeDevice: MBP@nray I tried 1a, 1b, 3a, 3b about 10 times each and only saw amc:false once. Additionally when the instructions say "no event shows up" does this mean no event with the amc:false parameter, or no event at all. I get 3 beacon events every time for 1a and 1b, and two beacon events for 3a and 3b.
In the likely event I'm doing it wrong, after I click on the hamburger menu, I right click on Random and open that in a new window. I have Chrome to open dev tools automatically so it captures the traffic in the newly opened window. Given the randomness of 1a, 1b, 3a, and 3b, what sample size would you expect to see a 50/50 breakdown?

First, the last step for 2a was wrong. I just corrected it to say "Inspect the request's "Query string parameters" and verify that "amc":true is in it". Sorry about that.

Instead of filtering by "beacon", filter by "MobileWebMainMenuClickTracking". I've updated the QA steps to reflect this. If you don't filter by "MobileWebMainMenuClickTracking", beacon requests might show up when the page loads that we don't care about for this ticket.

I think the other reason you might be seeing multiple beacon requests is that when you click on "Random", you might be checking the chrome dev tools for the new tab (the random page). Instead, you should be checking the chrome dev tools of the tab that you clicked the menu in (on the Spain page if you testing on https://en.m.wikipedia.beta.wmflabs.org/wiki/Spain) . When I check the beta cluster I only see 1 beacon event request (see screenshot) when I do this (and assuming other steps are followed). In addition, I ran through all of the scenarios and it seemed like I was correctly getting 50% for 1a, 1b, 3a, 3b, and 100% for 2a

Another FYI but it doesn't sound like this is affecting you, it is crucial when testing these that you are at least opening a new tab (or a completely new window) for EACH test. You cannot simply refresh the page or else you will get the same results. So if you are testing 1a/1b, roughly 5 out of 10 tabs (or windows) should have a beacon request with 'amc: false' in the query params. The rest should not have a beacon request at all.

Please ensure that you are following @phuedx 's advice as well (above)

@nray Thanks for the clarifications. @phuedx thanks for the mac tailored advice! That's some serious attention to detail. I've added screencasts so you can identify if I'm doing something wrong for 1a/b and 3a/b.

Test Result

❌ AC1a - The event and Query string parameter appeared 100% of the time instead of 50% of the time.
❌ AC1b - The event and Query string parameter appeared 100% of the time instead of NOT appearing 50% of the time.

❌ AC3a - The event and Query string parameter appeared 100% of the time instead of 50% of the time.
❌ AC3b - The event and Query string parameter appeared 100% of the time instead of NOT appearing 50% of the time.

@nray Thanks for the clarifications. @phuedx thanks for the mac tailored advice! That's some serious attention to detail. I've added screencasts so you can identify if I'm doing something wrong for 1a/b and 3a/b.

Test Result

Status: ❌ FAILOS: macOS MojaveBrowser: ChromeDevice: MBPEmulated Device: iPhoneXTest Artifact(s):
❌ AC1a - The event and Query string parameter appeared 100% of the time instead of 50% of the time.
❌ AC1b - The event and Query string parameter appeared 100% of the time instead of NOT appearing 50% of the time.

❌ AC3a - The event and Query string parameter appeared 100% of the time instead of 50% of the time.
❌ AC3b - The event and Query string parameter appeared 100% of the time instead of NOT appearing 50% of the time.

@Edtadros I think it's because you are opening up https://en.m.wikipedia.beta.wmflabs.org/wiki/Spain once and then clicking on the menu link multiple times. Instead, for a test of 1a/1b you should open up a tab of the Spain page, command-click once on a menu link and check for the request in dev tools from the Spain page. Then open up a NEW tab of the Spain page, command-click once on the menu link and check for the request in dev tools from the Spain page. Rinse and repeat this process as many times as needed to see if you follow the required percentages.

I've attached a video that shows the process for 1a/1b. You'll notice that the first time I open the Spain page there is a request with amc: true which meets the criteria for 1a, and the second time I open the Spain page there is not any request after clicking the menu link which meets the criteria for 1b. You should follow this process when testing for 2a/3a/3b as well. Does that make sense? Sorry for the confusion!

Update the schema revision number in MobileFrontend and move to Minerva codebase
Sampling rate should be 100% if the user is AMC, otherwise use the configurable sampling rate
Log whether the user is in AMC mode using the new schema revision ID