[06:16:37] 10DBA, 10MediaWiki-extensions-WikibaseRepository, 10Wikidata, 10MW-1.32-notes (WMF-deploy-2018-05-22 (1.32.0-wmf.5)), and 2 others: Clean up indexes of wb_terms table - https://phabricator.wikimedia.org/T194273 (10Marostegui) 05stalled>03declined Let's decline this then (feel free to reopen if you thin...
[06:21:20] 10DBA, 10MediaWiki-Database, 10Performance-Team (Radar), 10User-Marostegui: Replace parsercache keys to something more meaningful on db-XXXX.php - https://phabricator.wikimedia.org/T210725 (10Marostegui)
[06:21:26] 10DBA, 10Operations, 10Patch-For-Review, 10codfw-rollout: [RFC] improve parsercache replication and sharding handling - https://phabricator.wikimedia.org/T133523 (10Marostegui)
[06:23:50] 10DBA, 10Operations, 10decommission, 10ops-codfw: Decommission parsercache hosts: pc2004 pc2005 pc2006 (Dec 2018 lease return) - https://phabricator.wikimedia.org/T209858 (10Marostegui)
[07:15:41] 10Blocked-on-schema-change, 10DBA, 10MediaWiki-Database, 10Scoring-platform-team, and 2 others: Schema change for rc_this_oldid index - https://phabricator.wikimedia.org/T202167 (10Marostegui)
[07:15:48] 10Blocked-on-schema-change, 10DBA, 10Patch-For-Review, 10Schema-change: Dropping page.page_counter on wmf databases - https://phabricator.wikimedia.org/T86338 (10Marostegui)
[07:23:58] 10Blocked-on-schema-change, 10DBA, 10Patch-For-Review, 10Schema-change: Dropping page.page_counter on wmf databases - https://phabricator.wikimedia.org/T86338 (10Marostegui)
[07:24:05] 10Blocked-on-schema-change, 10DBA, 10MediaWiki-Database, 10Scoring-platform-team, and 2 others: Schema change for rc_this_oldid index - https://phabricator.wikimedia.org/T202167 (10Marostegui)
[08:08:28] 10DBA, 10MediaWiki-API, 10Patch-For-Review, 10Performance: plwiki API request is excessively slow when including a badrevid - https://phabricator.wikimedia.org/T140302 (10Marostegui) What should we do with this bug? Should https://gerrit.wikimedia.org/r/#/c/mediawiki/core/+/299018/ get some more reviewers...
[08:08:36] 10DBA, 10MediaWiki-API, 10Patch-For-Review, 10Performance: plwiki API request is excessively slow when including a badrevid - https://phabricator.wikimedia.org/T140302 (10Marostegui) p:05Triage>03Normal
[08:40:38] 10Blocked-on-schema-change, 10DBA, 10Patch-For-Review, 10Schema-change: Dropping page.page_counter on wmf databases - https://phabricator.wikimedia.org/T86338 (10Marostegui)
[08:40:44] 10Blocked-on-schema-change, 10DBA, 10MediaWiki-Database, 10Scoring-platform-team, and 2 others: Schema change for rc_this_oldid index - https://phabricator.wikimedia.org/T202167 (10Marostegui)
[08:41:04] 10Blocked-on-schema-change, 10DBA, 10Patch-For-Review, 10Schema-change: Dropping page.page_counter on wmf databases - https://phabricator.wikimedia.org/T86338 (10Marostegui) s4 eqiad progress [] labsdb1011 [] labsdb1010 [] labsdb1009 [] dbstore1002 [] db1125 [] db1121 [] db1103 [] db1102 [] db1097 [] db10...
[08:41:10] 10Blocked-on-schema-change, 10DBA, 10MediaWiki-Database, 10Scoring-platform-team, and 2 others: Schema change for rc_this_oldid index - https://phabricator.wikimedia.org/T202167 (10Marostegui) s4 eqiad progress [] labsdb1011 [] labsdb1010 [] labsdb1009 [] dbstore1002 [] db1125 [] db1121 [] db1103 [] db110...
[08:41:22] 10Blocked-on-schema-change, 10DBA, 10Patch-For-Review, 10Schema-change: Dropping page.page_counter on wmf databases - https://phabricator.wikimedia.org/T86338 (10Marostegui)
[08:41:37] 10Blocked-on-schema-change, 10DBA, 10MediaWiki-Database, 10Scoring-platform-team, and 2 others: Schema change for rc_this_oldid index - https://phabricator.wikimedia.org/T202167 (10Marostegui)
[09:59:19] marostegui: hey, next week lots of changes to API database query patterns go live (Changing queries to use change_tag table instead of tag_summary) It should actually improve the performance but if something happened. Tell me
[09:59:25] FYI mostly
[09:59:50] Amir1: Is that a good idea considering how close we are to the Christmas holidays?
[10:00:07] Amir1: Please monitor graphs closely
[10:00:20] marostegui: it's part of the train *shrugs*
[10:00:24] Sure thing
[10:01:09] Amir1: Yeah, but "lots of changes to API database query patterns" sounds like something we have to worry about
[10:01:46] When will that be released?
[10:01:54] wmf.8 train
[10:02:06] Tuesday 18th?
[10:02:17] https://phabricator.wikimedia.org/T206662
[10:02:25] to group0 (test wikis)
[10:02:36] then group1 at Wednesday
[10:02:52] I think group1 would be the best to check as wikidata and commons are there
[10:03:03] Yeah, group 0 will not make any difference really
[10:03:08] but please monitor _closely_
[10:03:31] I am sure you've done your tests and all that, but we know that when we hit production data things might not go as planned
[10:03:39] I will do my part but if something also happened there, it might be related
[10:03:59] more of "where to look if things exploded"
[10:04:39] If we are changing query patterns I would expect more load on the DBs, because of unexpected query plans and all that
[10:04:42] Is it easy to revert?
[10:04:54] as I said, it actually improves performance (that already happened in LogPager and UserContribs and RC itself)
[10:05:14] I do trust you, I just don't trust the MySQL optimizer _)
[10:05:16] :)
[10:05:23] No one does :D
[10:05:29] I prefer to be careful
[10:05:31] and neither should
[10:05:41] marostegui: yup, let me grab the gerrit patches
[10:06:02] When we start hitting production we certainly go into the unknown
[10:06:22] We have seen that unexpected queries plans multiple times
[10:06:30] So better to be ready :)
[10:07:06] And as we are close to Christmas holidays, coverage will be reduced, that's my worry :-)
[10:08:27] https://gerrit.wikimedia.org/r/c/mediawiki/core/+/473617 https://gerrit.wikimedia.org/r/c/mediawiki/core/+/476059 https://gerrit.wikimedia.org/r/c/mediawiki/core/+/476069 https://gerrit.wikimedia.org/r/c/mediawiki/core/+/476058
[10:08:38] marostegui: each one of them is for a different funcationality
[10:08:51] one is for ApiQuery, one is for Special:Undelete
[10:09:41] So you'll stop hitting tag_summary and instead go to change_tag?
[10:16:12] marostegui: yup and given that change_tag is normalized and being used in lots of places already it would read it from memory
[10:16:40] and stop loading tag_summary into memory (which we will drop it soon (TM))
[10:18:34] that is great
[10:18:44] let's hope the new queries will behave as expected
[10:33:18] 10DBA, 10Analytics, 10Analytics-Kanban, 10Data-Services, and 2 others: Create materialized views on Wiki Replica hosts for better query performance - https://phabricator.wikimedia.org/T210693 (10Banyek) In https://gerrit.wikimedia.org/r/#/c/operations/puppet/+/477503/ I have a proposal about how to install...
[10:43:11] 10DBA, 10MediaWiki-extensions-WikibaseRepository, 10Wikidata, 10MW-1.32-notes (WMF-deploy-2018-05-22 (1.32.0-wmf.5)), and 2 others: Clean up indexes of wb_terms table - https://phabricator.wikimedia.org/T194273 (10Lucas_Werkmeister_WMDE)
[12:44:05] marostegui: how much disk does commonswiki have free?
[12:51:26] And could we load a copy of wb_terms onto one of those test servers?
[13:01:07] addshore: the master has 3.3T and has used 1.9T
[13:01:27] addshore: we could copy wb_terms onto one of those, but it will take some time (even more than a day)
[13:02:58] marostegui: do we need a ticket for the copy?
[13:03:22] We want to experiment and see how much space setting up a normalized version would take
[13:03:26] Yeah, if you can that'd be nice
[13:25:42] Can do
[13:26:39] cheers
[14:51:36] 10DBA, 10Analytics, 10Analytics-Kanban, 10Data-Services, and 3 others: Create materialized views on Wiki Replica hosts for better query performance - https://phabricator.wikimedia.org/T210693 (10Banyek) @Milimetric could you be a little bit more specific please? As I understood currently you run the querie...
[14:52:02] 10DBA, 10Operations, 10Availability (MediaWiki-MultiDC), 10Performance-Team (Radar): Investigate solutions for MySQL connection pooling - https://phabricator.wikimedia.org/T196378 (10Imarlier) @jcrespo Why would we need to deploy Mediawiki in order to repoint when the master is switched? Wouldn't the prox...
[15:10:33] 10DBA, 10Operations, 10Availability (MediaWiki-MultiDC), 10Performance-Team (Radar): Investigate solutions for MySQL connection pooling - https://phabricator.wikimedia.org/T196378 (10Marostegui) >>! In T196378#4797793, @Imarlier wrote: > @jcrespo Why would we need to deploy Mediawiki in order to repoint wh...
[15:48:57] 10DBA, 10Analytics, 10Analytics-Kanban, 10Data-Services, and 3 others: Create materialized views on Wiki Replica hosts for better query performance - https://phabricator.wikimedia.org/T210693 (10Milimetric) Thanks to @Banyek for the questions and a talk we just had over hangouts, we decided to go in two di...
[15:49:04] 10DBA, 10MediaWiki-API, 10MW-1.33-notes (1.33.0-wmf.8; 2018-12-11), 10Patch-For-Review, 10Performance: plwiki API request is excessively slow when including a badrevid - https://phabricator.wikimedia.org/T140302 (10Anomie) 05Open>03Resolved a:03Anomie This should be resolved in 1.33.0-wmf.8.
[15:59:47] 10DBA, 10Analytics, 10Analytics-Kanban, 10Data-Services, and 3 others: Create materialized views on Wiki Replica hosts for better query performance - https://phabricator.wikimedia.org/T210693 (10Milimetric) 2. exact query for a materialized view that would allow us to import the revision table into Hadoop....
[16:55:53] 10DBA, 10Patch-For-Review, 10cloud-services-team (Kanban): cloudvps: dedicated openstack database - https://phabricator.wikimedia.org/T202889 (10Andrew) My current understanding is: 1) db1073 is already only being used for WMCS uses 2) The recurring issue with connections is resolved So I'm not sure there'...
[16:58:18] 10DBA, 10Patch-For-Review, 10cloud-services-team (Kanban): cloudvps: dedicated openstack database - https://phabricator.wikimedia.org/T202889 (10Marostegui) db1073 currently has wikitech db (but there's another task to follow up on its migration already)
[18:06:32] 10DBA, 10Analytics, 10Data-Services, 10User-Elukey: Hardware for cloud db replicas for analytics usage - https://phabricator.wikimedia.org/T210749 (10elukey) Opened a procurement task for 1 Cloudb replica in T211135. We are not planning to buy two hosts with the following assumption: 1) We import data onc...
[18:39:32] 10DBA, 10Gerrit, 10Operations: Convert Gerrit's to use H2 as the database after 2.16 upgrade - https://phabricator.wikimedia.org/T211139 (10Paladox)
[18:41:12] 10DBA, 10Gerrit, 10Operations: Convert Gerrit to use H2 as the database after 2.16 upgrade - https://phabricator.wikimedia.org/T211139 (10Paladox)
[18:41:20] 10DBA, 10Gerrit, 10Operations: Convert Gerrit to use H2 as the database after 2.16 upgrade - https://phabricator.wikimedia.org/T211139 (10Paladox)
[19:42:42] 10DBA, 10Analytics, 10Analytics-Kanban, 10Data-Services, and 3 others: Create materialized views on Wiki Replica hosts for better query performance - https://phabricator.wikimedia.org/T210693 (10Bstorm) To sum up discussion with @Banyek this morning: From the Cloud perspective this is a problem with the re...
[19:58:46] 10DBA, 10Analytics, 10Analytics-Kanban, 10Data-Services, and 3 others: Create materialized views on Wiki Replica hosts for better query performance - https://phabricator.wikimedia.org/T210693 (10Milimetric) +1 to @Bstorm's framing of the problem. I was going to say the same thing, and withdraw my revision...
[20:13:00] 10DBA, 10Analytics, 10Analytics-Kanban, 10Data-Services, and 3 others: Create materialized views on Wiki Replica hosts for better query performance - https://phabricator.wikimedia.org/T210693 (10Milimetric) And to follow up on my first bullet from before: 1. after more closely looking at our temporary sol...
[20:17:35] 10DBA, 10Analytics, 10Data-Services, 10User-Elukey: Hardware for cloud db replicas for analytics usage - https://phabricator.wikimedia.org/T210749 (10Milimetric) The other reason for a single host instead of redundant hosts is this: our only critical use of the box is during the first few days of the month...