Server Admin Log

• 12:58 btullis@deploy2002: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/analytics-test: apply
• 12:58 btullis@deploy2002: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/analytics-test: apply
• 12:52 moritzm: uploaded Bird 2.18-1~wmf13u1 to component/bird-routed-ganeti for trixie-wikimedia T413740
• 12:29 andrew@dns1004: END - running authdns-update
• 12:28 andrew@dns1004: START - running authdns-update
• 12:18 ladsgroup@deploy2002: Finished scap sync-world: Backport for Reduce VP9 transcode resolution steps (T413031) (duration: 08m 02s)
• 12:17 ayounsi@cumin1003: END (PASS) - Cookbook sre.network.peering (exit_code=0) with action 'email' for AS: 20940
• 12:14 ayounsi@cumin1003: START - Cookbook sre.network.peering with action 'email' for AS: 20940
• 12:14 ladsgroup@deploy2002: ladsgroup: Continuing with sync
• 12:12 ladsgroup@deploy2002: ladsgroup: Backport for Reduce VP9 transcode resolution steps (T413031) synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
• 12:10 ladsgroup@deploy2002: Started scap sync-world: Backport for Reduce VP9 transcode resolution steps (T413031)
• 11:59 cgoubert@deploy2002: Finished scap sync-world: 1223643 : Revert "apache: Don't redirect RestSandbox on wikimedia.org" (duration: 03m 34s)
• 11:58 cgoubert@deploy2002: cgoubert: Continuing with sync
• 11:58 cgoubert@deploy2002: cgoubert: 1223643 : Revert "apache: Don't redirect RestSandbox on wikimedia.org" synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
• 11:58 Amir1: ladsgroup@deploy2002:~$ mwscript-k8s --dblist=all -- purgeUserOptions.php --login-age 11 echo-subscriptions-email-page-review (T406724)
• 11:57 cgoubert@deploy2002: Started scap sync-world: 1223643 : Revert "apache: Don't redirect RestSandbox on wikimedia.org"
• 11:32 cgoubert@deploy2002: cgoubert: 1223188 : apache: Don't redirect RestSandbox on wikimedia.org synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
• 11:31 cgoubert@deploy2002: Started scap sync-world: 1223188 : apache: Don't redirect RestSandbox on wikimedia.org
• 11:21 claime: Deploying apache config change T396807 - 1223188
• 10:58 ayounsi@cumin1003: END (PASS) - Cookbook sre.network.peering (exit_code=0) with action 'clear' for AS: 400566
• 10:58 ayounsi@cumin1003: START - Cookbook sre.network.peering with action 'clear' for AS: 400566
• 10:47 ayounsi@cumin1003: END (PASS) - Cookbook sre.network.peering (exit_code=0) with action 'email' for AS: 3223
• 10:46 ayounsi@cumin1003: START - Cookbook sre.network.peering with action 'email' for AS: 3223
• 10:43 kharlan@deploy2002: Finished scap sync-world: Backport for QuickSurveys: Enable coverage percentages for safety survey (T413022) (duration: 09m 27s)
• 10:40 moritzm: installing aom bugfix updates from Bookworm point release
• 10:36 kharlan@deploy2002: kharlan: Continuing with sync
• 10:35 kharlan@deploy2002: kharlan: Backport for QuickSurveys: Enable coverage percentages for safety survey (T413022) synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
• 10:33 kharlan@deploy2002: Started scap sync-world: Backport for QuickSurveys: Enable coverage percentages for safety survey (T413022)
• 10:32 ayounsi@cumin1003: END (PASS) - Cookbook sre.network.peering (exit_code=0) with action 'clear' for AS: 55818
• 10:31 ayounsi@cumin1003: START - Cookbook sre.network.peering with action 'clear' for AS: 55818
• 10:30 ayounsi@cumin1003: END (PASS) - Cookbook sre.network.peering (exit_code=0) with action 'configure' for AS: 18734
• 10:30 ayounsi@cumin1003: START - Cookbook sre.network.peering with action 'configure' for AS: 18734
• 10:28 ayounsi@cumin1003: END (PASS) - Cookbook sre.network.peering (exit_code=0) with action 'configure' for AS: 400566
• 10:28 ayounsi@cumin1003: START - Cookbook sre.network.peering with action 'configure' for AS: 400566
• 10:27 ayounsi@cumin1003: END (PASS) - Cookbook sre.network.peering (exit_code=0) with action 'configure' for AS: 208915
• 10:27 ayounsi@cumin1003: START - Cookbook sre.network.peering with action 'configure' for AS: 208915
• 10:22 mvernon@cumin2002: END (FAIL) - Cookbook sre.swift.remove-ghost-objects (exit_code=99) from container wikipedia-commons-local-public.f7 in codfw
• 10:19 mvernon@cumin2002: START - Cookbook sre.swift.remove-ghost-objects from container wikipedia-commons-local-public.f7 in codfw
• 09:40 kevinbazira@deploy2002: helmfile [ml-staging-codfw] Ran 'sync' command on namespace 'experimental' for release 'main' .
• 09:14 XioNoX: push pfw policies - T413833
• 09:07 moritzm: installing e2fsprogs updates from Bookworm point release
• 08:12 moritzm: installing distro-info-data updates
• 08:08 hashar@deploy2002: Finished deploy [gerrit/gerrit@c323101]: Disable banner for the 2025 developer survey (duration: 00m 11s)
• 08:08 hashar@deploy2002: Started deploy [gerrit/gerrit@c323101]: Disable banner for the 2025 developer survey
• 07:54 moritzm: installing bash updates from Bookworm point release
• 04:55 eileen: civicrm upgraded from 68ea4eab to dc2511ee
• 04:47 mwpresync@deploy2002: Finished scap sync-world: testwikis to 1.46.0-wmf.10 refs T408280 (duration: 44m 16s)
• 04:03 mwpresync@deploy2002: Started scap sync-world: testwikis to 1.46.0-wmf.10 refs T408280
• 03:04 eileen: config revision changed from 76dff80b to 596df184
• 02:44 eileen: revision changed from ddb2f36b to 76dff80b
• 02:26 eileen: civicrm upgraded from 6bd3c988 to 8751ed14
• 01:13 mwpresync@deploy2002: Finished scap build-images: Publishing wmf/next image (duration: 12m 32s)
• 01:00 mwpresync@deploy2002: Started scap build-images: Publishing wmf/next image
• 00:36 eileen: civicrm upgraded from 9d26c426 to 6bd3c988

• 23:38 ejegg: payments-wiki upgraded from 857e80f2 to 4dbf3b57
• 22:58 reedy@deploy2002: Finished scap sync-world: Backport for Remove WebAuthn keys that no longer need Wikimedia overrides (T413287) (duration: 06m 24s)
• 22:54 reedy@deploy2002: reedy: Continuing with sync
• 22:53 reedy@deploy2002: reedy: Backport for Remove WebAuthn keys that no longer need Wikimedia overrides (T413287) synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
• 22:51 reedy@deploy2002: Started scap sync-world: Backport for Remove WebAuthn keys that no longer need Wikimedia overrides (T413287)
• 22:36 andrew@cumin2002: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host cloudvirtlocal1002.eqiad.wmnet with OS trixie
• 22:31 cjming: end of UTC late backport window
• 22:29 cjming@deploy2002: Finished scap sync-world: Backport for Fetch user from primary DB when saving settings (T411804) (duration: 06m 06s)
• 22:25 cjming@deploy2002: d3r1ck01, cjming: Continuing with sync
• 22:25 cjming@deploy2002: d3r1ck01, cjming: Backport for Fetch user from primary DB when saving settings (T411804) synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
• 22:23 cjming@deploy2002: Started scap sync-world: Backport for Fetch user from primary DB when saving settings (T411804)
• 21:55 cjming@deploy2002: Finished scap sync-world: Backport for [config] Set Category Collation for arwiktionary (T413338) (duration: 09m 23s)
• 21:51 cjming@deploy2002: hubaishan, cjming: Continuing with sync
• 21:48 cjming@deploy2002: hubaishan, cjming: Backport for [config] Set Category Collation for arwiktionary (T413338) synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
• 21:46 cjming@deploy2002: Started scap sync-world: Backport for [config] Set Category Collation for arwiktionary (T413338)
• 21:44 andrew@cumin2002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on cloudvirtlocal1002.eqiad.wmnet with reason: host reimage
• 21:44 cjming@deploy2002: Finished scap sync-world: Backport for arbcom_zhwiki: Logo Changes (T413649) (duration: 11m 12s)
• 21:38 cjming@deploy2002: zhaofjx, cjming: Continuing with sync
• 21:38 jhancock@cumin1003: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host db2249.codfw.wmnet with OS bookworm
• 21:38 jhancock@cumin1003: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.hosts.reimage: Host reimage - jhancock@cumin1003"
• 21:37 andrew@cumin2002: START - Cookbook sre.hosts.downtime for 2:00:00 on cloudvirtlocal1002.eqiad.wmnet with reason: host reimage
• 21:37 cjming@deploy2002: zhaofjx, cjming: Backport for arbcom_zhwiki: Logo Changes (T413649) synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
• 21:33 cjming@deploy2002: Started scap sync-world: Backport for arbcom_zhwiki: Logo Changes (T413649)
• 21:30 jhancock@cumin1003: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.hosts.reimage: Host reimage - jhancock@cumin1003"
• 21:20 andrew@cumin2002: START - Cookbook sre.hosts.reimage for host cloudvirtlocal1002.eqiad.wmnet with OS trixie
• 21:15 jhancock@cumin1003: END (FAIL) - Cookbook sre.hosts.downtime (exit_code=99) for 2:00:00 on db2249.codfw.wmnet with reason: host reimage
• 21:07 jhancock@cumin1003: START - Cookbook sre.hosts.downtime for 2:00:00 on db2249.codfw.wmnet with reason: host reimage
• 20:58 jhancock@cumin1003: START - Cookbook sre.hosts.reimage for host db2249.codfw.wmnet with OS bookworm
• 20:50 jhancock@cumin1003: END (FAIL) - Cookbook sre.hosts.reimage (exit_code=99) for host db2249.codfw.wmnet with OS bookworm
• 20:07 jhancock@cumin1003: START - Cookbook sre.hosts.reimage for host db2249.codfw.wmnet with OS bookworm
• 19:13 swfrench@deploy2002: helmfile [codfw] DONE helmfile.d/services/shellbox-video: apply
• 19:12 swfrench@deploy2002: helmfile [codfw] START helmfile.d/services/shellbox-video: apply
• 19:12 swfrench@deploy2002: helmfile [codfw] DONE helmfile.d/services/shellbox-timeline: apply
• 19:11 swfrench@deploy2002: helmfile [codfw] START helmfile.d/services/shellbox-timeline: apply
• 19:11 swfrench@deploy2002: helmfile [codfw] DONE helmfile.d/services/shellbox-syntaxhighlight: apply
• 19:10 swfrench@deploy2002: helmfile [codfw] START helmfile.d/services/shellbox-syntaxhighlight: apply
• 19:10 swfrench@deploy2002: helmfile [codfw] DONE helmfile.d/services/shellbox-media: apply
• 19:10 swfrench@deploy2002: helmfile [codfw] START helmfile.d/services/shellbox-media: apply
• 19:09 swfrench@deploy2002: helmfile [codfw] DONE helmfile.d/services/shellbox-constraints: apply
• 19:09 swfrench@deploy2002: helmfile [codfw] START helmfile.d/services/shellbox-constraints: apply
• 19:08 swfrench@deploy2002: helmfile [codfw] DONE helmfile.d/services/shellbox: apply
• 19:07 swfrench@deploy2002: helmfile [codfw] START helmfile.d/services/shellbox: apply
• 18:59 swfrench@deploy2002: helmfile [eqiad] DONE helmfile.d/services/shellbox-video: apply
• 18:58 swfrench@deploy2002: helmfile [eqiad] START helmfile.d/services/shellbox-video: apply
• 18:58 swfrench@deploy2002: helmfile [eqiad] DONE helmfile.d/services/shellbox-timeline: apply
• 18:57 swfrench@deploy2002: helmfile [eqiad] START helmfile.d/services/shellbox-timeline: apply
• 18:57 swfrench@deploy2002: helmfile [eqiad] DONE helmfile.d/services/shellbox-syntaxhighlight: apply
• 18:56 swfrench@deploy2002: helmfile [eqiad] START helmfile.d/services/shellbox-syntaxhighlight: apply
• 18:56 swfrench@deploy2002: helmfile [eqiad] DONE helmfile.d/services/shellbox-media: apply
• 18:56 swfrench@deploy2002: helmfile [eqiad] START helmfile.d/services/shellbox-media: apply
• 18:55 swfrench@deploy2002: helmfile [eqiad] DONE helmfile.d/services/shellbox-constraints: apply
• 18:54 swfrench@deploy2002: helmfile [eqiad] START helmfile.d/services/shellbox-constraints: apply
• 18:54 swfrench@deploy2002: helmfile [eqiad] DONE helmfile.d/services/shellbox: apply
• 18:53 swfrench@deploy2002: helmfile [eqiad] START helmfile.d/services/shellbox: apply
• 18:51 swfrench@deploy2002: helmfile [staging] DONE helmfile.d/services/shellbox-video: apply
• 18:51 swfrench@deploy2002: helmfile [staging] START helmfile.d/services/shellbox-video: apply
• 18:50 swfrench@deploy2002: helmfile [staging] DONE helmfile.d/services/shellbox-timeline: apply
• 18:50 swfrench@deploy2002: helmfile [staging] START helmfile.d/services/shellbox-timeline: apply
• 18:49 swfrench@deploy2002: helmfile [staging] DONE helmfile.d/services/shellbox-syntaxhighlight: apply
• 18:49 swfrench@deploy2002: helmfile [staging] START helmfile.d/services/shellbox-syntaxhighlight: apply
• 18:48 swfrench@deploy2002: helmfile [staging] DONE helmfile.d/services/shellbox-media: apply
• 18:48 swfrench@deploy2002: helmfile [staging] START helmfile.d/services/shellbox-media: apply
• 18:48 swfrench@deploy2002: helmfile [staging] DONE helmfile.d/services/shellbox-constraints: apply
• 18:47 swfrench@deploy2002: helmfile [staging] START helmfile.d/services/shellbox-constraints: apply
• 18:47 swfrench@deploy2002: helmfile [staging] DONE helmfile.d/services/shellbox: apply
• 18:46 swfrench@deploy2002: helmfile [staging] START helmfile.d/services/shellbox: apply
• 18:36 swfrench@deploy2002: Finished scap sync-world: Rebuild deployment to pick up new production image (duration: 36m 10s)
• 18:33 andrew@dns1004: END - running authdns-update
• 18:32 andrew@dns1004: START - running authdns-update
• 18:15 bking@deploy2002: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/opensearch-ipoid: apply
• 18:15 bking@deploy2002: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/opensearch-ipoid: apply
• 18:01 swfrench@deploy2002: Started scap sync-world: Rebuild deployment to pick up new production image
• 17:57 bking@deploy2002: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/opensearch-ipoid: apply
• 17:57 bking@deploy2002: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/opensearch-ipoid: apply
• 17:55 bking@deploy2002: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/opensearch-ipoid: apply
• 17:55 bking@deploy2002: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/opensearch-ipoid: apply
• 17:47 swfrench-wmf: reprepro include php8.3_8.3.29-1+wmf11u2 in component/php83
• 17:47 jhancock@cumin1003: END (FAIL) - Cookbook sre.hosts.reimage (exit_code=99) for host db2249.codfw.wmnet with OS bookworm
• 17:08 dancy@deploy2002: Installation of scap version "4.231.0" completed for 2 hosts
• 17:07 dancy@deploy2002: Installing scap version "4.231.0" for 2 host(s)
• 16:48 jdrewniak@deploy2002: Synchronized portals: Wikimedia Portals Update: Bumping portals to master (T128546) (duration: 01m 56s)
• 16:46 jdrewniak@deploy2002: Synchronized portals/wikipedia.org/assets: Wikimedia Portals Update: Bumping portals to master (T128546) (duration: 06m 08s)
• 16:30 btullis@deploy2002: Finished scap build-images: Building new mediawiki-cli version to pick up https://gerrit.wikimedia.org/r/c/operations/dumps/+/1223213 (duration: 00m 51s)
• 16:29 btullis@deploy2002: Started scap build-images: Building new mediawiki-cli version to pick up https://gerrit.wikimedia.org/r/c/operations/dumps/+/1223213
• 16:26 jhancock@cumin1003: START - Cookbook sre.hosts.reimage for host db2249.codfw.wmnet with OS bookworm
• 16:24 jhancock@cumin1003: END (FAIL) - Cookbook sre.hosts.provision (exit_code=99) for host db2249.mgmt.codfw.wmnet with chassis set policy FORCE_RESTART
• 16:23 jhancock@cumin1003: START - Cookbook sre.hosts.provision for host db2249.mgmt.codfw.wmnet with chassis set policy FORCE_RESTART
• 16:11 fceratto@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 4:00:00 on dbstore1007.eqiad.wmnet with reason: Maintenance
• 16:11 fceratto@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 4:00:00 on db1263.eqiad.wmnet with reason: Maintenance
• 16:11 fceratto@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 4:00:00 on db1262.eqiad.wmnet with reason: Maintenance
• 16:11 fceratto@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 4:00:00 on db1261.eqiad.wmnet with reason: Maintenance
• 16:10 fceratto@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 4:00:00 on db1260.eqiad.wmnet with reason: Maintenance
• 16:10 fceratto@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 4:00:00 on db1252.eqiad.wmnet with reason: Maintenance
• 16:10 fceratto@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 4:00:00 on db1249.eqiad.wmnet with reason: Maintenance
• 16:10 fceratto@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 4:00:00 on db1248.eqiad.wmnet with reason: Maintenance
• 16:09 fceratto@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 4:00:00 on db1247.eqiad.wmnet with reason: Maintenance
• 16:09 fceratto@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 4:00:00 on db1245.eqiad.wmnet with reason: Maintenance
• 16:09 fceratto@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 4:00:00 on db1244.eqiad.wmnet with reason: Maintenance
• 16:09 fceratto@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 4:00:00 on db1243.eqiad.wmnet with reason: Maintenance
• 16:08 fceratto@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 4:00:00 on db1242.eqiad.wmnet with reason: Maintenance
• 16:08 fceratto@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 4:00:00 on db1241.eqiad.wmnet with reason: Maintenance
• 16:08 fceratto@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 4:00:00 on db1238.eqiad.wmnet with reason: Maintenance
• 16:08 fceratto@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 8:00:00 on 6 hosts with reason: Maintenance
• 16:07 fceratto@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 4:00:00 on db1221.eqiad.wmnet with reason: Maintenance
• 16:07 fceratto@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 4:00:00 on db1199.eqiad.wmnet with reason: Maintenance
• 16:07 fceratto@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 4:00:00 on db1190.eqiad.wmnet with reason: Maintenance
• 16:07 fceratto@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 4:00:00 on db1150.eqiad.wmnet with reason: Maintenance
• 15:54 fceratto@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 4:00:00 on dbstore1009.eqiad.wmnet with reason: Maintenance
• 15:53 fceratto@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 4:00:00 on db1226.eqiad.wmnet with reason: Maintenance
• 15:53 fceratto@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 4:00:00 on db1214.eqiad.wmnet with reason: Maintenance
• 15:53 fceratto@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 4:00:00 on db1203.eqiad.wmnet with reason: Maintenance
• 15:53 fceratto@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 4:00:00 on db1193.eqiad.wmnet with reason: Maintenance
• 15:52 fceratto@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 4:00:00 on db1192.eqiad.wmnet with reason: Maintenance
• 15:52 fceratto@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 4:00:00 on db1178.eqiad.wmnet with reason: Maintenance
• 15:52 fceratto@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 4:00:00 on db1177.eqiad.wmnet with reason: Maintenance
• 15:52 fceratto@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 4:00:00 on db1172.eqiad.wmnet with reason: Maintenance
• 15:51 fceratto@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 4:00:00 on db1171.eqiad.wmnet with reason: Maintenance
• 15:51 fceratto@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 8:00:00 on an-redacteddb1001.eqiad.wmnet,clouddb[1016,1020].eqiad.wmnet,db1154.eqiad.wmnet with reason: Maintenance
• 15:51 fceratto@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 4:00:00 on db1167.eqiad.wmnet with reason: Maintenance
• 15:48 fceratto@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 4:00:00 on dbstore1008.eqiad.wmnet with reason: Maintenance
• 15:48 fceratto@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 4:00:00 on db1253.eqiad.wmnet with reason: Maintenance
• 15:47 fceratto@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 4:00:00 on db1231.eqiad.wmnet with reason: Maintenance
• 15:47 fceratto@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 4:00:00 on db1227.eqiad.wmnet with reason: Maintenance
• 15:47 fceratto@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 4:00:00 on db1202.eqiad.wmnet with reason: Maintenance
• 15:46 fceratto@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 4:00:00 on db1194.eqiad.wmnet with reason: Maintenance
• 15:46 fceratto@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 4:00:00 on db1191.eqiad.wmnet with reason: Maintenance
• 15:46 fceratto@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 4:00:00 on db1181.eqiad.wmnet with reason: Maintenance
• 15:45 fceratto@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 4:00:00 on db1174.eqiad.wmnet with reason: Maintenance
• 15:45 fceratto@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 4:00:00 on db1171.eqiad.wmnet with reason: Maintenance
• 15:45 fceratto@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 4:00:00 on db1170.eqiad.wmnet with reason: Maintenance
• 15:45 fceratto@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 8:00:00 on an-redacteddb1001.eqiad.wmnet,clouddb[1014,1018].eqiad.wmnet,db1155.eqiad.wmnet with reason: Maintenance
• 15:45 fceratto@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 4:00:00 on db1158.eqiad.wmnet with reason: Maintenance
• 14:46 hashar@deploy2002: Finished deploy [gerrit/gerrit@8c906b0]: Reword schedule deployment link - T412992 (duration: 00m 12s)
• 14:46 hashar@deploy2002: Started deploy [gerrit/gerrit@8c906b0]: Reword schedule deployment link - T412992
• 14:43 stran@deploy2002: Finished scap sync-world: Backport for trwikisource: Create rollbacker user group (T410931), Enable protection indicators for ruwiki, CommonSettings: Remove EOL REL1_39 from ExtensionDistributor (T403199) (duration: 20m 31s)
• 14:42 fceratto@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 4:00:00 on dbstore1007.eqiad.wmnet with reason: Maintenance
• 14:40 fceratto@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 4:00:00 on db1240.eqiad.wmnet with reason: Maintenance
• 14:39 stran@deploy2002: reedy, stran, neriah: Continuing with sync
• 14:37 fceratto@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 8:00:00 on 6 hosts with reason: Maintenance
• 14:37 fceratto@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 4:00:00 on db1212.eqiad.wmnet with reason: Maintenance
• 14:35 fceratto@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 4:00:00 on db1198.eqiad.wmnet with reason: Maintenance
• 14:33 fceratto@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 4:00:00 on db1189.eqiad.wmnet with reason: Maintenance
• 14:31 fceratto@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 4:00:00 on db1175.eqiad.wmnet with reason: Maintenance
• 14:28 fceratto@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 4:00:00 on db1166.eqiad.wmnet with reason: Maintenance
• 14:26 fceratto@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 4:00:00 on db1157.eqiad.wmnet with reason: Maintenance
• 14:25 stran@deploy2002: reedy, stran, neriah: Backport for trwikisource: Create rollbacker user group (T410931), Enable protection indicators for ruwiki, CommonSettings: Remove EOL REL1_39 from ExtensionDistributor (T403199) synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
• 14:23 fceratto@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 4:00:00 on db1150.eqiad.wmnet with reason: Maintenance
• 14:23 stran@deploy2002: Started scap sync-world: Backport for trwikisource: Create rollbacker user group (T410931), Enable protection indicators for ruwiki, CommonSettings: Remove EOL REL1_39 from ExtensionDistributor (T403199)
• 14:16 stran@deploy2002: Finished scap sync-world: Backport for wgEnableProtectionIndicators true for frwiktionary (T413724) (duration: 06m 41s)
• 14:12 stran@deploy2002: stran, bunnypranav: Continuing with sync
• 14:11 stran@deploy2002: stran, bunnypranav: Backport for wgEnableProtectionIndicators true for frwiktionary (T413724) synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
• 14:09 stran@deploy2002: Started scap sync-world: Backport for wgEnableProtectionIndicators true for frwiktionary (T413724)
• 14:05 moritzm: installing jinja2 security updates
• 13:37 fceratto@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 4:00:00 on dbstore1007.eqiad.wmnet with reason: Maintenance
• 13:36 fceratto@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 4:00:00 on db1259.eqiad.wmnet with reason: Maintenance
• 13:36 fceratto@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 4:00:00 on db1254.eqiad.wmnet with reason: Maintenance
• 13:36 fceratto@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 4:00:00 on db1239.eqiad.wmnet with reason: Maintenance
• 13:36 fceratto@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 4:00:00 on db1233.eqiad.wmnet with reason: Maintenance
• 13:35 fceratto@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 4:00:00 on db1229.eqiad.wmnet with reason: Maintenance
• 13:35 fceratto@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 4:00:00 on db1225.eqiad.wmnet with reason: Maintenance
• 13:35 fceratto@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 4:00:00 on db1197.eqiad.wmnet with reason: Maintenance
• 13:35 fceratto@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 4:00:00 on db1188.eqiad.wmnet with reason: Maintenance
• 13:34 fceratto@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 4:00:00 on db1182.eqiad.wmnet with reason: Maintenance
• 13:34 fceratto@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 4:00:00 on db1162.eqiad.wmnet with reason: Maintenance
• 13:34 fceratto@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 8:00:00 on an-redacteddb1001.eqiad.wmnet,clouddb[1014,1018].eqiad.wmnet,db1155.eqiad.wmnet with reason: Maintenance
• 13:33 fceratto@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 4:00:00 on db1156.eqiad.wmnet with reason: Maintenance
• 13:28 fceratto@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 4:00:00 on dbstore1008.eqiad.wmnet with reason: Maintenance
• 13:28 fceratto@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 4:00:00 on db1245.eqiad.wmnet with reason: Maintenance
• 13:27 fceratto@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 4:00:00 on db1230.eqiad.wmnet with reason: Maintenance
• 13:27 fceratto@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 4:00:00 on db1216.eqiad.wmnet with reason: Maintenance
• 13:26 fceratto@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 4:00:00 on db1207.eqiad.wmnet with reason: Maintenance
• 13:25 fceratto@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 4:00:00 on db1200.eqiad.wmnet with reason: Maintenance
• 13:25 fceratto@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 4:00:00 on db1185.eqiad.wmnet with reason: Maintenance
• 13:25 fceratto@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 8:00:00 on an-redacteddb1001.eqiad.wmnet,clouddb[1016,1020].eqiad.wmnet,db1154.eqiad.wmnet with reason: Maintenance
• 13:24 fceratto@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 4:00:00 on db1161.eqiad.wmnet with reason: Maintenance
• 13:24 fceratto@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 4:00:00 on db1159.eqiad.wmnet with reason: Maintenance
• 13:22 fceratto@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 4:00:00 on dbstore1009.eqiad.wmnet with reason: Maintenance
• 13:22 fceratto@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 4:00:00 on db1225.eqiad.wmnet with reason: Maintenance
• 13:22 fceratto@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 4:00:00 on db1187.eqiad.wmnet with reason: Maintenance
• 13:22 fceratto@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 4:00:00 on db1180.eqiad.wmnet with reason: Maintenance
• 13:21 fceratto@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 4:00:00 on db1173.eqiad.wmnet with reason: Maintenance
• 13:21 fceratto@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 4:00:00 on db1168.eqiad.wmnet with reason: Maintenance
• 13:21 fceratto@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 8:00:00 on an-redacteddb1001.eqiad.wmnet,clouddb[1015,1019].eqiad.wmnet,db1155.eqiad.wmnet with reason: Maintenance
• 13:20 fceratto@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 4:00:00 on db1165.eqiad.wmnet with reason: Maintenance
• 13:00 kharlan@deploy2002: Finished scap sync-world: Backport for QuickSurveys: Enable safety survey at 0% coverage (T413022) (duration: 09m 14s)
• 12:55 kharlan@deploy2002: kharlan: Continuing with sync
• 12:52 kharlan@deploy2002: kharlan: Backport for QuickSurveys: Enable safety survey at 0% coverage (T413022) synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
• 12:50 kharlan@deploy2002: Started scap sync-world: Backport for QuickSurveys: Enable safety survey at 0% coverage (T413022)
• 12:39 btullis@deploy2002: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/postgresql-airflow-test-k8s: apply
• 12:39 btullis@deploy2002: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/postgresql-airflow-test-k8s: apply
• 12:28 btullis@deploy2002: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/analytics-test: apply
• 12:26 kevinbazira@deploy2002: helmfile [ml-staging-codfw] Ran 'sync' command on namespace 'experimental' for release 'main' .
• 12:26 btullis@deploy2002: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/analytics-test: apply
• 11:26 cgoubert@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 30 days, 0:00:00 on 34 hosts with reason: up for decom
• 11:25 cgoubert@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 30 days, 0:00:00 on 16 hosts with reason: up for decom
• 11:24 cgoubert@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 30 days, 0:00:00 on 24 hosts with reason: up for decom
• 10:57 moritzm: installing Postgresql 13 security updates
• 09:22 zabe@deploy2002: Finished scap sync-world: Backport for Pin imagelinks migration to old schema (T299953) (duration: 06m 55s)
• 09:18 zabe@deploy2002: zabe: Continuing with sync
• 09:17 zabe@deploy2002: zabe: Backport for Pin imagelinks migration to old schema (T299953) synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
• 09:15 zabe@deploy2002: Started scap sync-world: Backport for Pin imagelinks migration to old schema (T299953)
• 09:14 moritzm: installing Django security updates
• 09:12 zabe@deploy2002: Finished scap sync-world: Backport for SpecialPageLanguage: Use OOUI infuse if language selector is present (T413313) (duration: 48m 25s)
• 08:59 zabe@deploy2002: abi, zabe: Continuing with sync
• 08:59 zabe@deploy2002: abi, zabe: Backport for SpecialPageLanguage: Use OOUI infuse if language selector is present (T413313) synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
• 08:59 Emperor: depool / restart swift / repool ms-fe1014 T360913
• 08:58 Emperor: depool / restart swift / repool ms-fe1010 T360913
• 08:24 zabe@deploy2002: Started scap sync-world: Backport for SpecialPageLanguage: Use OOUI infuse if language selector is present (T413313)
• 07:20 moritzm: installing net-snmp security updates
• 04:07 Amir1: mwscript-k8s --dblist=all -- purgeUserOptions.php --login-age 11 echo-subscriptions-web-article-linked (T406724)

• 01:00 mwpresync@deploy2002: Started scap build-images: Publishing wmf/next image

• 01:19 mwpresync@deploy2002: Finished scap build-images: Publishing wmf/next image (duration: 18m 24s)
• 01:00 mwpresync@deploy2002: Started scap build-images: Publishing wmf/next image

• 14:32 inflatador: bking@wmf3062 restarting blazegraph on wdqs1014.eqiad.wmnet,wdqs1013.eqiad.wmnet,wdqs1019.eqiad.wmnet
• 01:13 mwpresync@deploy2002: Finished scap build-images: Publishing wmf/next image (duration: 13m 05s)
• 01:00 mwpresync@deploy2002: Started scap build-images: Publishing wmf/next image

• 12:31 Emperor: restart corto.service on alert1002
• 01:36 mwpresync@deploy2002: Finished scap build-images: Publishing wmf/next image (duration: 35m 31s)
• 01:00 mwpresync@deploy2002: Started scap build-images: Publishing wmf/next image

See Server Admin Log/Archives.

• SAL
• Operations