I've checked our current preseed setup (more detailed notes in T436180), and I think we're good-to-go from a software POV.

OK, I've gone through the setup for this task and T436181, and I think we're good to go as-is:
manifests/site.pp has

node /^apus-be100[46789]\.eqiad\./ {
    role(cephadm::storage)
}
node /^apus-be200[46789]\.codfw\./ {
    role(cephadm::storage)
}

which is correct, and modules/profile/data/profile/installserver/preseed.yaml has

'apus-be*':
  - partman/hwraid-1dev-nvme.cfg
  - partman/custom/boss_leavelvm.cfg

I've checked modules/profile/data/profile/installserver/preseed.yaml and ms-fe102[1-9] is already set up correctly, so there's nothing that needs doing here.

I've checked modules/profile/data/profile/installserver/preseed.yaml and ms-fe202[1-9] is already set up correctly, so there's nothing needing doing here.

Someone renamed another file to this name after I reported it here, but if you see the history, there are 82 deleted edits.

Update: The deletion is still ongoing, there is a bit left but I also made one last run that is done much faster (probably done in a couple of days at most). It goes through all thumbnail containers and delete anything that is not standard size, in 33 it was only 16K thumbnails (less than 1% of the size of that container). That should take care of anything left out.

Just a note: The proper fix for this is to get rid of circular service dependency between thumbor and swift. Having something in front of swift that checks if a file exists and if not, then hit thumbor instead (don't quote me on that but a simple apache2 service can do it even I think) would also mean swift won't depend on thumbor anymore and if we wan to depool thumbor, we can simply depool the service in front of it and that would point to the service in the other dc.

@Sreejithk2000 that file appears visible to me...?

Here is another example. I am unable to restore this file as well.

@MatthewVernon totally missed/forgot the other task, sorry for the noise! 5TB are more than enough :)

After T438339 there is 5T quota for docker-registry (documented on wikitech).

Next steps:

Change #1359879 merged by jenkins-bot:

[operations/docker-images/docker-report@master] registryctl: accept relative values for --older-than

https://gerrit.wikimedia.org/r/1359879

In T320947#12485715, @Ladsgroup wrote:

They all have been decommissioned. Shall we close this ticket?

In T320947#10604227, @MatthewVernon wrote:

I expect it to be refreshed in Q1 or maybe Q2 (purchase date was 2020-08-11).

Got fixed some time in the past 12 years

I've marked this as closed. Thanks!

Change #1359879 had a related patch set uploaded (by Ahmon Dancy; author: Ahmon Dancy):

[operations/docker-images/docker-report@master] registryctl: accept relative values for --older-than

https://gerrit.wikimedia.org/r/1359879

https://thumb.wikimedia.org/wikipedia/commons/thumb/3/37/Montreal_Skyline_from_Mont_Royal_raw.png/3840px-Montreal_Skyline_from_Mont_Royal_raw.png works now and that's the highest size we allow. Even after purge of the thumbs in swift. Can we close this? The original is 1.1GB which could explain the mess.

It looks like it should be closed. I leave it to you.

Change #1359287 had a related patch set uploaded (by Elukey; author: Elukey):

[operations/puppet@production] Add a systemd timer to periodically drop the old MediaWiki images

https://gerrit.wikimedia.org/r/1359287

Change #1358802 merged by jenkins-bot:

[operations/docker-images/docker-report@master] Release version 0.0.22

https://gerrit.wikimedia.org/r/1358802

Change #1358802 had a related patch set uploaded (by Elukey; author: Elukey):

[operations/docker-images/docker-report@master] Release version 0.0.22

https://gerrit.wikimedia.org/r/1358802

Change #1358244 merged by jenkins-bot:

[operations/docker-images/docker-report@master] registryctl: add --keep-latest-n-images

https://gerrit.wikimedia.org/r/1358244

In T437297#12466879, @elukey wrote:

@dancy while creating the timer I realized that it would be nice to be able to protect the most recent "n" tags regardless if the match or not --older-than, to avoid deleting too much when we don't regularly push images to the registry. I created https://gerrit.wikimedia.org/r/c/operations/docker-images/docker-report/+/1358244, lemme know your thoughts :)

@dancy while creating the timer I realized that it would be nice to be able to protect the most recent "n" tags regardless if the match or not --older-than, to avoid deleting too much when we don't regularly push images to the registry. I created https://gerrit.wikimedia.org/r/c/operations/docker-images/docker-report/+/1358244, lemme know your thoughts :)

Change #1358244 had a related patch set uploaded (by Elukey; author: Elukey):

[operations/docker-images/docker-report@master] registryctl: add --keep-latest-n-images

https://gerrit.wikimedia.org/r/1358244

In T437297#12462237, @dancy wrote:

In T437297#12462208, @MatthewVernon wrote:

Presumably the timer could calculate the correct date itself when it fires? it's pretty straightforward in date (1) e.g.

matthew@aragorn:~$ date -Idate -d "4 weeks ago"
2026-09-24

Indeed. I hope we can spend the time to make the interface better than that.

In T437297#12462208, @MatthewVernon wrote:

Presumably the timer could calculate the correct date itself when it fires? it's pretty straightforward in date (1) e.g.

matthew@aragorn:~$ date -Idate -d "4 weeks ago"
2026-09-24

Presumably the timer could calculate the correct date itself when it fires? it's pretty straightforward in date (1) e.g.

matthew@aragorn:~$ date -Idate -d "4 weeks ago"
2026-09-24

In T437297#12460604, @elukey wrote:

Today I released the new docker-registryctl version, able to drop images with tags containing a date older than X. We could easily create a dedicated systemd timer to do it every week, decoupled from scap clean-images if we want to keep them separate.

@dancy hi! Sorry I missed your pings.

Change #1357687 merged by jenkins-bot:

[operations/docker-images/docker-report@master] Release 0.0.21

https://gerrit.wikimedia.org/r/1357687

Change #1357687 had a related patch set uploaded (by Elukey; author: Elukey):

[operations/docker-images/docker-report@master] Release 0.0.21

https://gerrit.wikimedia.org/r/1357687

Change #1354834 merged by jenkins-bot:

[operations/docker-images/docker-report@master] registryctl: add an option to drop tags containing a date

https://gerrit.wikimedia.org/r/1354834

Change #1357295 merged by MVernon:

[operations/puppet@production] swift: add ms-be1[098-100] to the rings

https://gerrit.wikimedia.org/r/1357295

Change #1357295 had a related patch set uploaded (by MVernon; author: MVernon):

[operations/puppet@production] swift: add ms-be1[098-100] to the rings

https://gerrit.wikimedia.org/r/1357295

mvernon opened https://gitlab.wikimedia.org/repos/data_persistence/swift-ring/-/merge_requests/23

as long as we address T316553: Setup alerting for GitLab projects size limits and/or T441673: Add S3 bucket metrics for apus tenants soon I guess we should be OK