Skip to content

Use macos-26 runners and test on Intel/ARM - #2348

Merged
webknjaz merged 3 commits into
pypa:mainfrom
sirosen:use-explicit-macos-runners
Apr 1, 2026
Merged

webknjaz merged 3 commits into
pypa:mainfrom
sirosen:use-explicit-macos-runners

Conversation

@sirosen

@sirosen sirosen commented Mar 6, 2026 •

Copy link
Copy Markdown
Contributor
Contributor checklist
  • Included tests for the changes.
  • A change note is created in changelog.d/ (see changelog.d/README.md
    for instructions) or the PR text says "no changelog needed".
Maintainer checklist
  • If no changelog is needed, apply the bot:chronographer:skip label.
  • Assign the PR to an existing or new milestone for the target version
    (following Semantic Versioning).

@pyba-chronographer pyba-chronographer Bot added the bot:chronographer:provided A mark meaning that a new change log entry is present within the patch. label Mar 6, 2026
@sirosen

sirosen commented Mar 6, 2026

Copy link
Copy Markdown
Contributor Author

This appears to be working as desired, but I find the results in CI wall clock time a bit concerning.
CI time in this PR seems to be dominated by macOS runs, and they're taking ~10 minutes (or 100%) more time to complete than previous CI runs, e.g. on main.

I'm also worried that by increasing the number of builds by so much, we're increasing the changes of flaky failures. We want to solve those anyway, but "want to fix" != "have fixed".

Many projects consider it sufficient to test their max and min Python versions on Windows and macOS. I think I would be comfortable doing that at least for these new Intel-mac builds, to keep CI times more under control. It would require refactoring the job config a bit, but I think it's necessary.

@webknjaz

webknjaz commented Mar 7, 2026

Copy link
Copy Markdown
Member

I had an idea some time ago inspired by what CPython does with buildbots on exotic OS/hardware variants but haven't tried it yet. What they do is they run testing at some point post-merge due to resource availability. And if something fails, PRs get a comment with a link to the failure. The authors are supposed to fix it within 24 hours or a PR gets reverted.

The gist of what I'm considering is that we have a separate matrix for slow jobs that isn't tied into alls-green. That matrix has a special GitHub Environment assigned (slow-jobs or whatever), similar to the publishing job. But the difference would be that this env would have a cooldown timer as the only protection rule enabled. With this, the usual jobs would run quick and get a responsive status update. And if one would be pushing more PR updates, their older commits will only have results for the quick jobs (as the slow ones would be auto-canceled). But once things settle and that cooldown timer completes (in like 20 minutes or even an hour), the PR would get the slow jobs unsuspended and they'll eventually produce some results.

One corner case to think of is merge queues as for those I imagine we wouldn't want the timer.

Would you like to experiment with this idea here? I've been thinking about it for quite a while in the context of projects with C-extensions that would be noticeably slower.

@webknjaz

webknjaz commented Mar 7, 2026

Copy link
Copy Markdown
Member

Is macos 15 slower or faster, by the way?

@sirosen

sirosen commented Mar 8, 2026

Copy link
Copy Markdown
Contributor Author

It's hard to be confident, but so far it looks like

ARM macos 26
is faster than
(Intel) macos 15
is faster than
Intel macos 26

The main issue with CI time is that with a large number of macOS runners, we get a lot of queuing in GitHub. None of the jobs in our matrix are particularly slow -- I'm actually very happy with how fast they all are.

I'm game to experiment with new CI strategies here, but I don't want to complicate this "simple" PR with those changes. ... I think I'd like to merge this as-is, accepting that it raises the overall CI time to ~20 minutes. With the benefit of more time to think about it, that seems fine to me, since CI time isn't a major bottleneck for us. And i think we both have ideas about CI refinements that will make this better over time.

I'm marking this PR "ready" without further changes. Yes, things will get a little slower for a while. But we'll make them faster over time.

@sirosen
sirosen marked this pull request as ready for review March 8, 2026 03:00
@sirosen
sirosen requested a review from a team March 8, 2026 03:00
Comment thread .github/workflows/ci.yml Outdated
- Windows
- macOS
- name: Ubuntu
runs_on: ubuntu-latest

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

We should be using kebab-case keys for GHA to follow the convention consistently.

Also, complex non-primitive values in the matrix tend to be difficult to work with (and didn't use to be supported officially in the past). I wouldn't mind rendering the VM identifiers as is, w/o any extra fancy naming.

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yep, I can make those adjustments, along with the other edits.

I also don't mind rendering the identifiers verbatim in our job names -- the previous config seemed carefully arranged to produce short names (but in, IMO, a fragile way).

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yeah, I've found that we can get short enough names by spreading the parts across the reusable job name and the matrix name in the calling workflow, while still keeping the original identifiers most of the time.

Comment thread .github/workflows/ci.yml Outdated
CI-GHA,
OS-${{ runner.os }},
VM-${{ matrix.os }},
VM-${{ matrix.os.name }},

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Let's just turn this (and the matrix) into a simple

Suggested change
VM-${{ matrix.os.name }},
VM-${{ matrix.runner-vm }},

so it matches what I have in other repos as it's easier to sync.

Comment thread .github/workflows/ci.yml Outdated
strategy:
fail-fast: false
matrix:
os:

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
os:
runner-vm:

Comment thread .github/workflows/ci.yml Outdated
Comment on lines +313 to +320
- name: Ubuntu
runs_on: ubuntu-latest
- name: Windows
runs_on: windows-latest
- name: macOS
runs_on: macos-26
- name: "macOS (Intel)"
runs_on: macos-26-intel

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
- name: Ubuntu
runs_on: ubuntu-latest
- name: Windows
runs_on: windows-latest
- name: macOS
runs_on: macos-26
- name: "macOS (Intel)"
runs_on: macos-26-intel
- ubuntu-latest
- windows-latest
- macos-26
- macos-26-intel

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

(though, I'd prefer getting rid of latest for good in favor of more predicatability and tracability when looking at old CI runs)

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

100%!

I plan to come back with a dedicated PR to simply swap latest for the appropriate version numbers. 😁

sirosen added 2 commits March 23, 2026 09:05
In the matrix config, separate the notions of "os.name" (the nice name
shown in the CI GUI) from "os.runs_on" (the value used for selecting the
runner).

They are still aligned after this change, but can now be manipulated
independently.
Pin the macOS CI runners specifically to `macos-26` and `macos-26-intel`.
@sirosen
sirosen force-pushed the use-explicit-macos-runners branch from ca1b28e to 9921816 Compare March 23, 2026 14:17
Comment thread .pre-commit-config.yaml

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@sirosen would you mind moving from CLI args to a config like CPython does? Here's an example: https://github.com/python/cpython/blob/362145c20ebb08d2f850a49d356ecee858a281ae/.github/actionlint.yaml.

Using the lowest-level tool config allows people who happen to bypass our workflow tool layers (tox + pre-commit framework) to still benefit from some configuration that'd be shared with direct invocations.

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I agree fully with the rationale. I often reach past layers (especially tox) when necessary, and it's nice for things to work smoothly.

However! We got preempted on this one. actionlint just released a new version with support for macos-26-intel, so we can just update to use that.

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Ah, okay. I guess that's fine too. Though, next time we need it, I'd like to leave the config in the repo (even when it's empty).

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Does this identifier intentionally avoid linking the PR or commits?


Side note: it just occurred to me that with with #2353 we'll finally have a good case to use that highlights category soon.

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Oh, no, not on purpose! I noted at the top that I needed to link to issues. I apparently never fixed it. I've amended to link things all together.


πŸ‘ I think highlights will be the right way to announce when we move!

- Use a flat array, rather than objects in an array
- Name the key `runner-vm` to match other projects
- Allow the verbatim `runs-on` identifier to appear in job names
- Update actionlint version os that it will recognize the runner name

Co-authored-by: πŸ‡ΊπŸ‡¦ Sviatoslav Sydorenko (Бвятослав Π‘ΠΈΠ΄ΠΎΡ€Π΅Π½ΠΊΠΎ) <578543+webknjaz@users.noreply.github.com>
@sirosen
sirosen force-pushed the use-explicit-macos-runners branch from 9921816 to b1f9a0d Compare April 1, 2026 02:13
@webknjaz
webknjaz enabled auto-merge April 1, 2026 17:13
@webknjaz webknjaz closed this Apr 1, 2026
auto-merge was automatically disabled April 1, 2026 17:26

Pull request was closed

@webknjaz webknjaz reopened this Apr 1, 2026
@webknjaz
webknjaz enabled auto-merge April 1, 2026 17:26
@webknjaz

webknjaz commented Apr 1, 2026

Copy link
Copy Markdown
Member

Tip

πŸ€“

GitHub only updates ephemeral merge commits like pull/2348/merge (this is a read-only namespace that's not really visible by default) when a PR is updated (like with new pushes). Merging #2364 doesn't count since it's a different PR, even though it's been applied to the same target branch as this one (main). So just restarting the CI after merging that PR was pulling in the old ephemeral commit via pull/2348/merge.

I closed and re-opened this PR just to trigger GitHub's internal machinery that updates the ephemeral merge commit so that the CI would pick up the changes from #2364.

@webknjaz
webknjaz added this pull request to the merge queue Apr 1, 2026
@github-merge-queue
github-merge-queue Bot removed this pull request from the merge queue due to failed status checks Apr 1, 2026
@webknjaz

webknjaz commented Apr 1, 2026

Copy link
Copy Markdown
Member

New macOS seems to be getting stuck and hitting timeouts. I dropped this from the queue and will re-add it but it looks like this is going to cause problems long term.

@webknjaz
webknjaz added this pull request to the merge queue Apr 1, 2026
@sirosen

sirosen commented Apr 1, 2026

Copy link
Copy Markdown
Contributor Author

We could hold this if it seems risky.

There's a lot of unrelated cleanup we can consider which might help (dropping pypy, py3.9), and we can discuss increasing the timeouts a bit if we're running too close to those numbers.

Merged via the queue into pypa:main with commit cbd2db0 Apr 1, 2026
185 of 282 checks passed
@sirosen
sirosen deleted the use-explicit-macos-runners branch April 1, 2026 20:09
@webknjaz

webknjaz commented Apr 2, 2026

Copy link
Copy Markdown
Member

Looks like it worked in the end. Maybe, GHA was overloaded at the time. Let's observe and see how it goes in the long run.

@webknjaz

webknjaz commented Apr 2, 2026

Copy link
Copy Markdown
Member

@sirosen I've added "resolves" keywords linking #2079 and #2132 in the PR description. I overlooked that those weren't linked during review. I closed the other PR but left a question for you in the issue.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

bot:chronographer:provided A mark meaning that a new change log entry is present within the patch.

Projects

None yet

Development

Successfully merging this pull request may close these issues.

[TODO][CI] Add macOS 14 ARM 64 to the matrix

2 participants