Repository navigation
Use macos-26 runners and test on Intel/ARM - #2348
Conversation
|
This appears to be working as desired, but I find the results in CI wall clock time a bit concerning. I'm also worried that by increasing the number of builds by so much, we're increasing the changes of flaky failures. We want to solve those anyway, but "want to fix" != "have fixed". Many projects consider it sufficient to test their max and min Python versions on Windows and macOS. I think I would be comfortable doing that at least for these new Intel-mac builds, to keep CI times more under control. It would require refactoring the job config a bit, but I think it's necessary. |
|
I had an idea some time ago inspired by what CPython does with buildbots on exotic OS/hardware variants but haven't tried it yet. What they do is they run testing at some point post-merge due to resource availability. And if something fails, PRs get a comment with a link to the failure. The authors are supposed to fix it within 24 hours or a PR gets reverted. The gist of what I'm considering is that we have a separate matrix for slow jobs that isn't tied into alls-green. That matrix has a special GitHub Environment assigned ( One corner case to think of is merge queues as for those I imagine we wouldn't want the timer. Would you like to experiment with this idea here? I've been thinking about it for quite a while in the context of projects with C-extensions that would be noticeably slower. |
|
Is macos 15 slower or faster, by the way? |
|
It's hard to be confident, but so far it looks like ARM macos 26 The main issue with CI time is that with a large number of macOS runners, we get a lot of queuing in GitHub. None of the jobs in our matrix are particularly slow -- I'm actually very happy with how fast they all are. I'm game to experiment with new CI strategies here, but I don't want to complicate this "simple" PR with those changes. ... I think I'd like to merge this as-is, accepting that it raises the overall CI time to ~20 minutes. With the benefit of more time to think about it, that seems fine to me, since CI time isn't a major bottleneck for us. And i think we both have ideas about CI refinements that will make this better over time. I'm marking this PR "ready" without further changes. Yes, things will get a little slower for a while. But we'll make them faster over time. |
| - Windows | ||
| - macOS | ||
| - name: Ubuntu | ||
| runs_on: ubuntu-latest |
There was a problem hiding this comment.
We should be using kebab-case keys for GHA to follow the convention consistently.
Also, complex non-primitive values in the matrix tend to be difficult to work with (and didn't use to be supported officially in the past). I wouldn't mind rendering the VM identifiers as is, w/o any extra fancy naming.
There was a problem hiding this comment.
Yep, I can make those adjustments, along with the other edits.
I also don't mind rendering the identifiers verbatim in our job names -- the previous config seemed carefully arranged to produce short names (but in, IMO, a fragile way).
There was a problem hiding this comment.
Yeah, I've found that we can get short enough names by spreading the parts across the reusable job name and the matrix name in the calling workflow, while still keeping the original identifiers most of the time.
| CI-GHA, | ||
| OS-${{ runner.os }}, | ||
| VM-${{ matrix.os }}, | ||
| VM-${{ matrix.os.name }}, |
There was a problem hiding this comment.
Let's just turn this (and the matrix) into a simple
| VM-${{ matrix.os.name }}, | |
| VM-${{ matrix.runner-vm }}, |
so it matches what I have in other repos as it's easier to sync.
| strategy: | ||
| fail-fast: false | ||
| matrix: | ||
| os: |
There was a problem hiding this comment.
| os: | |
| runner-vm: |
| - name: Ubuntu | ||
| runs_on: ubuntu-latest | ||
| - name: Windows | ||
| runs_on: windows-latest | ||
| - name: macOS | ||
| runs_on: macos-26 | ||
| - name: "macOS (Intel)" | ||
| runs_on: macos-26-intel |
There was a problem hiding this comment.
| - name: Ubuntu | |
| runs_on: ubuntu-latest | |
| - name: Windows | |
| runs_on: windows-latest | |
| - name: macOS | |
| runs_on: macos-26 | |
| - name: "macOS (Intel)" | |
| runs_on: macos-26-intel | |
| - ubuntu-latest | |
| - windows-latest | |
| - macos-26 | |
| - macos-26-intel |
There was a problem hiding this comment.
(though, I'd prefer getting rid of latest for good in favor of more predicatability and tracability when looking at old CI runs)
There was a problem hiding this comment.
100%!
I plan to come back with a dedicated PR to simply swap latest for the appropriate version numbers. π
In the matrix config, separate the notions of "os.name" (the nice name shown in the CI GUI) from "os.runs_on" (the value used for selecting the runner). They are still aligned after this change, but can now be manipulated independently.
Pin the macOS CI runners specifically to `macos-26` and `macos-26-intel`.
ca1b28e to
9921816
Compare
There was a problem hiding this comment.
@sirosen would you mind moving from CLI args to a config like CPython does? Here's an example: https://github.com/python/cpython/blob/362145c20ebb08d2f850a49d356ecee858a281ae/.github/actionlint.yaml.
Using the lowest-level tool config allows people who happen to bypass our workflow tool layers (tox + pre-commit framework) to still benefit from some configuration that'd be shared with direct invocations.
There was a problem hiding this comment.
I agree fully with the rationale. I often reach past layers (especially tox) when necessary, and it's nice for things to work smoothly.
However! We got preempted on this one. actionlint just released a new version with support for macos-26-intel, so we can just update to use that.
There was a problem hiding this comment.
Ah, okay. I guess that's fine too. Though, next time we need it, I'd like to leave the config in the repo (even when it's empty).
There was a problem hiding this comment.
Does this identifier intentionally avoid linking the PR or commits?
Side note: it just occurred to me that with with #2353 we'll finally have a good case to use that highlights category soon.
There was a problem hiding this comment.
Oh, no, not on purpose! I noted at the top that I needed to link to issues. I apparently never fixed it. I've amended to link things all together.
π I think highlights will be the right way to announce when we move!
- Use a flat array, rather than objects in an array - Name the key `runner-vm` to match other projects - Allow the verbatim `runs-on` identifier to appear in job names - Update actionlint version os that it will recognize the runner name Co-authored-by: πΊπ¦ Sviatoslav Sydorenko (Π‘Π²ΡΡΠΎΡΠ»Π°Π² Π‘ΠΈΠ΄ΠΎΡΠ΅Π½ΠΊΠΎ) <578543+webknjaz@users.noreply.github.com>
9921816 to
b1f9a0d
Compare
Pull request was closed
|
Tip π€ GitHub only updates ephemeral merge commits like I closed and re-opened this PR just to trigger GitHub's internal machinery that updates the ephemeral merge commit so that the CI would pick up the changes from #2364. |
|
New macOS seems to be getting stuck and hitting timeouts. I dropped this from the queue and will re-add it but it looks like this is going to cause problems long term. |
|
We could hold this if it seems risky. There's a lot of unrelated cleanup we can consider which might help (dropping pypy, py3.9), and we can discuss increasing the timeouts a bit if we're running too close to those numbers. |
|
Looks like it worked in the end. Maybe, GHA was overloaded at the time. Let's observe and see how it goes in the long run. |
Contributor checklist
Included tests for the changes.changelog.d/(seechangelog.d/README.mdfor instructions) or the PR text says "no changelog needed".
Maintainer checklist
bot:chronographer:skiplabel.(following Semantic Versioning).