Skip to content

Rework test overrides repo - #5514

Merged
lihaoyi merged 18 commits into
com-lihaoyi:mainfrom
alexarchambault:rework-test-overrides-repo
Jul 21, 2025
Merged

lihaoyi merged 18 commits into
com-lihaoyi:mainfrom
alexarchambault:rework-test-overrides-repo

Conversation

@alexarchambault

@alexarchambault alexarchambault commented Jul 11, 2025 •

Copy link
Copy Markdown
Collaborator

"Test overrides" stuff is being used to make integration tests or examples use modules and artifacts from the build rather than downloading those from Maven Central. Until now, the way it was injecting modules from the build was somehow low-level: the injected modules don't have dependencies anymore, and their full class path as provided by the build is put in the resolution artifacts as is.

The PR here makes it more principled: the build now provides a local Maven repository (out/dist/localRepo.dest, resulting in things like out/dist/localRepo.dest/com/lihaoyi/mill-libs-javalib_3/SNAPSHOT/mill-libs-javalib_3-SNAPSHOT.{pom,jar}), and the path to that local repository is passed around via an environment variable. Then, that local repository is added to the repository list, where coursier can pick things from it, and we don't blindly drop all dependencies and inject class paths as is any more.

This makes the resolutions happening in examples and ITs more similar to how it happens when using a Mill release.

I've been needing while working on #5398, although I don't recall exactly which part of the fix requires it. But this makes things more principled, and less hackish IMO (no more transitiveLocalTestOverrides thing to maintain), so I think it's a good thing in itself.

playMinorVersion() match {
case "2.6" | "2.7" | "2.8" =>
Seq(mvn"com.typesafe.play::routes-compiler:${playVersion()}")
case "2.6" =>

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

We make the Scala version suffixes explicit here, in accordance with what's done in playRouteCompilerWorkerClasspath a bit below. Without this, playRouteCompilerWorkerClasspath and routerClasspath, which are merged in playRouterToolsClasspath, might bring artifacts for different Scala versions. I don't recall if this breaks the CI here or only with some more work for #5398, but this needs fixing anyway.

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This makes the resolution of com.lihaoyi:mill-contrib-playlib-worker-*, which now happens with the new local repo in examples and integration tests, more consistent

@alexarchambault
alexarchambault marked this pull request as ready for review July 11, 2025 13:47
@alexarchambault
alexarchambault requested a review from lihaoyi July 11, 2025 13:49
@lihaoyi

lihaoyi commented Jul 11, 2025

Copy link
Copy Markdown
Member

One concern about using a single localRepo for everything is that it results in every test depending on every non-test module, which would result in poor selective testing precision since any code change will trigger every test to run. Part of the goal of the original transitiveTestOverrides infrastructure was to try and keep the dependencies narrow such that tests only depended on modules upstream of their enclosing non-test module

Is there some way to preserve some of the precision that transitiveTestOverrides gave us? Maybe having a separate localRepo for each module that aggregates only the transitive upstream module's classpaths, with a "global" localRepo used for integration/example tests? That would mirror the prior setup, just migrated to a localRepo folder rather than MILL_LOCAL_TEST_OVERRIDES_* env var

@alexarchambault

Copy link
Copy Markdown
Collaborator Author

Is there some way to preserve some of the precision that transitiveTestOverrides gave us? Maybe having a separate localRepo for each module that aggregates only the transitive upstream module's classpaths, with a "global" localRepo used for integration/example tests? That would mirror the prior setup, just migrated to a localRepo folder rather than MILL_LOCAL_TEST_OVERRIDES_* env var

We probably can do something like that, yes. Looking into it.

def localTestExtraModules: Seq[MillJavaModule] = Nil
def localTestRepositories: T[Seq[PathRef]] = {

def recursive[T](start: T, deps: T => Seq[T]): Seq[T] = {

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I find this code surprisingly confusing. What is going on here? Is it just a depth-first or breadth-first search? If so, could we implement it with a more traditional while-loop and mutable queue/stack?

m => {
val localTestExtraModules0 = m match {
case m0: MillJavaModule => m0.localTestExtraModules
case _ => Nil

@lihaoyi lihaoyi Jul 16, 2025 •

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Given the type signatures involved, it seems like this case will never occur? And the .collect below can just be a .map?

* artifact resolution, or be merged together and then used as repository,
* for example.
*/
def stagePublish: Task[PathRef] = Task {

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Let's put the publish prefix first, to namespace the task a bit better. publishLocal is already taken, maybe publishStage? publishTestFolder?

@lihaoyi
lihaoyi force-pushed the rework-test-overrides-repo branch from f472544 to 3226805 Compare July 21, 2025 03:42
@lihaoyi
lihaoyi merged commit 2fc6433 into com-lihaoyi:main Jul 21, 2025
@lefou lefou added this to the 1.0.1 milestone Aug 1, 2025
lihaoyi pushed a commit that referenced this pull request Feb 13, 2026
…6819)

This gets rid of the need of `MILL_LOCAL_TEST_REPO` and
`MILL_USER_TEST_REPO` from
#5514, and manages those repos
solely via the `COURSIER_REPOSITORIES` environment variable. That way,
the repositories from `mill-repositories` of
#6713 automatically get
precedence over the former repos.

This addresses issues I've been seeing locally, where some artifacts
under `~/.ivy2/local` were being used by the JvmWorker, while they
shouldn't.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants