Skip to content

Maven 3.9.10 duplicates projects in parallel build #2487

Closed

Description

@alzimmermsft

Affected version

3.9.10

Bug description

We have a build attempting to parallelize work, using -T 1C, while only building part of a larger set of projects, using --pl, which is now hitting an odd bug where the same project is built multiple times.

The following command is being ran (seen both on Linux in CI and on Window on my local machine):

mvn clean verify -f pom.xml --fail-at-end "-Dmaven.javadoc.skip=true" "-Dcodesnippet.skip=true" "-Dspotbugs.skip=true" "-Dcheckstyle.skip=true" "-Drevapi.skip=true" -DtrimStackTrace=false "-Dspotless.apply.skip=true" "-Dspotless.check.skip=true" -pl "com.azure:azure-core-http-vertx,com.azure:azure-core-tracing-opentelemetry-samples,com.azure:azure-core-perf,com.azure:perf-test-core,com.azure:azure-core" -DAZURE_TEST_DEBUG=false -Dparallel-test-playback=true -DcreateSourcesJar=false -T 1C

On the following POM file:

https://github.com/Azure/azure-sdk-for-java/blob/main/pom.xml

At some point in the build I see the following output:

Image

Where azure-core-perf is being built three times, and seeing 6/5 projects being built, which eventually results in the build failing.

Running the build without parallelization works, as well as reverting back to Mavan 3.9.6 (the other version I have on my local machine).

Additionally, what is odd, adding more projects to the project list doesn't run into this problem either:

mvn clean verify -f pom.xml --fail-at-end "-Dmaven.javadoc.skip=true" "-Dcodesnippet.skip=true" "-Dspotbugs.skip=true" "-Dcheckstyle.skip=true" "-Drevapi.skip=true" -DtrimStackTrace=false "-Dspotless.apply.skip=true" "-Dspotless.check.skip=true" -pl "com.azure:azure-core,com.azure:azure-core-amqp,com.azure:azure-core-amqp-experimental,com.azure:azure-core-experimental,com.azure:azure-core-http-jdk-httpclient,com.azure:azure-core-http-netty,com.azure:azure-core-http-okhttp,com.azure:azure-core-http-vertx,com.azure:azure-core-management,com.azure:azure-core-metrics-opentelemetry,com.azure:azure-core-serializer-avro-apache,com.azure:azure-core-serializer-json-gson,com.azure:azure-core-serializer-json-jackson,com.azure:azure-core-test,com.azure:azure-core-tracing-opentelemetry,com.azure:azure-core-tracing-opentelemetry-samples,com.azure:azure-core-perf,com.azure:perf-test-core" -DAZURE_TEST_DEBUG=false -Dparallel-test-playback=true -DcreateSourcesJar=false -T 1C

Activity

  1. cstamas commented on Jun 16, 2025

    @cstamas
    Member

    Seems related to this comment (otherwise not directly to the issue in which comment was made):
    #2455 (comment)

    Can you by any chance try with locally built 3.9.x Maven?

  2. alzimmermsft commented on Jun 16, 2025

    @alzimmermsft
    ContributorAuthor

    I cloned the repo, switched to branch maven-3.9.x, and ran this command (mvn -DdistributionTargetDir="$HOME/app/maven/apache-maven-3.9.x-SNAPSHOT" clean package) mentioned in the README to build using the current source code (https://github.com/apache/maven/tree/maven-3.9.x?tab=readme-ov-file#quick-build).

    Image

    Unfortunately, using that build I'm still seeing this issue

    Image

    Unless I was supposed to check out #2467

  3. cstamas commented on Jun 16, 2025

    @cstamas
    Member

    Nay, I wanted you to test 69267a1
    (that is already merged to maven-3.9.x, so all fine)

  4. alzimmermsft commented on Jun 16, 2025

    @alzimmermsft
    ContributorAuthor

    Is there is anything else I can test I'll be happy to do so!

    I'll also add some additional details around the projects being built.

    com.azure:perf-test-core has direct dependencies on released versions of com.azure:azure and com.azure:azure-core-http-vertx.

    com.azure:azure-core-perf has direct source dependencies on com.azure:azure-core and com.azure:perf-test-core.

    com.azure:azure-core-http-vertx has a direct source dependency on com.azure:azure-core

    com.azure:azure-core-tracing-opentelemetry-samples has a direct source dependency on com.azure:azure-core

    com.azure:azure-core has no dependencies on anything above.

  5. TomaszGaweda commented on Jun 16, 2025

    @TomaszGaweda

    FYI we in Hazelcast also encountered this issue.
    I'll try to test the possible fix tomorrow (already pretty late for me 😅 )

    But as @k-jamroz found, https://github.com/apache/maven/pull/1724/files#diff-66b7fb702fdfdb6b018fdee43a543f5e3b1d38f598858f4952e7de3261a89047R104 may cause duplicates - we do not de-duplicate while adding projects.

  6. alzimmermsft commented on Jun 17, 2025

    @alzimmermsft
    ContributorAuthor

    @TomaszGaweda I think that file may be the issue as well. Debugging the project list I shared in my initial issue description, I see this at one point when ConcurrencyDependencyGraph (calling getSchedulableNewProcesses) is finding additional project to build as com.azure:azure-core completes:

    Image

    @cstamas could this be the bug causing duplicate project builds?

  7. alzimmermsft commented on Jun 17, 2025

    @alzimmermsft
    ContributorAuthor

    Not the best code, but making this change to FilteredProjectDependencyGraph resolved my issue

    private List<MavenProject> applyFilter(
                Collection<? extends MavenProject> projects, boolean transitive, boolean upstream) {
            List<MavenProject> filtered = new ArrayList<>(projects.size());
            for (MavenProject project : projects) {
                if (whiteList.containsKey(project)) {
                    filtered.add(project);
                } else if (!transitive) {
                    filtered.addAll(upstream ? getUpstreamProjects(project, false) : getDownstreamProjects(project, false));
                }
            }
            return filtered.stream().distinct().collect(Collectors.toList());
        }
  8. alzimmermsft commented on Jun 17, 2025

    @alzimmermsft
    ContributorAuthor

    I've opened the PR #2489 which provides one possible solution for this issue.

    I've tested it locally and it fixes the problem.

  9. cstamas commented on Jun 18, 2025

    @cstamas
    Member

    Thank you all for contribution!

  10. added 2 commits that reference this issue on Jun 19, 2025
    4e02a89
    8925fbd
  11. alzimmermsft commented on Jun 19, 2025

    @alzimmermsft
    ContributorAuthor

    @cstamas is it possible to get a ping on the issue once the next version of 3.9.x ships? We'll want to upgrade to it eagerly given this is causing us build failures and we heavily leverage parallelization to speed up our CI.

  12. cstamas commented on Jun 19, 2025

    @cstamas
    Member

    We usually announce releases on MLs... https://maven.apache.org/mailing-lists.html

    Sadly, cannot guarantee it.

    @slawekjaranowski what is our plan with 3.9.11?

  13. slawekjaranowski commented on Jun 20, 2025

    @slawekjaranowski
    Member

    We usually announce releases on MLs... https://maven.apache.org/mailing-lists.html

    Sadly, cannot guarantee it.

    @slawekjaranowski what is our plan with 3.9.11?

    If no more bug will be reported I can prepare next version at end of week. So will be published at first week of July

  14. JackPGreen commented on Jul 16, 2025

    @JackPGreen

    @cstamas is it possible to get a ping on the issue once the next version of 3.9.x ships?

    @alzimmermsft ping!

  15. TomaszGaweda commented on Jul 16, 2025

    @TomaszGaweda

    I can confirm that 3.9.11 with this fix fixes our (Hazelcast) issues on CI. Thank you @alzimmermsft for the fix and @slawekjaranowski for quick release!

  16. alzimmermsft commented on Jul 16, 2025

    @alzimmermsft
    ContributorAuthor

    Thank you so much for getting this released!

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    bugSomething isn't working

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions