Skip to content

[improve][build] Adopt Gradle best practices: convention plugins, build-logic composite build, and unconditional module inclusion - #25435

Merged
lhotari merged 39 commits into
apache:masterfrom
lhotari:improve/gradle-best-practices
Mar 31, 2026
Merged

lhotari merged 39 commits into
apache:masterfrom
lhotari:improve/gradle-best-practices

Conversation

@lhotari

@lhotari lhotari commented Mar 29, 2026 •

Copy link
Copy Markdown
Member

Motivation

Align the Gradle build with Gradle's documented best practices to improve build performance, maintainability, and configuration cache effectiveness. The main issues addressed:

  1. Monolithic subprojects{}/allprojects{} blocks β€” 260+ lines of shared config injected non-obviously, preventing Gradle's Isolated Projects optimization
  2. Build logic in buildSrc β€” Gradle recommends composite builds over buildSrc because buildSrc changes invalidate the entire build, causing all tasks to lose build cache entries (gradle/gradle#6045). Migrated to build-logic/ composite build.
  3. Conditional module inclusion in settings.gradle.kts β€” dynamic project structure breaks configuration cache and IDE integration
  4. Eager task configuration β€” tasks.withType<> without .configureEach causes unnecessary configuration overhead
  5. Configuration cache invalidation β€” generatePulsarVersion task's gitDirty/gitCommitId inputs caused cascading rebuilds across the entire project on every file change

Modifications

Convention plugins (build-logic composite build replacing buildSrc):

  • Migrated buildSrc to build-logic/ composite build per Gradle best practice, so build logic changes only invalidate projects that depend on the changed code
  • Created pulsar.java-conventions, pulsar.code-quality-conventions, pulsar.test-certs-conventions, pulsar.nar-conventions convention plugins
  • Moved existing pulsar.shadow-conventions and pulsar.client-shade-conventions from buildSrc to build-logic
  • Each subproject now explicitly opts in via plugins { id("pulsar.java-conventions") } instead of inheriting from subprojects{}
  • Eliminated afterEvaluate + reflection for license plugin configuration
  • Deleted buildSrc/ and gradle/code-quality.gradle.kts

Unconditional module inclusion:

  • Removed all 4 conditional blocks from settings.gradle.kts (-PcoreModules, -Pdocker, -PintegrationTests, shade-test detection)
  • All modules are always included; Docker builds only run when explicitly requested
  • Removed assembleβ†’dockerBuild dependency from Docker modules
  • Made pulsar-bom project references unconditional

Configuration cache & build performance:

  • All tasks.withType<> changed to tasks.withType<>().configureEach for lazy task configuration
  • Removed gitDirty and gitCommitId from generatePulsarVersion task inputs to prevent cascading rebuilds
  • Migrated pulsar-functions-runtime-all fat JAR from manual dependsOn(runtimeClasspath) + zipTree() to Shadow plugin for configuration-cache-compatible lazy resolution
  • Updated distribution/server/build.gradle.kts to reference the new shadowJar output from pulsar-functions-runtime-all instead of the plain jar task
  • Used shadow { addShadowVariantIntoJavaComponent.set(false) } in shadow conventions
  • Disabled microbench shadowJar from assemble lifecycle via addShadowJarToAssembleLifecycle

Dependency updates:

  • Updated Shadow plugin from 9.3.2 to 9.4.1
  • Added version-catalog-update plugin (1.1.0) and versions plugin by ben-manes (0.53.0) for dependency management

CI & documentation:

  • Removed -PcoreModules and -Pdocker flags from GitHub Actions workflows (codeql.yaml, pulsar-ci.yaml) and docker/build.sh
  • Updated README to remove documentation of the -PcoreModules minimal build option

Verifying this change

  • Make sure that the change passes the CI checks.

This change is already covered by existing tests. Build verification:

  • ./gradlew help β€” configuration phase succeeds
  • ./gradlew assemble β€” full build produces identical artifacts
  • ./gradlew assemble (second run) β€” "Reusing configuration cache", all tasks UP-TO-DATE
  • ./gradlew clean && ./gradlew assemble β€” all compileJava/shadowJar/nar tasks served FROM-CACHE
  • ./gradlew :pulsar-broker:test --dry-run β€” test task graph unchanged
  • ./gradlew :pulsar-io:pulsar-io-data-generator:nar --dry-run β€” NAR packaging works

Does this pull request potentially affect one of the following parts:

  • Dependencies (add or upgrade a dependency)
  • The public API
  • The schema
  • The default values of configurations
  • The threading model
  • The binary protocol
  • The REST endpoints
  • The admin CLI options
  • The metrics
  • Anything that affects deployment

Documentation

  • doc-not-needed

@github-actions github-actions Bot added the doc-not-needed Your PR changes do not impact docs label Mar 29, 2026
…ld-logic composite build, and unconditional module inclusion

Replace buildSrc with a build-logic composite build to avoid full build
invalidation on build logic changes (gradle/gradle#6045). Extract the
monolithic subprojects{}/allprojects{} blocks into convention plugins
(pulsar.java-conventions, pulsar.code-quality-conventions,
pulsar.test-certs-conventions, pulsar.nar-conventions) so each subproject
explicitly opts in via its plugins{} block. Remove all conditional module
inclusion from settings.gradle.kts (-PcoreModules, -Pdocker,
-PintegrationTests, shade-test detection) so the project structure is
deterministic and configuration cache friendly. Use
tasks.withType<>().configureEach instead of eager withType<> blocks.

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
@lhotari
lhotari force-pushed the improve/gradle-best-practices branch from 4562fdd to 7a17a67 Compare March 29, 2026 21:20
@lhotari
lhotari requested a review from merlimat March 29, 2026 21:24
lhotari added 24 commits March 30, 2026 09:20
…nt in different classloader"

This reverts commit 9ca3eae.
@lhotari
lhotari merged commit d1eb652 into apache:master Mar 31, 2026
81 of 84 checks passed
@lhotari lhotari added this to the 5.0.0-M1 milestone Jun 12, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

doc-not-needed Your PR changes do not impact docs ready-to-test

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants