Skip to content

Try to improve precision of codesig callgraph analysis - #6880

Merged
lihaoyi merged 55 commits into
com-lihaoyi:mainfrom
lihaoyi:codesig-precision
Mar 8, 2026
Merged

lihaoyi merged 55 commits into
com-lihaoyi:mainfrom
lihaoyi:codesig-precision

Conversation

@lihaoyi

@lihaoyi lihaoyi commented Feb 24, 2026 •

Copy link
Copy Markdown
Member

Attempts to resolve #6871.

⏺ Summary of Key Algorithm and Data Structure Changes

  1. Per-call-site precise type analysis (LocalSummary.scala)

New dependency: asm-analysis (ASM's dataflow analysis framework).

New algorithm: analyzeMethodArgTypes runs ASM's Analyzer with a custom BasicInterpreter that preserves precise object
types on the operand stack (the default collapses all references to one REFERENCE_VALUE). For each method invoke
instruction, it records:

  • callSiteReceiverType: Map[MethodCall, JCls] β€” the precise type of this on the stack at virtual/special call sites
  • callSiteArgTypes: Map[MethodCall, Set[JCls]] β€” precise types of reference-typed arguments on the stack

These are stored in MethodInfo and flow into ReachabilityAnalysis for narrowing.

Optimization: Methods without any MethodInsnNode skip the expensive Analyzer entirely. The ClassNode tree is parsed once
and shared between MyClassVisitor (call graph extraction) and Analyzer (stack type analysis).

  1. Compound ExternalClsCall(dest, narrow) nodes (ReachabilityAnalysis.scala)

Old structure: One ExternalClsCall(cls) node per loaded external class. Call nodes connected to ExternalClsCall nodes.
ExternalClsCall(X) had edges to ExternalClsCall(parent(X)), forming an external class hierarchy tree. Callback methods
were resolved per-level, with ancestor methods reached transitively through parent edges.

New structure: ExternalClsCall(dest, narrow) is a compound key. dest determines which callback method signatures are
collected (by walking ancestors of dest in externalClassLocalDests). narrow determines which local subtypes receive
those callbacks (subtypes of narrow via typeToLocalSubtypes). No ExternalClsCall β†’ ExternalClsCall edges exist β€” the
hierarchy walk is done eagerly inside each node's resolution.

Key algorithm β€” narrowForDest: For each externalDest at a call site, collects all precise types (receiver + args) and
filters to those that are subtypes of dest (via getTransitiveAncestors(pt).contains(dest)). Subtypes that match become
the narrow values; if none match, falls back to dest itself (no narrowing, same as old behavior). This naturally handles
the fact that externalDests conflates receiver and arg types β€” the subtype check separates them since a receiver type
won't be a subtype of an unrelated arg-derived dest.

Node creation is demand-driven: Only (dest, narrow) pairs actually referenced by call sites are created, rather than one
node per loaded external class. This typically produces fewer graph nodes.

Edge movement: External callback edges moved from Call β†’ ExternalClsCall to LocalDef β†’ ExternalClsCall. This is because
the precise types needed for narrowing live on the enclosing MethodInfo, not on the Call node.

  1. Precomputed lookup tables (ReachabilityAnalysis.scala)
  • callbackMethodsPerDest: Map[Cls, Set[MethodSig]] β€” Callback methods depend only on dest, not narrow. Precomputed once
    per unique dest, shared across all ExternalClsCall nodes with the same dest.
  • typeToLocalSubtypes: HashMap[Cls, Set[Cls]] β€” Reverse ancestor index mapping each type to its local subtypes. Built by
    iterating all local classes and indexing them under each of their ancestors. Enables O(1) lookup during ExternalClsCall
    resolution.
  • ancestorCache: HashMap[Cls, Set[Cls]] β€” Memoized getTransitiveAncestors BFS results, shared between pair computation,
    callbackMethodsPerDest, and typeToLocalSubtypes.
  1. transitiveCallGraphValues dedup fix (ReachabilityAnalysis.scala)

Old: If multiple nodes in an SCC had edges to the same upstream SCC group, that group's hash was reduced in multiple
times. With reduce = _ + _, transitive hashes depended on edge count, not just reachability.

New: A BitSet tracks seen upstream groups, ensuring each contributes exactly once. This makes transitive hashes depend
only on reachability, reducing spurious invalidations when edges change between already-connected nodes.

Also replaced nodeGroups: Map with nodeGroupsArray: Array[Int] for O(1) group lookup.

  1. Iterative Tarjans (Tarjans.scala)

Converted recursive DFS to iterative using an explicit call stack (three parallel flat arrays: stackNode, stackChildIdx,
stackIsRoot). Prevents StackOverflowError on deep graphs. Same algorithm, same results.

  1. Reverse graph construction (spanningInvalidationTree)

Replaced groupMap (allocates millions of tuples) with a two-pass approach: first pass counts in-degrees to pre-allocate
arrays, second pass fills them. Same result, fewer allocations.

  1. Parallel LocalSummary processing (LocalSummary.scala, JvmModel.scala)

SymbolTable.Table changed from LinkedHashMap to ConcurrentHashMap with computeIfAbsent. LocalSummary.apply accepts an
optional TaskCtx and processes class files in parallel batches using fork.async.

  1. Lazy val hash stability (LocalSummary.scala)

More robust handling of Scala 3's VarHandle-based lazy val initialization sequences in , using regex patterns
($lzy\d+$, $lzy\d+\$lzyHandle$) and a state machine (lazyNameSeenInClinit) to skip the entire
lookup()/findVarHandle/PUTSTATIC sequence.

Test changes

  • Test 15 (15-static-method): Removed false positive Hello#() β†’ Foo#toString. The old code conservatively fanned
    out from ExternalClsCall(Object) to all local subtypes of Object. The new compound key ExternalClsCall(Object, Hello)
    correctly narrows to Hello's subtypes, and Hello doesn't override toString.
  • Tests 19, 20: New regression tests for precise narrowing of external constructor receivers and static method argument
    types.

@lihaoyi
lihaoyi force-pushed the codesig-precision branch from 14af494 to 947b89d Compare March 5, 2026 10:44
@lihaoyi
lihaoyi force-pushed the codesig-precision branch from 75c0874 to bee9503 Compare March 5, 2026 13:52
@lihaoyi

lihaoyi commented Mar 6, 2026 •

Copy link
Copy Markdown
Member Author

This seems to increase the runtime of def codeSignatures from ~1.3s to ~7s on the Mill repo. So although it would be nice to have the precision improvements, it's probably not worth the performance hit unless we can speed it up further

@lihaoyi

lihaoyi commented Mar 6, 2026

Copy link
Copy Markdown
Member Author

Overhead seems down to 1-2s, may be ok to merge now

@lihaoyi
lihaoyi force-pushed the codesig-precision branch from 0b28870 to f62f9ff Compare March 7, 2026 03:03
@lihaoyi
lihaoyi force-pushed the codesig-precision branch from f57bc92 to 60e0bbe Compare March 7, 2026 03:29
@lihaoyi
lihaoyi force-pushed the codesig-precision branch from 686e29c to ec73fc7 Compare March 7, 2026 15:12
@lihaoyi
lihaoyi merged commit fe15bb5 into com-lihaoyi:main Mar 8, 2026
67 of 69 checks passed
@lefou lefou added this to the 1.1.3 milestone Mar 8, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Selective Execution Invalidates resolvePublishDependency Task When Adding Modules To A Build

2 participants