Skip to content

Commit 1b2380c

Browse files
authored
fix(cache): always leave a manifest to the fs module cache (#11858)
1 parent c015ad5 commit 1b2380c

12 files changed

Lines changed: 394 additions & 56 deletions

File tree

‎.gitignore‎

Lines changed: 0 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -32,5 +32,4 @@ test/browser/fixtures/browser-multiple/basic-*
3232
*.tsbuildinfo
3333
.vitest
3434
explainFiles.txt
35-
.vitest-dump
3635
flakiness-report

‎docs/config/fsmodulecache.md‎

Lines changed: 26 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -27,6 +27,32 @@ DEBUG=vitest:cache:fs vitest --fsModuleCache
2727
The location of the cache is a single, workspace-wide directory. See [`fsModuleCachePath`](/config/fsmodulecachepath) to move it.
2828
:::
2929

30+
## Manifest {#manifest}
31+
32+
After every test run, each project writes a `_manifest-<project>-<hash>.json` file into the cache directory, next to the cached modules. `<project>` is the [project name](/config/name), or the name of the project's root folder if the project has no name. The manifest is grouped by Vite environment. `modules` maps the URL of every cached module to the name of its cache file in that directory. `externalized` maps the URL of every [externalized](/config/server#server-deps-external) dependency to the file that Node imports without a transform:
33+
34+
```json [node_modules/.vitest-cache/_manifest-app-1a2b3c4d.json]
35+
{
36+
"name": "",
37+
"root": "/projects/app",
38+
"modules": {
39+
"ssr": {
40+
"/src/math.test.ts": "3b1af86df9d029d4281e7ae6ef10059153a80e14",
41+
"/src/math.ts": "e8e20f586e230bde9b163dfc191f4440b241272a"
42+
}
43+
},
44+
"externalized": {
45+
"ssr": {
46+
"/node_modules/lodash-es/lodash.js": "/projects/app/node_modules/lodash-es/lodash.js"
47+
}
48+
}
49+
}
50+
```
51+
52+
The manifest keeps the entries from previous runs, so a run that imports only some files does not remove the others.
53+
54+
A cache file contains the code that Vitest runs, so you can edit it to debug a transform. Vitest runs the edited code until the module is transformed again: when the source file or the configuration changes, or when the cache is cleared with [`--clearCache`](/guide/cli#clearcache). Keep the `//# vitestCache=` comment at the end of the file, otherwise Vitest transforms the module again.
55+
3056
## Known Issues
3157

3258
Vitest creates a persistent file hash based on file content, its id, Vite's environment configuration and coverage status. Vitest tries to use as much information as it has about the configuration, but it is still incomplete. At the moment, it is not possible to track your plugin options because there is no standard interface for it.

‎docs/config/server.md‎

Lines changed: 5 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -7,7 +7,7 @@ outline: deep
77

88
Before Vitest 4, this option was used to define the configuration for the `vite-node` server.
99

10-
At the moment, this option allows you to configure the inlining and externalization mechanisms, along with the module runner debugging configuration.
10+
At the moment, this option allows you to configure the inlining and externalization mechanisms.
1111

1212
::: warning
1313
These options should be used only as the last resort to improve performance by externalizing auto-inlined dependencies or to fix issues by inlining invalid external dependencies.
@@ -71,3 +71,7 @@ If a `RegExp` is provided, it is matched against the full file path.
7171
When enabled, Vitest will try to guess a CommonJS build for an ESM entry by checking a few common CJS/UMD file name and folder patterns (like `.mjs`, `.umd.js`, `.cjs.js`, `umd/`, `cjs/`, `lib/`).
7272

7373
This is a best-effort heuristic to work around confusing or incorrect ESM/CJS packaging and may not work for all dependencies.
74+
75+
## server.debug <Deprecated />
76+
77+
This option has no effect. To inspect or edit the code that Vitest runs, look up the module in the [`fsModuleCache` manifest](/config/fsmodulecache#manifest).

‎packages/vitest/src/node/cache/fsModuleCache.ts‎

Lines changed: 160 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -6,7 +6,7 @@ import type { ResolvedConfig } from '../types/config'
66
import fs, { existsSync, mkdirSync, readFileSync } from 'node:fs'
77
import { readFile, rename, rm, stat, unlink, writeFile } from 'node:fs/promises'
88
import { parse, stringify } from 'flatted'
9-
import { dirname, join } from 'pathe'
9+
import { basename, dirname, join } from 'pathe'
1010
import c from 'tinyrainbow'
1111
import { searchForWorkspaceRoot } from 'vite'
1212
import { createDebugger } from '../../utils/debugger'
@@ -22,6 +22,35 @@ const cacheCommentLength = cacheComment.length
2222

2323
const METADATA_FILE = '_metadata.json'
2424

25+
interface CacheManifest extends ManifestContent {
26+
name: string
27+
root: string
28+
}
29+
30+
interface ManifestContent {
31+
// environment name -> module url -> cache file name
32+
modules: Record<string, Record<string, string>>
33+
// environment name -> module url -> the file that Node imports
34+
externalized: Record<string, Record<string, string>>
35+
}
36+
37+
type ManifestSection = keyof ManifestContent
38+
39+
const MANIFEST_SECTIONS: ManifestSection[] = ['modules', 'externalized']
40+
41+
interface ManifestState {
42+
path: string
43+
name: string
44+
root: string
45+
// the content on disk, loaded by the first write in this process
46+
content?: ManifestContent
47+
// environment name -> entries recorded since the last write
48+
environments: Map<string, ManifestEntries>
49+
}
50+
51+
// module url -> cache path for `modules`, or the externalized file
52+
type ManifestEntries = Record<ManifestSection, Map<string, string>>
53+
2554
// Default location of the module cache. It lives inside `node_modules` at the
2655
// workspace root so it's shared by every project and is naturally invalidated
2756
// whenever dependencies are reinstalled.
@@ -56,6 +85,8 @@ export class FileSystemModuleCache {
5685
// Map<id, tmp | null>
5786
Map<string, string | null>
5887
>()
88+
private manifests = new Map<string, ManifestState>()
89+
private manifestEntries = new WeakMap<DevEnvironment, ManifestEntries>()
5990

6091
constructor(private vitest: Vitest) {
6192
this.rootCache =
@@ -363,6 +394,125 @@ export class FileSystemModuleCache {
363394
return fsResultPath
364395
}
365396

397+
recordManifestModule(
398+
vitestConfig: ResolvedConfig,
399+
environment: DevEnvironment,
400+
url: string,
401+
cachePath: string,
402+
): void {
403+
this.getManifestEntries(vitestConfig, environment)?.modules.set(url, cachePath)
404+
}
405+
406+
recordManifestExternal(
407+
vitestConfig: ResolvedConfig,
408+
environment: DevEnvironment,
409+
url: string,
410+
externalized: string,
411+
): void {
412+
this.getManifestEntries(vitestConfig, environment)?.externalized.set(url, externalized)
413+
}
414+
415+
private getManifestEntries(
416+
vitestConfig: ResolvedConfig,
417+
environment: DevEnvironment,
418+
): ManifestEntries | undefined {
419+
let entries = this.manifestEntries.get(environment)
420+
if (entries) {
421+
return entries
422+
}
423+
const cacheRoot = this.getCacheRoot(vitestConfig)
424+
if (cacheRoot == null) {
425+
return
426+
}
427+
const manifest = this.getManifest(vitestConfig, cacheRoot)
428+
entries = manifest.environments.get(environment.name)
429+
if (!entries) {
430+
entries = { modules: new Map(), externalized: new Map() }
431+
manifest.environments.set(environment.name, entries)
432+
}
433+
this.manifestEntries.set(environment, entries)
434+
return entries
435+
}
436+
437+
private getManifest(vitestConfig: ResolvedConfig, cacheRoot: string): ManifestState {
438+
// several projects and Vitest processes can share the cache directory
439+
const label = (vitestConfig.name || basename(vitestConfig.root))
440+
.replace(/[^\w@-]+/g, '_')
441+
.slice(0, 50)
442+
const key = hash('sha1', `${vitestConfig.root}\0${vitestConfig.name}`, 'hex').slice(0, 8)
443+
const path = join(cacheRoot, `_manifest-${label}-${key}.json`)
444+
let manifest = this.manifests.get(path)
445+
if (!manifest) {
446+
manifest = { path, name: vitestConfig.name, root: vitestConfig.root, environments: new Map() }
447+
this.manifests.set(path, manifest)
448+
}
449+
return manifest
450+
}
451+
452+
async writeManifests(): Promise<void> {
453+
await Promise.all(
454+
Array.from(this.manifests.values(), (manifest) => this.writeManifest(manifest)),
455+
)
456+
}
457+
458+
private async writeManifest(manifest: ManifestState): Promise<void> {
459+
const recorded = Array.from(manifest.environments.values())
460+
if (recorded.every((entries) => !entries.modules.size && !entries.externalized.size)) {
461+
return
462+
}
463+
const loaded = manifest.content
464+
const content = loaded ?? (await readManifestContent(manifest.path))
465+
manifest.content = content
466+
467+
const changes: [section: ManifestSection, environment: string, url: string, value: string][] =
468+
[]
469+
for (const [environment, entries] of manifest.environments) {
470+
for (const section of MANIFEST_SECTIONS) {
471+
const values = content[section][environment]
472+
for (const [url, recordedValue] of entries[section]) {
473+
const value =
474+
section === 'modules'
475+
? recordedValue.slice(recordedValue.lastIndexOf('/') + 1)
476+
: recordedValue
477+
if (values?.[url] !== value) {
478+
changes.push([section, environment, url, value])
479+
}
480+
}
481+
entries[section].clear()
482+
}
483+
}
484+
if (!changes.length) {
485+
return
486+
}
487+
488+
let target = content
489+
if (loaded) {
490+
// another process can update the manifest after this one loaded it
491+
target = await readManifestContent(manifest.path)
492+
for (const section of MANIFEST_SECTIONS) {
493+
for (const environment in content[section]) {
494+
target[section][environment] = {
495+
...content[section][environment],
496+
...target[section][environment],
497+
}
498+
}
499+
}
500+
}
501+
for (const [section, environment, url, value] of changes) {
502+
;(target[section][environment] ??= {})[url] = value
503+
// a module can switch between being inlined and externalized
504+
delete target[section === 'modules' ? 'externalized' : 'modules'][environment]?.[url]
505+
}
506+
manifest.content = target
507+
try {
508+
const data: CacheManifest = { name: manifest.name, root: manifest.root, ...target }
509+
await atomicWriteFile(manifest.path, JSON.stringify(data, null, 2))
510+
} catch (error) {
511+
manifest.content = undefined
512+
debugFs?.(`failed to write fs cache manifest ${manifest.path}: ${error}`)
513+
}
514+
}
515+
366516
private saveMemoryCache(environment: DevEnvironment, id: string, cache: string | null) {
367517
let environmentKeys = this.fsCacheKeys.get(environment)
368518
if (!environmentKeys) {
@@ -435,6 +585,15 @@ export class FileSystemModuleCache {
435585
}
436586
}
437587

588+
async function readManifestContent(path: string): Promise<ManifestContent> {
589+
try {
590+
const manifest = JSON.parse(await readFile(path, 'utf-8'))
591+
return { modules: manifest.modules ?? {}, externalized: manifest.externalized ?? {} }
592+
} catch {
593+
return { modules: {}, externalized: {} }
594+
}
595+
}
596+
438597
/**
439598
* Performs an atomic write operation using the write-then-rename pattern.
440599
*

‎packages/vitest/src/node/cache/index.ts‎

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -61,6 +61,7 @@ export class VitestCache {
6161
await Promise.all([
6262
this._results.update(specifications, startTime),
6363
this._dependencies.update(specifications, startTime),
64+
this._modules.writeManifests(),
6465
])
6566
}
6667

‎packages/vitest/src/node/config/resolveConfig.ts‎

Lines changed: 4 additions & 6 deletions
Original file line numberDiff line numberDiff line change
@@ -1004,12 +1004,10 @@ export function resolveTestConfig(
10041004
resolved.server ??= {}
10051005
resolved.server.deps ??= {}
10061006

1007-
if (resolved.server.debug?.dump || process.env.VITEST_DEBUG_DUMP) {
1008-
const userFolder = resolved.server.debug?.dump || process.env.VITEST_DEBUG_DUMP
1009-
resolved.dumpDir = resolve(
1010-
resolved.root,
1011-
typeof userFolder === 'string' && userFolder !== 'true' ? userFolder : '.vitest-dump',
1012-
resolved.name || 'root',
1007+
// projects inherit the root option, so they warn only when they set it themselves
1008+
if (resolved.server.debug && (!globalConfig || !globalConfig.server.debug)) {
1009+
logger.deprecate(
1010+
'`server.debug` is deprecated and has no effect. To inspect or edit the transformed code, look up the module in the `fsModuleCache` manifest: https://vitest.dev/config/fsmodulecache#manifest',
10131011
)
10141012
}
10151013

‎packages/vitest/src/node/environments/transformService.ts‎

Lines changed: 10 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -147,6 +147,14 @@ export class ModuleTransformService {
147147

148148
const externalize = await this.resolver.shouldExternalize(moduleGraphModule.id!)
149149
if (externalize) {
150+
if (this.fsCacheEnabled) {
151+
this.fsCache.recordManifestExternal(
152+
this.config,
153+
environment,
154+
moduleGraphModule.url,
155+
externalize,
156+
)
157+
}
150158
return { externalize, type: 'module' }
151159
}
152160

@@ -192,6 +200,8 @@ export class ModuleTransformService {
192200
})
193201
}
194202

203+
this.fsCache.recordManifestModule(this.config, environment, moduleGraphModule.url, cachePath)
204+
195205
if (saveCachePromises.has(cachePath)) {
196206
return saveCachePromises.get(cachePath)!.then((result) => {
197207
this.recordResult(trace, result)

‎packages/vitest/src/node/pools/rpc.ts‎

Lines changed: 0 additions & 11 deletions
Original file line numberDiff line numberDiff line change
@@ -6,7 +6,6 @@ import type { OTELCarrier } from '../../utils/traces'
66
import type { StaticMockedModules } from '../environments/staticMocks'
77
import type { TestProject } from '../project'
88
import type { ResolveSnapshotPathHandlerContext } from '../types/config'
9-
import { existsSync, mkdirSync } from 'node:fs'
109
import { fileURLToPath } from 'node:url'
1110
import { cleanUrl } from '@vitest/utils/helpers'
1211
import { isBuiltin, toBuiltin } from '../../utils/modules'
@@ -40,14 +39,8 @@ export function createMethodsRPC(
4039
const vitest = project.vitest
4140
const cacheFs = methodsOptions.cacheFs ?? false
4241
project.vitest.state.metadata[project.name] ??= {
43-
externalized: {},
4442
duration: {},
45-
tmps: {},
4643
}
47-
if (project.config.dumpDir && !existsSync(project.config.dumpDir)) {
48-
mkdirSync(project.config.dumpDir, { recursive: true })
49-
}
50-
project.vitest.state.metadata[project.name].dumpDir = project.config.dumpDir
5144

5245
function getEnvironment(environmentName: string): DevEnvironment {
5346
const environment = project.vite.environments[environmentName]
@@ -76,7 +69,6 @@ export function createMethodsRPC(
7669
.then((result) => {
7770
const metadata = state.metadata[project.name]
7871
if ('externalize' in result) {
79-
metadata.externalized[url] = result.externalize
8072
// builtins and network urls are already resolved inside the worker
8173
// without a round-trip, only module externalizations are worth sharing
8274
if (result.type === 'module' && url[0] === '/') {
@@ -88,9 +80,6 @@ export function createMethodsRPC(
8880
externals[url] = result
8981
}
9082
}
91-
if ('tmp' in result) {
92-
metadata.tmps[url] = result.tmp
93-
}
9483
if (accountModuleDuration) {
9584
const duration = performance.now() - start
9685
metadata.duration[url] ??= []

‎packages/vitest/src/node/state.ts‎

Lines changed: 0 additions & 7 deletions
Original file line numberDiff line numberDiff line change
@@ -36,14 +36,7 @@ export class StateManager {
3636
metadata: Record<
3737
string,
3838
{
39-
externalized: Record<string, string>
4039
duration: Record<string, number[]>
41-
tmps: Record<string, string>
42-
dumpDir?: string
43-
outline?: {
44-
externalized: number
45-
inlined: number
46-
}
4740
}
4841
> = {}
4942

‎packages/vitest/src/node/test-run.ts‎

Lines changed: 1 addition & 15 deletions
Original file line numberDiff line numberDiff line change
@@ -24,7 +24,7 @@ import type { TestRunEndReason } from './types/reporter'
2424
import assert from 'node:assert'
2525
import { createHash } from 'node:crypto'
2626
import { existsSync, readFileSync } from 'node:fs'
27-
import { copyFile, mkdir, writeFile } from 'node:fs/promises'
27+
import { copyFile, mkdir } from 'node:fs/promises'
2828
import { isPrimitive, sanitizeFilePath } from '@vitest/utils/helpers'
2929
import { serializeValue } from '@vitest/utils/serialize'
3030
import { parseErrorStacktrace, parseStacktrace } from '@vitest/utils/source-map'
@@ -176,20 +176,6 @@ export class TestRun {
176176
}
177177

178178
await this.vitest.report('onTestRunEnd', modules, [...errors] as SerializedError[], state)
179-
180-
for (const project in this.vitest.state.metadata) {
181-
const meta = this.vitest.state.metadata[project]
182-
if (!meta?.dumpDir) {
183-
continue
184-
}
185-
const path = resolve(meta.dumpDir, 'vitest-metadata.json')
186-
meta.outline = {
187-
externalized: Object.keys(meta.externalized).length,
188-
inlined: Object.keys(meta.tmps).length,
189-
}
190-
await writeFile(path, JSON.stringify(meta, null, 2), 'utf-8')
191-
this.vitest.logger.log(`Metadata written to ${path}`)
192-
}
193179
}
194180

195181
private hasFailed(modules: TestModule[]) {

0 commit comments

Comments
 (0)