Repository navigation
Composable turbo.json - #2706
Conversation
|
The latest updates on your projects. Learn more about Vercel for Git βοΈ
9 Ignored Deployments
|
π’ CI successful π’Thanks |
7607a0f to
533d0f0
Compare
ca5c1e3 to
9a71af5
Compare
c10c116 to
5cab00b
Compare
5cab00b to
1525aee
Compare
0333956 to
3fcffe1
Compare
47dbfaa to
79a63da
Compare
f0ac23c to
e9fd2ad
Compare
NicholasLYang
left a comment
There was a problem hiding this comment.
Just a question or two
Co-authored-by: Nicholas Yang <nicholas.yang@vercel.com>
This comment was marked as resolved.
This comment was marked as resolved.
nathanhammond
left a comment
There was a problem hiding this comment.
Integration tests could, in many cases, remove possible future failures by using a bit of jq.
(I know it's a different code path under the hood, so I don't know if that was intentional, thus a comment.)
| # 3. Change input file and assert cache miss | ||
| $ echo "more text" >> $TARGET_DIR/apps/add-keys/src/foo.txt | ||
| $ ${TURBO} run add-keys-task --filter=add-keys | ||
| \xe2\x80\xa2 Packages in scope: add-keys (esc) | ||
| \xe2\x80\xa2 Running add-keys-task in 1 packages (esc) | ||
| \xe2\x80\xa2 Remote caching disabled (esc) | ||
| add-keys:add-keys-underlying-task: cache miss, executing b7390d43fae5a180 | ||
| add-keys:add-keys-underlying-task: | ||
| add-keys:add-keys-underlying-task: > add-keys-underlying-task | ||
| add-keys:add-keys-underlying-task: > echo "running add-keys-underlying-task" | ||
| add-keys:add-keys-underlying-task: | ||
| add-keys:add-keys-underlying-task: running add-keys-underlying-task | ||
| add-keys:add-keys-task: cache miss, executing b256f31e3957fee3 | ||
| add-keys:add-keys-task: | ||
| add-keys:add-keys-task: > add-keys-task | ||
| add-keys:add-keys-task: > echo "running add-keys-task" > out/foo.min.txt | ||
| add-keys:add-keys-task: | ||
|
|
||
| Tasks: 2 successful, 2 total | ||
| Cached: 0 cached, 2 total | ||
| Time:\s*[\.0-9]+m?s (re) | ||
|
|
||
| # 4. Set env var and assert cache miss | ||
| $ SOME_VAR=somevalue ${TURBO} run add-keys-task --filter=add-keys | ||
| \xe2\x80\xa2 Packages in scope: add-keys (esc) | ||
| \xe2\x80\xa2 Running add-keys-task in 1 packages (esc) | ||
| \xe2\x80\xa2 Remote caching disabled (esc) | ||
| add-keys:add-keys-underlying-task: cache hit, replaying output b7390d43fae5a180 | ||
| add-keys:add-keys-underlying-task: | ||
| add-keys:add-keys-underlying-task: > add-keys-underlying-task | ||
| add-keys:add-keys-underlying-task: > echo "running add-keys-underlying-task" | ||
| add-keys:add-keys-underlying-task: | ||
| add-keys:add-keys-underlying-task: running add-keys-underlying-task | ||
| add-keys:add-keys-task: cache miss, executing 8e76bee49319e00e | ||
| add-keys:add-keys-task: | ||
| add-keys:add-keys-task: > add-keys-task | ||
| add-keys:add-keys-task: > echo "running add-keys-task" > out/foo.min.txt | ||
| add-keys:add-keys-task: | ||
|
|
||
| Tasks: 2 successful, 2 total | ||
| Cached: 1 cached, 2 total | ||
| Time:\s*[\.0-9]+m?s (re) | ||
|
|
There was a problem hiding this comment.
What if we used dry-run to reduce visual complexity for non-output-related tests? We're testing the hashes here, not the log messages.
| # 3. Change input file and assert cache miss | |
| $ echo "more text" >> $TARGET_DIR/apps/add-keys/src/foo.txt | |
| $ ${TURBO} run add-keys-task --filter=add-keys | |
| \xe2\x80\xa2 Packages in scope: add-keys (esc) | |
| \xe2\x80\xa2 Running add-keys-task in 1 packages (esc) | |
| \xe2\x80\xa2 Remote caching disabled (esc) | |
| add-keys:add-keys-underlying-task: cache miss, executing b7390d43fae5a180 | |
| add-keys:add-keys-underlying-task: | |
| add-keys:add-keys-underlying-task: > add-keys-underlying-task | |
| add-keys:add-keys-underlying-task: > echo "running add-keys-underlying-task" | |
| add-keys:add-keys-underlying-task: | |
| add-keys:add-keys-underlying-task: running add-keys-underlying-task | |
| add-keys:add-keys-task: cache miss, executing b256f31e3957fee3 | |
| add-keys:add-keys-task: | |
| add-keys:add-keys-task: > add-keys-task | |
| add-keys:add-keys-task: > echo "running add-keys-task" > out/foo.min.txt | |
| add-keys:add-keys-task: | |
| Tasks: 2 successful, 2 total | |
| Cached: 0 cached, 2 total | |
| Time:\s*[\.0-9]+m?s (re) | |
| # 4. Set env var and assert cache miss | |
| $ SOME_VAR=somevalue ${TURBO} run add-keys-task --filter=add-keys | |
| \xe2\x80\xa2 Packages in scope: add-keys (esc) | |
| \xe2\x80\xa2 Running add-keys-task in 1 packages (esc) | |
| \xe2\x80\xa2 Remote caching disabled (esc) | |
| add-keys:add-keys-underlying-task: cache hit, replaying output b7390d43fae5a180 | |
| add-keys:add-keys-underlying-task: | |
| add-keys:add-keys-underlying-task: > add-keys-underlying-task | |
| add-keys:add-keys-underlying-task: > echo "running add-keys-underlying-task" | |
| add-keys:add-keys-underlying-task: | |
| add-keys:add-keys-underlying-task: running add-keys-underlying-task | |
| add-keys:add-keys-task: cache miss, executing 8e76bee49319e00e | |
| add-keys:add-keys-task: | |
| add-keys:add-keys-task: > add-keys-task | |
| add-keys:add-keys-task: > echo "running add-keys-task" > out/foo.min.txt | |
| add-keys:add-keys-task: | |
| Tasks: 2 successful, 2 total | |
| Cached: 1 cached, 2 total | |
| Time:\s*[\.0-9]+m?s (re) | |
| # 3. Change input file and assert cache miss | |
| $ echo "more text" >> $TARGET_DIR/apps/add-keys/src/foo.txt | |
| $ ${TURBO} run add-keys-task --filter=add-keys --dry=json | jq -r ".tasks[].hash" | |
| b7390d43fae5a180 | |
| b256f31e3957fee3 | |
| # 4. Set env var and assert cache miss | |
| $ SOME_VAR=somevalue ${TURBO} run add-keys-task --filter=add-keys --dry=json | jq -r ".tasks[].hash" | |
| b7390d43fae5a180 | |
| 8e76bee49319e00e |
| $ ${TURBO} run added-task --filter=add-tasks | ||
| \xe2\x80\xa2 Packages in scope: add-tasks (esc) | ||
| \xe2\x80\xa2 Running added-task in 1 packages (esc) | ||
| \xe2\x80\xa2 Remote caching disabled (esc) | ||
| add-tasks:added-task: cache miss, executing 49e39f2b39207531 | ||
| add-tasks:added-task: | ||
| add-tasks:added-task: > added-task | ||
| add-tasks:added-task: > echo "running added-task" > out/foo.min.txt | ||
| add-tasks:added-task: | ||
|
|
||
| Tasks: 1 successful, 1 total | ||
| Cached: 0 cached, 1 total | ||
| Time:\s+[.0-9]+m?s (re) | ||
|
No newline at end of file |
There was a problem hiding this comment.
One bonus example:
| $ ${TURBO} run added-task --filter=add-tasks | |
| \xe2\x80\xa2 Packages in scope: add-tasks (esc) | |
| \xe2\x80\xa2 Running added-task in 1 packages (esc) | |
| \xe2\x80\xa2 Remote caching disabled (esc) | |
| add-tasks:added-task: cache miss, executing 49e39f2b39207531 | |
| add-tasks:added-task: | |
| add-tasks:added-task: > added-task | |
| add-tasks:added-task: > echo "running added-task" > out/foo.min.txt | |
| add-tasks:added-task: | |
| Tasks: 1 successful, 1 total | |
| Cached: 0 cached, 1 total | |
| Time:\s+[.0-9]+m?s (re) | |
| $ ${TURBO} run added-task --filter=add-tasks --dry=json | jq -r ".tasks[].hash" | |
| 49e39f2b39207531 |
| $ HASH=$(cat tmp.log | grep -E "cached:cached-task-1.* executing .*" | awk '{print $5}') | ||
| $ echo $HASH | ||
| [a-z0-9]{16} (re) | ||
| $ tar -tf $TARGET_DIR/node_modules/.cache/turbo/$HASH.tar.zst; |
There was a problem hiding this comment.
Can we use --cache-dir for these tests since many tests implicitly rely on that value?
There was a problem hiding this comment.
That's a good idea, (correct me if I'm wrong), we'd still have to do this dance to get the task hash. Is there any benefit to replacing$TARGET_DIR/node_modules/.cache/turbo with a custom value?
There was a problem hiding this comment.
The only reason is that it would make it possible to write a simple "check the result" helper function.
The "grab hash" thing is actually the thing I want to fix more than anything else, but we don't make that easy.
There was a problem hiding this comment.
I'll keep things the way they are, since they're already written, but I'll write new tests the way you suggest if applicable. E.g. #3778.
I'm open to this, but did it this way for a few reasons:
I do agree that reading log output for the purpose of reading tests is hard though. Two things on that:
In any case, let me know what you think, I could see some of these test cases being reduced |
gsoltis
left a comment
There was a problem hiding this comment.
I left a bunch of notes, but I think it nearly all boils down to one suggestion: at some level, either in fs or in graph, I think we should distinguish between a turbo file not being found vs some unexpected error occurring that needs to be reported to the user.
Adds a generic method to validate TurboJSON structs and validate when getting the tasks.
Co-authored-by: Greg Soltis <greg.soltis@vercel.com>
Co-authored-by: Greg Soltis <greg.soltis@vercel.com>
Only allow the error where workspace config is missing, since this config is not required. Throw other errors, including when the contents of the workspace config are malformed. This allows us to provide better error messaging to the end user
When `inputs` is configured for a task, we want to ensure that changes to the `turbo.json` in that workspace cause a cache miss. This is similar to how we treat the `package.json` in the workspace, and also how we treat the root `turbo.json` for the global hash.
This feature allows monorepos to place a
turbo.jsonfile in workspaces and add a"extends"key to extend from the root configuration to add/remove to the pipeline. After this PR, individual workspaces should be able to:dependsOnoroutputsconfigs)e.g.dependsOn: []`)lintcommand only in the workspace that implements it)Limitations
Some of these limitations are edge cases, but still listing them out. You cannot:
pipelinetasks. For example,remoteCache,globalEnv, etc cannot be changed for individual workspaces.turbo.jsonto extend from anything.dependsOn: ['something'], you can't append to this array, you can only overwrite it.)package#tasksyntax in workspaces. Thepackageis inferred.These limitations are based on a lot of research to keep the design space open, gather use cases, and avoid making the wrong decisions before putting the feature into the hands of real users. In short: to make it possible to ship this sooner!
Example
./turbo.json(root){ "pipeline": { "build": { "outputs": ["dist/**"], "dependsOn": ["lint"] } } }./packages/ui/turbo.json(uiworkspace){ "extends": ["//"], "pipeline": { "build": { "dependsOn": ["lint", "compile"], "outputs": ["lib/**"] } } }With these configs, when you run
turbo build, all workspaces except theuiworkspace will cache thedist/directory and run theirlintcommands first. Theuiworkspace will runlintandcompilefirst, and cache thelib/directory instead.Review Notes
*.tfiles) to see what use cases this supportsTests
Pre-req work
Deferred Work
fieldsMeta+ Merge is won't scale as new keys are added. Need a more elegant solution