You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
{{ message }}
Repository navigation
ClassLoader can't solve directories on Graal binaryΒ #1108
Reacted by Kazuki Shimizu, Guanchao Yang, Adam Richeimer, gudaoxuri, Ronald Dehuysser, Martin, Derek Adams, Badr NASS LAHSEN, Khurelkhuyag, Hengqian Ling and 1 more
@guanchao-yang, @mageddo as of now (GraalVM 19.1.1) we only support adding resources that are real data files to native-images. A directory on its own is not seen as a resource because we cannot embed a directory into a native-image. If we'd add support for that, which kind of usage patterns would need to be supported on resources that represent directories to make spring work as expected?
which kind of usage patterns would need to be supported on resources that represent directories to make spring work as expected?
It's nothing really to do with Spring. The question above was about Flyway, where they sort the filenames in a directory and execute them as scripts in order. Webjars also likes to traverse "directories" in classpath resources (so it can resolve the "best" available resource matching the request). It's just a common pattern, to search for stuff in packages and subpackages.
Here's a library dedicated to scanning classpath directories: https://github.com/classgraph/classgraph. It's used by the most recent versions of webjars, and naturally doesn't work in substratevm because of the above.
Reacted by Brice Dutheil, Lars Uffmann, Elvis Souza, Sam Brannen, Ronald Dehuysser, Omkar , Martin, Derek Adams, Badr NASS LAHSEN, PengEJ and 3 more
Any chances that this issue here will be implemented in the foreseeable future?
Right now, we have to come up with workarounds for the classpath enumeration. It would be great if an application running in a native image could just use the classpath enumeration libraries like Reflections / Classgraph / Springs PathMatchingResourcePatternResolver and get back the included resources like on the JVM. This would make all the workarounds no longer necessary.
This works because there is a filesystem named com.oracle.svm.core.jdk.resources.NativeImageResourceFileSystemProvider available which handles the resource scheme.
But this doesn't matter, because now we have to convince maintainers of Flyway, Reflections, etc. to add specific support for GraalVM. It would be nice if this would work in a JVM compatible way.
I see that the discussion is a bit old, but maybe I could help here.
The provided example is not building at all because of an old version of the native image.
After some manual changes, I managed to make it work. Here is a command that I have used: native-image -H:IncludeResources="folder/.*" -cp "build/classes/java/main/:build/resources/main/" com.mageddo.resources.resolution.App. So, the initial problem is not relevant anymore, right?
I'm the creator of NativeImageResourceFileSystemProvider, so that caveat is actually my mistake. I overlooked that somehow.
@mhalbritter How exactly we can help from the Native Image side? Is this the exampe that is not working?
I couldn't build the original sample either. If you could post your changes it might be useful. How do you mean "it worked"? What did you run and what was the output?
If you run ./gradlew :flyway:bootRun it works and reports the 2 migrations. If you try ./gradlew :flyway:nativeRun it will run but report that 0 migrations were found.
I'm not sure what is the problem with the original sample Gradle setup, so I will need to take a better look to see why it's not building using ./gradle nativeImage.
Regarding the spring aot smoke test, I'm running it right now.
I think @mhalbritter already alluded to this, but I wasn't following. It looks to me as if Flyway (and Spring etc.) could use the FileSystems abstraction, so they don't need to know about GraalVM, just java.nio.
Existing libraries have been working for over a decade scanning the classpath using ClassLoader#getResources.
Requiring all applications and libraries to add special support for resource: is therefore not a viable option IMO, since that would require many libraries to be rewritten or augmented to support resource:.
The only reasonable solution I see is for GraalVM to provide the support already available in NativeImageResourceFileSystemProvider transparently via ClassLoader#getResources.
But ClassLoader#getResources returns an array of URI so I don't see how GraalVM can change that in a sensible way. It's how the existing libraries resolve the URI that's the problem. They have been hacking on that for decades, making assumptions about "jar:" and "file:" prefixes, and ignoring the abstractions provided by java.nio.FileSystems. Isn't it time we used the abstractions provided by the JDK (which would work in a native image too)?
They have been hacking on that for decades, making assumptions about "jar:" and "file:" prefixes, and ignoring the abstractions provided by java.nio.FileSystems. Isn't it time we used the abstractions provided by the JDK (which would work in a native image too)?
Good point.
Are you proposing that libraries like Spring Framework attempt to use java.nio.FileSystems for any unrecognized protocol (like resource:)?
If so, then I question GraalVM's choice of resource as a protocol since it is a rather generic term that may already be in use. I would expect a protocol supported only within a GraalVM native image to have a more specific name such as native-resource:.
@mhalbritter's code above fails if the starting URI is a file:/.... The FileSystem implementation likes to have absolute paths that start and end with "/", but the resource: implementation in GraalVM doesn't. So you can't use the same code with both unless you do some ugly hacks:
StringrootPath = rootDirResource.getURI().getRawPath();
if (!("file").equals(rootDirResource.getURI().getScheme()) && rootPath.startsWith("/")) {
rootPath = rootPath.substring(1);
if (rootPath.length()==0) {
returnresult;
}
if (rootPath.endsWith("/")) {
rootPath = rootPath.substring(0, rootPath.length()-1);
}
if (rootPath.length()==0) {
returnresult;
}
}
@bclozel thanks for extracting the issue from this discussion, that is actually something that I've planned to do. Sure, we can close this one now. π―
Activity
guanchao-yang commentedon Jul 24, 2019
@olpaw Which version do we plan to release ? Thank you very much
olpaw commentedon Jul 24, 2019
@guanchao-yang, @mageddo as of now (GraalVM 19.1.1) we only support adding resources that are real data files to native-images. A directory on its own is not seen as a resource because we cannot embed a directory into a native-image. If we'd add support for that, which kind of usage patterns would need to be supported on resources that represent directories to make spring work as expected?
dsyer commentedon Sep 16, 2019
It's nothing really to do with Spring. The question above was about Flyway, where they sort the filenames in a directory and execute them as scripts in order. Webjars also likes to traverse "directories" in classpath resources (so it can resolve the "best" available resource matching the request). It's just a common pattern, to search for stuff in packages and subpackages.
Here's a library dedicated to scanning classpath directories: https://github.com/classgraph/classgraph. It's used by the most recent versions of webjars, and naturally doesn't work in substratevm because of the above.
rtfpessoa commentedon May 9, 2020
Any progress here? Are there any workarounds we can do to list all files in a directory or in the classpath in general?
Update tests regarding GraalVM native image issue
rdehuyss commentedon Jul 21, 2020
Also running in troubles because of this...
jantoniucci commentedon Mar 5, 2021
Me too. Are there any update nor workaround on it ? Thanks!
mhalbritter commentedon Aug 24, 2022
We're currently working on implementing Flyway support on native-image and it turns out quite harder than we initially thought it would be.
Any chances that this issue here will be implemented in the foreseeable future?
Right now, we have to come up with workarounds for the classpath enumeration. It would be great if an application running in a native image could just use the classpath enumeration libraries like Reflections / Classgraph / Springs
PathMatchingResourcePatternResolverand get back the included resources like on the JVM. This would make all the workarounds no longer necessary.mhalbritter commentedon Aug 25, 2022
Interestingly, one can list resources at runtime:
with one caveat: you can't list starting from the root folder, you have to know the name of the first directory where to start.
With
resource-config.jsonlike this:{ "resources": { "includes": [ { "pattern": "folder.*" } ] } }and a directory tree under
src/main/resourceslike this:this prints
This works because there is a filesystem named
com.oracle.svm.core.jdk.resources.NativeImageResourceFileSystemProvideravailable which handles theresourcescheme.But this doesn't matter, because now we have to convince maintainers of Flyway, Reflections, etc. to add specific support for GraalVM. It would be nice if this would work in a JVM compatible way.
jovanstevanovic commentedon Sep 14, 2022
I see that the discussion is a bit old, but maybe I could help here.
native-image -H:IncludeResources="folder/.*" -cp "build/classes/java/main/:build/resources/main/" com.mageddo.resources.resolution.App. So, the initial problem is not relevant anymore, right?NativeImageResourceFileSystemProvider, so that caveat is actually my mistake. I overlooked that somehow.dsyer commentedon Sep 14, 2022
I couldn't build the original sample either. If you could post your changes it might be useful. How do you mean "it worked"? What did you run and what was the output?
Instead I tried the Flyway sample at https://github.com/spring-projects/spring-aot-smoke-tests with a manual
resources-config.json:{ "resources": { "includes": [ { "pattern": "\\Qdb\/migration\/\\E.*" } ] } }If you run
./gradlew :flyway:bootRunit works and reports the 2 migrations. If you try./gradlew :flyway:nativeRunit will run but report that 0 migrations were found.jovanstevanovic commentedon Sep 14, 2022
Regarding the original sample (using the latest labsjdk11 and native image):
The output was as expected:
I'm not sure what is the problem with the original sample Gradle setup, so I will need to take a better look to see why it's not building using
./gradle nativeImage.Regarding the spring aot smoke test, I'm running it right now.
dsyer commentedon Sep 14, 2022
So the problem is that
resource:is the URL prefix for the resources found in the native image, but Flyway doesn't think it knows how to resolve them:I think @mhalbritter already alluded to this, but I wasn't following. It looks to me as if Flyway (and Spring etc.) could use the
FileSystemsabstraction, so they don't need to know about GraalVM, justjava.nio.sbrannen commentedon Sep 14, 2022
Right.
resource:is non-standard.Existing libraries have been working for over a decade scanning the classpath using
ClassLoader#getResources.Requiring all applications and libraries to add special support for
resource:is therefore not a viable option IMO, since that would require many libraries to be rewritten or augmented to supportresource:.The only reasonable solution I see is for GraalVM to provide the support already available in
NativeImageResourceFileSystemProvidertransparently viaClassLoader#getResources.sdeleuze commentedon Sep 14, 2022
I tend to agree with @sbrannen take above.
dsyer commentedon Sep 14, 2022
But
ClassLoader#getResourcesreturns an array ofURIso I don't see how GraalVM can change that in a sensible way. It's how the existing libraries resolve the URI that's the problem. They have been hacking on that for decades, making assumptions about "jar:" and "file:" prefixes, and ignoring the abstractions provided byjava.nio.FileSystems. Isn't it time we used the abstractions provided by the JDK (which would work in a native image too)?sbrannen commentedon Sep 14, 2022
Good point.
Are you proposing that libraries like Spring Framework attempt to use
java.nio.FileSystemsfor any unrecognized protocol (likeresource:)?If so, then I question GraalVM's choice of
resourceas a protocol since it is a rather generic term that may already be in use. I would expect a protocol supported only within a GraalVM native image to have a more specific name such asnative-resource:.dsyer commentedon Sep 15, 2022
@mhalbritter's code above fails if the starting URI is a
file:/.... TheFileSystemimplementation likes to have absolute paths that start and end with "/", but theresource:implementation in GraalVM doesn't. So you can't use the same code with both unless you do some ugly hacks:It would be good to not have to do that.
bclozel commentedon Sep 21, 2022
@jovanstevanovic I've raised #5020, could you have a look? I'll try extracting feedback from here into separate issues so that we can close this one.
jovanstevanovic commentedon Sep 21, 2022
@bclozel thanks for extracting the issue from this discussion, that is actually something that I've planned to do. Sure, we can close this one now. π―
Update tests regarding GraalVM native image issue