You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
Yeah, this will require quite a bit of work, likely. Let me know if I can help: Kotlin module has some challenges too, but I did help with some changes there.
to allow basic sanity/smoke tests; something that would run simple tests to ensure that a set (like 2.12.0-SNAPSHOT, 3.0.0-SNAPSHOT) of components in their latest publish snapshots versions still works. I have added tests for dataformats, some of datatypes (joda, guava), as well as certain combinations (Joda + XML/CSV/Properties wrt timestamp values).
I am not sure exactly how, but I wonder if there was a way to invoke some Scala mapper sanity tests from Java, built by Maven, to do simplest of validations that a scala-supporting object mapper can be constructed and used on something that exercises its functionality?
I suspect the main challenge is that of building scala types for testing; perhaps integration-tests needs to be split to either have a support package(s) of value types (Kotlin no doubt needs something similar), or even just fully separate sub-projects (making current main-level project a sub-project).
At any rate, I would really like to quickly find out if changes I am making break Scala module: I just integrated PR for
and doing refactoring for it. I will try to keep things backwards compatible for 2.12 obviously, but sometimes there are dependencies I do not see (Scala and Kotlin typically, although of course there are many 3rd party extensions beyond this). And with 3.x I will simply remove deprecated methods so there is more work anyway.
I'm not sure - all that can be done is to copy lots of the code and tests from jackson-module-scala into jackson-integration-tests - I'd be reluctant to maintain 2 copies of the code.
I have nightly builds running jackson-module-scala on travis but recently my travis account has had a problem that doesn't let me set up new builds for github orgs that I am member of. I need to either follow up with travis or try something else like github actions.
I'm afraid master version of jackson-module-scala was well broken before I even had a chance to set up builds. I have it compiling and tests running but many are broken - many tests are probably broken for similar reasons.
@cowtowncoder a lot of the broken tests appear to be because the DeserializerResolvers in master do not appear to have calls made to findReferenceDeserializer - only to findBeanDeserializer.
Tests that pass in v2.12 and that fail in master have lots of calls to findReferenceDeserializer in 2.12 but none in master.
@pjfanning Quick clarification wrt jackson-integration-tests: the idea would be a very small set of tests, similar to "hello world" style, even just one read, one write or such. To catch catastrophic cases. I wouldn't want to maintain separate sets of tests either.
One other benefit is ability to test combinations (like Scala + CSV, Scala + XML), without adding dependencies across components. Those tests would not be duplicated FWTW.
But it is up to you of course, I will mention this as something I plan to make use of and wanted to mention.
@pjfanning On findReferenceDeserializer: that sounds odd, as if anything, 3.0 should be making more accurate calls -- although without some of fallbacks, which may be the reason for breakage.
I would guess 2 likely sources of issues:
Some calls that relied on fallback (first check reference deserializer, if not found, try bean deserializer?) do not implement fallbacks any more
Type refinement is somehow not working so calls are being made to wrong lookup methods
Would it be possible to test to see if calls expected to go to findReferenceDeserializer() end up on findBeanDeserializer() or vice versa? Databind itself has only one reference type (for AtomicReference) and while that is tested is not extensively tested.
Yes, java.util.Optional is a reference type for sure; refined as such by jackson-module-java8 (in 2.x), and jackson-databind itself in master.
And I think scala.Option should become one: but it is up to module to make sure that happens with TypeModifier implementation. But I think this is done by Scala module, both for 2.x and 3.0, and should also be tested that this occurs (by resolving via TypeFactory on which appropriate TypeModifier has been registered).
In 2.12, the reference type deserializer is created in this stack - this does not happen in master, despite the same ReferenceType being in scope.
java.lang.Exception: >>>>>> EitherDeserializerResolver findReferenceDeserializer class scala.util.Either
at com.fasterxml.jackson.module.scala.deser.EitherDeserializerResolver$.findReferenceDeserializer(EitherDeserializer.scala:121)
at com.fasterxml.jackson.databind.deser.BasicDeserializerFactory._findCustomReferenceDeserializer(BasicDeserializerFactory.java:1976)
at com.fasterxml.jackson.databind.deser.BasicDeserializerFactory.createReferenceDeserializer(BasicDeserializerFactory.java:1553)
at com.fasterxml.jackson.databind.deser.DeserializerCache._createDeserializer2(DeserializerCache.java:409)
at com.fasterxml.jackson.databind.deser.DeserializerCache._createDeserializer(DeserializerCache.java:349)
at com.fasterxml.jackson.databind.deser.DeserializerCache._createAndCache2(DeserializerCache.java:264)
at com.fasterxml.jackson.databind.deser.DeserializerCache._createAndCacheValueDeserializer(DeserializerCache.java:244)
at com.fasterxml.jackson.databind.deser.DeserializerCache.findValueDeserializer(DeserializerCache.java:142)
at com.fasterxml.jackson.databind.DeserializationContext.findNonContextualValueDeserializer(DeserializationContext.java:581)
at com.fasterxml.jackson.databind.deser.BeanDeserializerBase.resolve(BeanDeserializerBase.java:524)
at com.fasterxml.jackson.databind.deser.DeserializerCache._createAndCache2(DeserializerCache.java:293)
at com.fasterxml.jackson.databind.deser.DeserializerCache._createAndCacheValueDeserializer(DeserializerCache.java:244)
at com.fasterxml.jackson.databind.deser.DeserializerCache.findValueDeserializer(DeserializerCache.java:142)
at com.fasterxml.jackson.databind.DeserializationContext.findRootValueDeserializer(DeserializationContext.java:591)
at com.fasterxml.jackson.databind.ObjectMapper._findRootDeserializer(ObjectMapper.java:4684)
at com.fasterxml.jackson.databind.ObjectMapper._readMapAndClose(ObjectMapper.java:4545)
at com.fasterxml.jackson.databind.ObjectMapper.readValue(ObjectMapper.java:3497)
at com.fasterxml.jackson.databind.ObjectMapper.readValue(ObjectMapper.java:3480)
@pjfanning Ok good wrt Either tests. Added a note on json-schema module: at this point I have no plans to support it for 3.x, will be end-of-life'd (not much work on 2.x either).
Ok that makes sense. The problem seems to be that introspection does not find creator property for some reason. Not MetricPath type itself. In fact it would seem that it could as well be any other type (String) and same should occur?
At this point, would it make sense to create a new issue for this one problem. Not sure if there are other ones, but discussion here has gotten bit long and covers a few different issues, most resolved?
I found some code in the Scala annotation classes that is not behaving as it does in Jackson 2 - let me investigate this - it could be the cause of the issue here
@cowtowncoder a possible/probable cause is that the jackson-module-scala was modified in jackson 3 because the interface in jackson-databind changed.
ValueInstantiators findValueInstantiator(DeserializationConfig var1, BeanDescription var2, ValueInstantiator var3) has dropped the final param - the defaultValueInstantiator
The scala code now struggles to replace that default. Any suggestions as to how to get a default instantiator in jackson 3?
@pjfanning Let me see, I think this was just split in two parts or so.
Ok, yes; so, there is new modifyValueInstantiator() which is what you want to use in case you want to use the default one: so existing findValueInstantiator() is meant for full override case; and then modifyValueInstantiator() for things like delegation.
I have no value instantiator so can't modify it. Is there any way to create a new default instantiator that has all the creator properties calculated - that's what I get in jackson 2 - automatically but now I am left with no instantiator and need to create from scratch .
findValueInstantiator(config: DeserializationConfig, ...) -- because this method does not give access to a full DeserializationContext, it is hard to try BeanDeserializerBuilder to try to get a value instantiator from it
you are given the default instantiator UNLESS earlier call to
public ValueInstantiator findValueInstantiator(DeserializationConfig config,
BeanDescription beanDesc);
already returned a ValueInstantiator.
So basically earlier single call was split in two parts: one to ask for instantiator, and if none given, a default POJO instantiator is created and passed to give another chance for custom variants.
So if you want to use that default instantiator it is given and you may return it as-is, or create different one based on it (or just ignore).
Ok now I am confused. This is a callback method of ValueInstantiators interface.
Scala module should not usually be calling it.... it is called by Jackson databind (specifically BasicDeserializerFactory). Modules would implement ValueInstantiators callback to be able to provide alternative instantiator. That handling was split from 1 to 2 methods.
If Scala module does need to call it, where is the code? Maybe I can help with it.
In jackson 2, the perfect value instantiator is created in BasicDeserializer.findValueInstantiator -- and specifically when this is called instantiator = this._constructDefaultValueInstantiator(ctxt, beanDesc);
This instantiator is then used when the call to the old ValueInstantiators findValueInstantiator(DeserializationConfig var1, BeanDescription var2, ValueInstantiator var3) function is called
I think I fixed it - I made the findValueInstantiator return null so now the modifyValueInstantiator is called with the instantiator created in the BasicDeserializer
@pjfanning Ok good, yes, findValueInstantiator() should return null to indicate "go ahead construct default one" and then that one will be passed for possible modification/replacement. So fix makes sense.
Activity
cowtowncoder commentedon Sep 22, 2019
Yeah, this will require quite a bit of work, likely. Let me know if I can help: Kotlin module has some challenges too, but I did help with some changes there.
cowtowncoder commentedon Jul 25, 2020
@pjfanning One question/suggestion, more related to 2.12 (but eventually to 3.0/master too): I created this project:
https://github.com/FasterXML/jackson-integration-tests
to allow basic sanity/smoke tests; something that would run simple tests to ensure that a set (like 2.12.0-SNAPSHOT, 3.0.0-SNAPSHOT) of components in their latest publish snapshots versions still works. I have added tests for dataformats, some of datatypes (joda, guava), as well as certain combinations (Joda + XML/CSV/Properties wrt timestamp values).
I am not sure exactly how, but I wonder if there was a way to invoke some Scala mapper sanity tests from Java, built by Maven, to do simplest of validations that a scala-supporting object mapper can be constructed and used on something that exercises its functionality?
I suspect the main challenge is that of building scala types for testing; perhaps integration-tests needs to be split to either have a support package(s) of value types (Kotlin no doubt needs something similar), or even just fully separate sub-projects (making current main-level project a sub-project).
At any rate, I would really like to quickly find out if changes I am making break Scala module: I just integrated PR for
FasterXML/jackson-databind#1296
and doing refactoring for it. I will try to keep things backwards compatible for 2.12 obviously, but sometimes there are dependencies I do not see (Scala and Kotlin typically, although of course there are many 3rd party extensions beyond this). And with 3.x I will simply remove deprecated methods so there is more work anyway.
pjfanning commentedon Jul 25, 2020
I'm not sure - all that can be done is to copy lots of the code and tests from jackson-module-scala into jackson-integration-tests - I'd be reluctant to maintain 2 copies of the code.
I have nightly builds running jackson-module-scala on travis but recently my travis account has had a problem that doesn't let me set up new builds for github orgs that I am member of. I need to either follow up with travis or try something else like github actions.
I'm afraid master version of jackson-module-scala was well broken before I even had a chance to set up builds. I have it compiling and tests running but many are broken - many tests are probably broken for similar reasons.
pjfanning commentedon Jul 25, 2020
@cowtowncoder a lot of the broken tests appear to be because the DeserializerResolvers in master do not appear to have calls made to findReferenceDeserializer - only to findBeanDeserializer.
Tests that pass in v2.12 and that fail in master have lots of calls to findReferenceDeserializer in 2.12 but none in master.
https://github.com/FasterXML/jackson-module-scala/blob/master/src/main/scala/com/fasterxml/jackson/module/scala/deser/EitherDeserializer.scala#L115
cowtowncoder commentedon Jul 25, 2020
@pjfanning Quick clarification wrt
jackson-integration-tests: the idea would be a very small set of tests, similar to "hello world" style, even just one read, one write or such. To catch catastrophic cases. I wouldn't want to maintain separate sets of tests either.One other benefit is ability to test combinations (like Scala + CSV, Scala + XML), without adding dependencies across components. Those tests would not be duplicated FWTW.
But it is up to you of course, I will mention this as something I plan to make use of and wanted to mention.
cowtowncoder commentedon Jul 25, 2020
@pjfanning On
findReferenceDeserializer: that sounds odd, as if anything, 3.0 should be making more accurate calls -- although without some of fallbacks, which may be the reason for breakage.I would guess 2 likely sources of issues:
Would it be possible to test to see if calls expected to go to
findReferenceDeserializer()end up onfindBeanDeserializer()or vice versa? Databind itself has only one reference type (forAtomicReference) and while that is tested is not extensively tested.pjfanning commentedon Jul 25, 2020
I'm not sure when I'll next have chance to spend time on this.
With reference types, wouldn't java.util.Optional be a reference type (akin to scala.Option and the master tests are failing for scala.Option)?
cowtowncoder commentedon Jul 25, 2020
Yes,
java.util.Optionalis a reference type for sure; refined as such byjackson-module-java8(in 2.x), andjackson-databinditself inmaster.And I think
scala.Optionshould become one: but it is up to module to make sure that happens withTypeModifierimplementation. But I think this is done by Scala module, both for 2.x and 3.0, and should also be tested that this occurs (by resolving viaTypeFactoryon which appropriateTypeModifierhas been registered).pjfanning commentedon Jul 25, 2020
https://github.com/FasterXML/jackson-module-scala/blob/master/src/main/scala/com/fasterxml/jackson/module/scala/modifiers/ScalaTypeModifier.scala seems to be what does this
pjfanning commentedon Jul 26, 2020
In 2.12, the reference type deserializer is created in this stack - this does not happen in master, despite the same ReferenceType being in scope.
pjfanning commentedon Jul 26, 2020
I found what was breaking the scala Either tests on master but quite a lot more stuff to fix still.
@cowtowncoder could you look at FasterXML/jackson-module-jsonSchema#140 ?
cowtowncoder commentedon Jul 26, 2020
@pjfanning Ok good wrt Either tests. Added a note on json-schema module: at this point I have no plans to support it for 3.x, will be end-of-life'd (not much work on 2.x either).
52 remaining items
cowtowncoder commentedon Feb 11, 2021
Ok that makes sense. The problem seems to be that introspection does not find creator property for some reason. Not
MetricPathtype itself. In fact it would seem that it could as well be any other type (String) and same should occur?At this point, would it make sense to create a new issue for this one problem. Not sure if there are other ones, but discussion here has gotten bit long and covers a few different issues, most resolved?
pjfanning commentedon Feb 11, 2021
I found some code in the Scala annotation classes that is not behaving as it does in Jackson 2 - let me investigate this - it could be the cause of the issue here
pjfanning commentedon Feb 11, 2021
@cowtowncoder a possible/probable cause is that the jackson-module-scala was modified in jackson 3 because the interface in jackson-databind changed.
ValueInstantiators findValueInstantiator(DeserializationConfig var1, BeanDescription var2, ValueInstantiator var3) has dropped the final param - the defaultValueInstantiator
The scala code now struggles to replace that default. Any suggestions as to how to get a default instantiator in jackson 3?
cowtowncoder commentedon Feb 11, 2021
@pjfanning Let me see, I think this was just split in two parts or so.
Ok, yes; so, there is new
modifyValueInstantiator()which is what you want to use in case you want to use the default one: so existingfindValueInstantiator()is meant for full override case; and thenmodifyValueInstantiator()for things like delegation.pjfanning commentedon Feb 11, 2021
I have no value instantiator so can't modify it. Is there any way to create a new default instantiator that has all the creator properties calculated - that's what I get in jackson 2 - automatically but now I am left with no instantiator and need to create from scratch .
findValueInstantiator(config: DeserializationConfig, ...) -- because this method does not give access to a full DeserializationContext, it is hard to try BeanDeserializerBuilder to try to get a value instantiator from it
cowtowncoder commentedon Feb 11, 2021
Please look at signature:
you are given the default instantiator UNLESS earlier call to
already returned a
ValueInstantiator.So basically earlier single call was split in two parts: one to ask for instantiator, and if none given, a default POJO instantiator is created and passed to give another chance for custom variants.
So if you want to use that default instantiator it is given and you may return it as-is, or create different one based on it (or just ignore).
pjfanning commentedon Feb 11, 2021
I have no defaultInstantiator to pass in
cowtowncoder commentedon Feb 11, 2021
Ok now I am confused. This is a callback method of
ValueInstantiatorsinterface.Scala module should not usually be calling it.... it is called by Jackson databind (specifically
BasicDeserializerFactory). Modules would implementValueInstantiatorscallback to be able to provide alternative instantiator. That handling was split from 1 to 2 methods.If Scala module does need to call it, where is the code? Maybe I can help with it.
pjfanning commentedon Feb 11, 2021
In jackson 2, the perfect value instantiator is created in BasicDeserializer.findValueInstantiator -- and specifically when this is called
instantiator = this._constructDefaultValueInstantiator(ctxt, beanDesc);This instantiator is then used when the call to the old
ValueInstantiators findValueInstantiator(DeserializationConfig var1, BeanDescription var2, ValueInstantiator var3)function is calledpjfanning commentedon Feb 11, 2021
https://github.com/FasterXML/jackson-module-scala/blob/2.13/src/main/scala/com/fasterxml/jackson/module/scala/introspect/ScalaAnnotationIntrospectorModule.scala#L130
pjfanning commentedon Feb 11, 2021
I think I fixed it - I made the findValueInstantiator return null so now the modifyValueInstantiator is called with the instantiator created in the BasicDeserializer
pjfanning commentedon Feb 11, 2021
We're now down to 2 or 3 commented out tests in master that pass in jackson v2 and that fail in v3 - today's fix fixed about 5 broken tests
cowtowncoder commentedon Feb 11, 2021
@pjfanning Ok good, yes,
findValueInstantiator()should returnnullto indicate "go ahead construct default one" and then that one will be passed for possible modification/replacement. So fix makes sense.Getting very close to parity which is good!
pjfanning commentedon Jul 31, 2021
created #536 for follow on work