Skip to content

Add configurable processing limits for JSON parser (StreamReadConstraints)Β #637

Closed

Description

@cowtowncoder

(note: related to/inspired by FasterXML/jackson-databind#2816)

Some aspects of input document are prone to possible abuse, so that malicious sender can create specifically crafted documents to try to overload server. This includes things like:

  1. Ridiculously deeply nested documents (it only takes 2 characters to create Array context so 8k document can have 4000 levels of nesting) -- implemented via Add StreamReadConstraints.maxNestingDepth() to constraint max nesting depth (default: 1000) [CVE-2025-52999]Β #943
  2. Very long documents (in general): while streaming decoder has no problems with length per se, higher level databind can easily exhaust memory even with somewhat normal amplification (that is, input content of 10 megabytes can result in a data structure of 100 megabytes) -- implemented via Add configurable limit for the maximum number of bytes/chars of content to parse before failingΒ #1046
  3. Very long names: Jackson's name decoding is optimized for short names and while it can handle any length (within bounds of total memory available) performance characteristics are not great beyond, say, couple of thousands of characters -- implemented via Add configurable limit for the maximum length of Object property names to parse before failing (default max: 50,000 chars)Β #1047
  4. Very long numbers: textual length of tens of thousands of numbers might be problematic for some uses -- implemented via Add numeric value size limits via StreamReadConstraints -- default 1000 chars (fixes sonatype-2022-6438)Β #827
  5. (possibly?) Huge number of properties per JSON Object -- for some use cases construction of data structures with tens or hundreds of thousands of distinct keys can be problematic -- not implemented, no immediate plans; can create new issue if this is still desired (ditto for big Arrays)
  6. Extra long text segments (megabytes of content) can similarly become problematic -- this could also result from broken encoding (missing closing quote) -- implemented via Add StreamReadConstraints limit for longest textual value to allow (default: 5M in 2.15.0; 20M in 2.15.1)Β #863

and although streaming parser can typically handle many of these cases quite well, they can be very problematic for higher-level processing -- and even for streaming, for highly parallel processing.

So. It would be good to create a configurable set of options that:

  1. Default to same safe set of limits for likely problematic cases (like limit nesting to what is known to typically fit in wrt stack frames; limit maximum property names)
  2. Leave more speculative limits (text length) to unlimited (or very high)
  3. Offer a simple way to configure limits (possibly only per JsonFactory, although it'd be really nice if per-parser overrides were possible)

Further reading: related material.

Here are some relevant links:

Activity

  1. cowtowncoder commented on Feb 13, 2022

    @cowtowncoder
    MemberAuthor

    Since 2.13 has been released marking as 2.14 (earliest) due to need for API additions and possible compatibility concerns for some maximum limits.

    Also: I think this issue is still important even if databind tackles some of more immediate concerns (via FasterXML/jackson-databind#2816).

  2. cowtowncoder commented on Feb 14, 2022

    @cowtowncoder
    MemberAuthor

    Some thinking out aloud:

    • Limits for total input document size can probably be checked at points when input blocks are read (blocking) or fed (async): this is what Woodstox does and appears to work beautifully with minimal overhead. Limits probably need to be in units of "whatever input unit is, byte/char"
    • Limit for nesting is a much bigger challenge but my first instinct is to integrate this to JsonStreamContext/JsonReadContext (2.x) (TokenStreamContext in 3.0) -- that is, check it when trying to add next context. This can get tricky due to inability to throw checked (in 2.x) JsonProcessingException but we'll cross that bridge if and when we get that far.... there's also the obvious challenge of how to pass limit configuration in the first place.

    As to configuration settings, there probably needs to be a shared base type (StreamProcessingLimits), as concrete implementation, and with a singleton "default" or "empty" instance to feed in case where no explicit configuration is used. And then with format-specific extensions allowed.
    Should probably use Builder-style construction, limits instances fully immutable (obviously) to be shareable.

    One immediate concern I have, however, is that of adding state to JsonReadContext: seems like we need both current level (int) and configuration settings; latter to be passed when creating "root" instance, then passed to children.
    Alternative would be to require config to be passed on every "createXxx()" call -- this would prevent need to hold on to a reference (save 1 field) but would be much more intrusive as callers would need to be changed.
    So probably better to add 2 fields.

  3. cowtowncoder commented on Oct 23, 2022

    @cowtowncoder
    MemberAuthor

    Will not happen for 2.14 but should be the first thing for 2.15.

  4. cowtowncoder commented on Feb 14, 2023

    @cowtowncoder
    MemberAuthor
  5. pjfanning commented on Feb 22, 2023

    @pjfanning
    Member

    @cowtowncoder are there any more of these that you want to do for 2.15?

  6. pjfanning commented on Feb 22, 2023

    @pjfanning
    Member

    The total size could readily be limited using stream wrappers like those in https://github.com/pjfanning/json-size-limiter/tree/main/src/main/java/com/github/pjfanning/json/util . That code is borrowed. It should also be possible to find ByteBuffer equivalents. There is an argument that many web service frameworks have support for limiting the acceptable size of inputs - and that users should prefer those approaches. The earlier that the large input size is spotted, the better

  7. cowtowncoder commented on Feb 22, 2023

    @cowtowncoder
    MemberAuthor

    Yes, ideally we'd be able to target (1) by JsonStreamContext, and (2) for input and output sizes.
    But I don't know how timing would work.

    I think existence of stream wrappers is useful and can work for now even if eventually in-core limits were added.

    But deeper nesting checks should definitely be added in core at some point. Just not sure if timing works for 2.15.

    And field names... that's an interesting thing. Not commonly reported as attack but conceivably it would be vector same way (... and more) as max String token length.

  8. pjfanning commented on Feb 23, 2023

    @pjfanning
    Member

    It seems like (2) is something that can be left to users. I think it is reasonable for jackson to concentrate on the cases where comparatively fewer bytes in input can lead to bad outcomes.

  9. cowtowncoder commented on Feb 24, 2023

    @cowtowncoder
    MemberAuthor

    @pjfanning It could be, altough I implemented this with Woodstox few years back and it really is rather easy to add. Then again Jackson-core supports decorators so it's... even easier to wrap by users.

    But I agree it's not a high priority at this point.

  10. cowtowncoder commented on Jun 12, 2023

    @cowtowncoder
    MemberAuthor

    Filed #1046 for max document length -- I think it is worth adding, after thinking it through. Hope to work on it for 2.16.

  11. changed the title [-]Add configurable "document/processing limits" for JSON parser[/-] [+]Add configurable processing limits for JSON parser (`StreamReadConstraints`)[/+] on Jun 12, 2023
  12. cowtowncoder commented on Oct 26, 2023

    @cowtowncoder
    MemberAuthor

    Out of 6 ideas, 5 now implemented and included in 2.16.0-rc1; no plans yet for last one (max # of properties per Object); can file a new issue if need be.

    Closing this issue as completed.

  13. added this to the 2.16.0-rc1 milestone on Oct 26, 2023
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    performanceIssue related to performance problems or enhancements

    Type

    No type

    Projects

    No projects

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions