Repository navigation
Escape params 2x faster with half the allocations, using CGI.escape - #1727
Conversation
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
There was a problem hiding this comment.
🟡 Changes recommended
The fast path changes output for valid Shift_JIS characters, breaking the intended byte-for-byte compatibility.
1 open issue
What changed in this PR
This PR optimizes Faraday’s shared parameter escaping with CGI.escape, reducing work in both flat and nested query encoding while aiming to preserve existing behavior.
Changes:
- Adds a CGI fast path with a regexp fallback for unsupported inputs.
- Adds utility specs covering escaping, encodings, conversion, and error behavior.
- Adds escaping regression coverage for both parameter encoders.
| File | Description |
|---|---|
spec/faraday/utils_spec.rb |
Adds escaping compatibility and edge-case specs. |
spec/faraday/params_encoders/nested_spec.rb |
Tests escaping nested keys and values. |
spec/faraday/params_encoders/flat_spec.rb |
Tests escaping flat keys and values. |
lib/faraday/utils.rb |
Introduces the CGI escaping fast path and regexp fallback. |
Review effort: Balanced
| str = str.to_s | ||
| # CGI.escape needs a String and works on raw bytes: anything else, including | ||
| # strings ESCAPE_RE can't scan (and raises on), keeps the regexp path. | ||
| if str.is_a?(String) && str.valid_encoding? && str.encoding.ascii_compatible? |
There was a problem hiding this comment.
I was surprised at first, but it looks like Encoding::Shift_JIS#ascii_compatible? does indeed resolve to true 🤔, I guess that means it's a bit too permissive. But limiting to ASCII_8BIT and US_ASCII feels a bit too restrictive.
@czj I'm no encoding expert to be honest, so I'll defer to you. Would limiting this to ASCII_8BIT and US_ASCII take away most of the performance gains, or simply work the same in the vast majority of requests?
There was a problem hiding this comment.
Good catch, and it goes beyond Shift_JIS: Big5, GBK, GB18030, CP949 and their variants can all end a multibyte character with an ASCII byte, which CGI.escape works byte by byte and leaves alone.
@iMacTia to your question: limiting it to ASCII-8BIT and US-ASCII would take away almost all of the gain, because string literals, JSON and Rails params are UTF-8. On 16 typical params, the old code takes 0.83 s, this PR 0.37 s, and ASCII-8BIT + US-ASCII only 0.85 s.
UTF-8 is safe by design, though: every byte of a multibyte character is ≥ 0x80, so escaping bytes or characters always gives the same output. So 84cfd4b keeps the fast path for UTF-8, US-ASCII and BINARY, and every other encoding goes back to the regexp. I then compared both implementations on 6.2M strings in every encoding Ruby knows, invalid byte sequences and UTF-16 included: same result, encoding, class and exception every time. The ア and 許 cases are now in utils_spec.rb.
Thanks a lot for the kind words and the review!
|
Thank you @czj, this code sits on the hot path, so any performance gain is always appreciated and the strategy of optimising for the majority of cases while keeping the slower fallback for the others makes a lot of sense. I've left a couple of comments asking for clarification, please forgive my encoding ignorance 😄 |
In Shift_JIS, Big5, GBK, GB18030 and CP949, a multibyte character can end with an ASCII byte: 'ア' is "\x83A" in Shift_JIS. CGI.escape works on bytes and leaves that 'A' alone, while ESCAPE_RE escapes the whole character (%83A vs %83%41). Every other encoding keeps the regexp path. The fast path also takes String instances only: CGI.escape returns a String subclass such as ActiveSupport::SafeBuffer as is when nothing needs escaping, where the regexp path returned a plain String. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
|
Thanks for the merge @iMacTia ! |

Description
Utils.escaperuns for every key and value encoded byNestedParamsEncoderandFlatParamsEncoder. It does a blockgsub, then anunpack/join/upcaseper escaped character, then a secondgsubfor spaces.CGI.escapeescapes exactly the same unreserved set (a-zA-Z0-9 . ~ _ -, space as+) in C. Faraday already loadscgi/escapeforCGI.unescape(#1623), andCGI.escapehas left~alone since Ruby 2.5.CGI.escapeworks on bytes, so the fast path is limited to UTF-8, US-ASCII and BINARY: in Shift_JIS, Big5, GBK or CP949, a multibyte character can end with an ASCII byte ('ア'is"\x83A"), whichESCAPE_REescapes andCGI.escapedoesn't. It also takes plainStringinstances only:CGI.escapereturns aSafeBufferas is when nothing needs escaping, and rejects objects whoseto_sisn't aString(the suite'sFakeSafeBuffer). Invalid byte sequences and UTF-16 strings, whichCGI.escapewould escape whereESCAPE_REraises, keep the regexp path too. So every input behaves exactly as before, andESCAPE_REand the public API are unchanged.Measured on Ruby 4.0.9 with benchmark-ips and memory_profiler,
main(d494fce) against this branch, average of two runs:Utils.escape, 16 wordsNestedParamsEncoder.encode, 8 paramsNestedParamsEncoder.encode, nested hash with arraysFlatParamsEncoder.encode, 8 paramsBenchmark script (
ruby -Ilib bench.rb)Todos
utils_speccompare the output with the regexp implementation byte for byte, and cover reserved and unreserved characters, multibyte UTF-8, other encodings (including Shift_JIS and GBK characters with an ASCII trailing byte, thanks Copilot), result encoding, String subclasses, a literal+versus a space,default_space_encoding = '%20', non-String values, and the raise on invalid bytes and UTF-16. One spec each innested_specandflat_spec. I also compared both implementations on 6.2M strings, every byte and two-byte sequence in every encoding Ruby knows, invalid ones included: identical result, encoding, class and exception.bundle exec rspec: 689 unit examples, 0 failures. RuboCop: no offenses.Additional Notes
If you'd rather call
CGI.escapedirectly and accept escaping invalid strings instead of raising, the fallback branch can go away.🤖 Generated with Claude Code, but prompted, read, and verified by @czj, a caring human :)