Skip to content

Normalize search field names case-insensitively in SearchScheme - #14154

Merged
RayBB merged 4 commits into
internetarchive:masterfrom
wanjinhao1:agent/fix-14142-isbn-lcc-case
Nov 5, 2026
Merged

RayBB merged 4 commits into
internetarchive:masterfrom
wanjinhao1:agent/fix-14142-isbn-lcc-case

Conversation

@wanjinhao1

@wanjinhao1 wanjinhao1 commented Oct 28, 2026 •

Copy link
Copy Markdown
Contributor

Closes #14142

Fix. Search fields were checked case-insensitively but sent to Solr with the case the user typed. As a result:

  • /search?q=ISBN:…, LCC:…, DDC:…, ID_GOODREADS:…, Edition.ISBN:… and Work.Title:… returned 500 (ValueError: Unknown field in convert_work_field_to_edition_field).
  • /search.json returned no results for those same queries.
  • Author, subject and edition search silently returned nothing. For example, on production /search/authors.json?q=NAME:Twain finds 0 authors, while name:Twain finds 128.

Technical

  • New SearchScheme.canonical_field_name(field). It matches a field case-insensitively, resolves unprefixed aliases from field_name_map, and applies the scheme's own is_search_field. It returns None for unknown fields.
  • process_user_query uses canonical_field_name both to decide which fields are valid and to rename them after parsing. Every scheme (works, editions, authors, subjects, lists) now sends Solr canonical, lowercase names.
  • Alias resolution moves out of WorkSearchScheme.transform_user_query. It now only does the ISBN/LCC/DDC transforms, on names that are already canonical. This replaces the first commit's works-only normalization.
  • Behavior change on /search/editions.json: the edition scheme's publishers β†’ publisher alias now applies. publishers: used to match nothing; now it matches.
  • Unchanged: lowercase queries produce identical output. Unknown fields are still escaped to plain text. Prefixed aliases (work.author) are still not resolved, as on master.

Testing

  • New rows in QUERY_PARSER_TESTS cover uppercase ISBN/LCC/DDC, aliases, negated and grouped fields, LCC ranges, id_*, edition./work. prefixes, unknown fields and *:*. The new schemes/tests/test_scheme.py covers canonical_field_name and every scheme. The new rows fail on master.
  • A differential run of about 4,000 generated queries, in lowercase, upper and title case, through all five schemes: lowercase output matches master except the edition alias above, and every cased variant now matches its lowercase form.
  • Local stack, master vs this branch:
query master this branch
/search?q=ISBN:9780060751043 500 303 to the work
/search?q=LCC:PS1303* 500 200
/search?q=DDC:818.402 500 200
/search?q=ID_GOODREADS:123 500 200
/search?q=Edition.ISBN:9780060751043 500 200
/search/authors.json?q=NAME:… 0 results 1
/search/subjects.json?q=NAME:… 0 results 1
/search/editions.json?q=publishers:… 0 results 1

Lowercase, alias, plain-text and malformed queries are unchanged.

Screenshot

No UI changes.

Stakeholders

@RayBB @cdrini


Attribution disclaimer: substantial parts of this change were developed with AI coding assistance; the approach follows the analysis in issue #14142.

@openlibrary-bot

Copy link
Copy Markdown
Collaborator

Thanks for the PR, @wanjinhao1!

The linked issue hasn't been triaged yet β€” triage happens on Mondays and Fridays. There are currently 120 open non-draft PRs ahead of yours.

PR triage checklist (maintainers / Pierre)
  • PR description β€” not empty; explains what the change does and how to verify it
  • References an issue β€” PR body contains a #NNN reference
    • Linked issue is triaged β€” has a Priority: * label (not just Needs: Triage)
    • Linked issue is assigned β€” has at least one assignee
  • Commit history clean β€” no WIP/fixup/conflict noise; commit messages are meaningful
  • CI passing β€” no failing check-runs
  • Test cases present β€” if the change touches substantive logic, test coverage exists or is explained
  • Proof of testing β€” PR body includes a description of what was tested, a screenshot, or a video

Note

This comment was automatically generated by Pierre, Open Library's PR first-touch assistant, on behalf of @mekarpeles. This is NOT a code review or sign-off.

@github-project-automation github-project-automation Bot moved this to Waiting Review/Merge from Staff in Ray's Project Nov 3, 2026
RayBB added 2 commits November 4, 2026 14:24
Field names were validated case-insensitively but never rewritten, so
cased fields (ISBN:, LCC:, ID_GOODREADS:, Edition.ISBN:, NAME:) reached
Solr unchanged: works search 500'd and author/subject/edition search
silently returned nothing.

Add SearchScheme.canonical_field_name(), which matches a field
case-insensitively, resolves unprefixed aliases and applies the scheme's
own is_search_field. process_user_query now uses it both to validate
fields and to rename them after parsing, so every scheme sends Solr
canonical names. Alias resolution moves out of
WorkSearchScheme.transform_user_query, which supersedes the earlier
works-only normalization. EditionSearchScheme's publishers -> publisher
alias now applies on /search/editions.json.
@RayBB RayBB changed the title Normalize cased search field names (ISBN:/LCC:) in WorkSearchScheme Normalize search field names case-insensitively in SearchScheme Nov 4, 2026

@RayBB RayBB left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks @wanjinhao1, great find, and the clear write-up made this easy to dig into!

Your fix worked for ISBN:/LCC:/DDC: in works search. While testing it, I found the same bug one level up: SearchScheme.process_user_query checks field names case-insensitively but never rewrites them. That meant author, subject, list and edition search had it too: /search/authors.json?q=NAME:Twain finds 0 authors on production, while name:Twain finds 128.

I pushed two commits on top of yours:

  • a merge of master;
  • a new SearchScheme.canonical_field_name(), which both validates and renames fields for every search type. It replaces your elif in works.py. I kept your test rows and added tests for the other schemes.

I've also updated the title and description to match. Verified on testing: the cased /search pages that 500 on production now load, and NAME:/ISBN: style queries return the same results as lowercase across works, authors, subjects, lists and editions. Thanks again!

@RayBB
RayBB merged commit 634874e into internetarchive:master Nov 5, 2026
5 checks passed
@github-project-automation github-project-automation Bot moved this from Waiting Review/Merge from Staff to Done in Ray's Project Nov 5, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

Status: Done

Development

Successfully merging this pull request may close these issues.

Uppercase ISBN and LCC search fields are not normalized before Solr queries

3 participants