Repository navigation
Normalize search field names case-insensitively in SearchScheme - #14154
Conversation
β¦CC) in WorkSearchScheme
|
Thanks for the PR, @wanjinhao1! The linked issue hasn't been triaged yet β triage happens on Mondays and Fridays. There are currently 120 open non-draft PRs ahead of yours. PR triage checklist (maintainers / Pierre)
Note This comment was automatically generated by Pierre, Open Library's PR first-touch assistant, on behalf of @mekarpeles. This is NOT a code review or sign-off. |
Field names were validated case-insensitively but never rewritten, so cased fields (ISBN:, LCC:, ID_GOODREADS:, Edition.ISBN:, NAME:) reached Solr unchanged: works search 500'd and author/subject/edition search silently returned nothing. Add SearchScheme.canonical_field_name(), which matches a field case-insensitively, resolves unprefixed aliases and applies the scheme's own is_search_field. process_user_query now uses it both to validate fields and to rename them after parsing, so every scheme sends Solr canonical names. Alias resolution moves out of WorkSearchScheme.transform_user_query, which supersedes the earlier works-only normalization. EditionSearchScheme's publishers -> publisher alias now applies on /search/editions.json.
RayBB
left a comment
There was a problem hiding this comment.
Thanks @wanjinhao1, great find, and the clear write-up made this easy to dig into!
Your fix worked for ISBN:/LCC:/DDC: in works search. While testing it, I found the same bug one level up: SearchScheme.process_user_query checks field names case-insensitively but never rewrites them. That meant author, subject, list and edition search had it too: /search/authors.json?q=NAME:Twain finds 0 authors on production, while name:Twain finds 128.
I pushed two commits on top of yours:
- a merge of master;
- a new
SearchScheme.canonical_field_name(), which both validates and renames fields for every search type. It replaces yourelifinworks.py. I kept your test rows and added tests for the other schemes.
I've also updated the title and description to match. Verified on testing: the cased /search pages that 500 on production now load, and NAME:/ISBN: style queries return the same results as lowercase across works, authors, subjects, lists and editions. Thanks again!
Closes #14142
Fix. Search fields were checked case-insensitively but sent to Solr with the case the user typed. As a result:
/search?q=ISBN:β¦,LCC:β¦,DDC:β¦,ID_GOODREADS:β¦,Edition.ISBN:β¦andWork.Title:β¦returned 500 (ValueError: Unknown fieldinconvert_work_field_to_edition_field)./search.jsonreturned no results for those same queries./search/authors.json?q=NAME:Twainfinds 0 authors, whilename:Twainfinds 128.Technical
SearchScheme.canonical_field_name(field). It matches a field case-insensitively, resolves unprefixed aliases fromfield_name_map, and applies the scheme's ownis_search_field. It returnsNonefor unknown fields.process_user_queryusescanonical_field_nameboth to decide which fields are valid and to rename them after parsing. Every scheme (works, editions, authors, subjects, lists) now sends Solr canonical, lowercase names.WorkSearchScheme.transform_user_query. It now only does the ISBN/LCC/DDC transforms, on names that are already canonical. This replaces the first commit's works-only normalization./search/editions.json: the edition scheme'spublishers β publisheralias now applies.publishers:used to match nothing; now it matches.work.author) are still not resolved, as on master.Testing
QUERY_PARSER_TESTScover uppercase ISBN/LCC/DDC, aliases, negated and grouped fields, LCC ranges,id_*,edition./work.prefixes, unknown fields and*:*. The newschemes/tests/test_scheme.pycoverscanonical_field_nameand every scheme. The new rows fail on master./search?q=ISBN:9780060751043/search?q=LCC:PS1303*/search?q=DDC:818.402/search?q=ID_GOODREADS:123/search?q=Edition.ISBN:9780060751043/search/authors.json?q=NAME:β¦/search/subjects.json?q=NAME:β¦/search/editions.json?q=publishers:β¦Lowercase, alias, plain-text and malformed queries are unchanged.
Screenshot
No UI changes.
Stakeholders
@RayBB @cdrini
Attribution disclaimer: substantial parts of this change were developed with AI coding assistance; the approach follows the analysis in issue #14142.