Skip to content

Add client-side opaque-token log scrubber - #6119

Merged
Nilesh Choudhary (4gust) merged 11 commits into
mainfrom
4gust-opaque-token-log-scrubber
Jul 23, 2026
Merged

Nilesh Choudhary (4gust) merged 11 commits into
mainfrom
4gust-opaque-token-log-scrubber

Conversation

@4gust

@4gust Nilesh Choudhary (4gust) commented Jul 14, 2026 •

Copy link
Copy Markdown
Contributor

Summary

Redacts HIT-tagged (Highly Identifiable Token) opaque tokens from MSAL''s own log output before they reach any logging sink. eSTS/CCS now tag outgoing opaque tokens with a marker inside the base64/base64url-encoded value; this change detects those markers and replaces the whole token run with [Redacted opaque token].

Per maintainer feedback, scrubbing is applied as a tactical fix scoped to exception log messages only (which carry ESTS token-call request/response/error bodies via MsalServiceException), rather than globally at every log line.

Changes

  • New src/client/Microsoft.Identity.Client/Internal/Logger/TokenScrubber.cs β€” internal static class TokenScrubber with public static string Scrub(string).
    • Detection patterns (ordinal, case-sensitive): the three base64 renderings of EvoStsArtifacts at each byte-offset alignment (RXZvU3RzQXJ0aWZhY3Rz, V2b1N0c0FydGlmYWN0c, dm9TdHNBcnRpZmFjdH) plus the MSA literal MsaArtifacts.
    • Fast path: returns the same string reference with no allocation when no pattern is present.
    • On a match, expands both left and right over the token charset [A-Za-z0-9+/_-=] (base64 + base64url + = padding) to cover the whole token run, then replaces it via StringBuilder.
    • Kill switch: AppContext switch Microsoft.Identity.Client.DisableOpaqueTokenScrubbing, read once into a cached static bool. Independent of PII logging.
  • Modified src/client/Microsoft.Identity.Client/Internal/Logger/MsalLoggerExtensions.cs β€” the exception-logging overloads (ErrorPii(Exception), ErrorPiiWithPrefix, WarningPii(Exception), WarningPiiWithPrefix, InfoPii(Exception)) now scrub the exception text via TokenScrubber.Scrub(...). The *WithPrefix overloads scrub only the exception message (prefix appended afterward) and tolerate a null exception. This is the single tactical choke point for ESTS token-call error bodies, which surface as MsalServiceException text.
  • Unchanged LoggerHelper.FormatLogMessage β€” no global scrubbing; only exception paths are scrubbed.
  • New tests/Microsoft.Identity.Test.Unit/PublicApiTests/TokenScrubberTests.cs β€” unit tests covering the three eSTS offsets, the MSA literal, a token inside JSON, a real Set-Cookie: esctx-... line, a negative same-reference case, and a look-alike that must not be redacted.
  • New tests in tests/Microsoft.Identity.Test.Unit/PublicApiTests/LoggerTests.cs β€” integration tests confirming tagged tokens in logged exceptions are scrubbed and that plain (non-exception) messages are left untouched.

Design note: scope of scrubbing

Two gates keep this tight:

  1. Only the exception-logging overloads call Scrub. eSTS token-call errors surface as MsalServiceException, whose text (including ResponseBody) is logged through these overloads. eSTS success bodies (real tokens) are only deserialized, never logged.
  2. Scrub only rewrites a string that contains a HIT-tag pattern; any other content is returned as the same reference (no-op).

Testing

  • Built Microsoft.Identity.Client (net8.0): succeeded, 0 errors.
  • TokenScrubberTests + LoggerTests (net48): 48 passed, 0 failed, 0 skipped.

Redact HIT-tagged opaque tokens from MSAL log output at the FormatLogMessage choke point. Adds TokenScrubber with an AppContext kill switch and unit tests.

Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
Copilot AI review requested due to automatic review settings July 14, 2026 13:43

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

Adds a client-side log scrubber that redacts HIT-tagged opaque tokens (eSTS/CCS marker encodings + MSA literal) from MSAL’s formatted log output before it reaches any logging sink, with a cached AppContext kill switch.

Changes:

  • Introduces TokenScrubber that detects HIT markers, expands to the full base64/base64url token run, and replaces with [Redacted opaque token] (fast-path returns same string reference when no marker is present).
  • Hooks scrubbing into the centralized LoggerHelper.FormatLogMessage choke point to cover all logger implementations.
  • Adds unit tests covering multiple marker alignments, MSA literal, JSON embedding, Set-Cookie examples, negative cases, and multiple tokens.

Reviewed changes

Copilot reviewed 3 out of 3 changed files in this pull request and generated no comments.

File Description
src/client/Microsoft.Identity.Client/Internal/Logger/TokenScrubber.cs Implements HIT-marker detection and whole-token redaction with a cached AppContext disable switch and no-allocation fast path.
src/client/Microsoft.Identity.Client/Internal/Logger/LoggerHelper.cs Routes formatted log messages through TokenScrubber.Scrub(...) so redaction occurs at a single central log-formatting point.
tests/Microsoft.Identity.Test.Unit/PublicApiTests/TokenScrubberTests.cs Adds coverage for redaction behavior across multiple marker variants and representative log-text embeddings.

Covers token at start/end, whole-string token, base64url expansion across '=', Bearer header, multiple Set-Cookie tokens on one line, adjacent tokens, mixed eSTS+MSA, HTTP response error blob, null/empty inputs.

Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
Copilot AI review requested due to automatic review settings July 14, 2026 16:03

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

Copilot reviewed 3 out of 3 changed files in this pull request and generated 1 comment.

Addresses PR review: Scrub() previously scanned for patterns twice on the redaction path (ContainsAnyPattern then FindEarliestMatch from index 0). Now performs a single FindEarliestMatch; the no-match fast path still returns the same reference with no allocation.

Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
Copilot AI review requested due to automatic review settings July 14, 2026 16:46

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

Copilot reviewed 3 out of 3 changed files in this pull request and generated no new comments.

Copilot AI review requested due to automatic review settings July 15, 2026 08:58

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

Copilot reviewed 3 out of 3 changed files in this pull request and generated no new comments.

@4gust
Nilesh Choudhary (4gust) marked this pull request as ready for review July 15, 2026 09:09
@4gust
Nilesh Choudhary (4gust) requested a review from a team as a code owner July 15, 2026 09:09
Per review feedback, replace the global FormatLogMessage scrub with a tactical fix: scrub only exception log messages (which carry ESTS request/response/error bodies via MsalServiceException) in the MsalLoggerExtensions exception overloads. Ordinary log lines are no longer scrubbed. TokenScrubber internals are unchanged. Adds LoggerTests integration tests proving exception messages are scrubbed and plain messages are not.

Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>

Copilot-Session: f5fe65a9-3405-47c0-ab3f-8a9a3233af3b
Copilot AI review requested due to automatic review settings July 15, 2026 10:39

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

Copilot reviewed 5 out of 5 changed files in this pull request and generated 2 comments.

Address review feedback to run the scrubber on just the message content rather than the whole formatted line. ErrorPiiWithPrefix/WarningPiiWithPrefix now scrub exWithPii.ToString() and append the prefix afterward, matching the other exception overloads.

Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>

Copilot-Session: f5fe65a9-3405-47c0-ab3f-8a9a3233af3b
Copilot AI review requested due to automatic review settings July 15, 2026 14:07

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

Copilot reviewed 5 out of 5 changed files in this pull request and generated 3 comments.

Copilot AI review requested due to automatic review settings July 15, 2026 14:29

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

Copilot reviewed 5 out of 5 changed files in this pull request and generated 4 comments.

Use exWithPii?.ToString() so ErrorPiiWithPrefix/WarningPiiWithPrefix do not
throw when the exception is null, matching the pre-scrubber behavior.

Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
Copilot-Session: f5fe65a9-3405-47c0-ab3f-8a9a3233af3b
Copilot AI review requested due to automatic review settings July 15, 2026 14:47

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

Copilot reviewed 5 out of 5 changed files in this pull request and generated 5 comments.

Add an early IsLoggingEnabled check to the exception-logging overloads so
exWithPii?.ToString() and TokenScrubber.Scrub(...) only run when the target
log level is enabled, avoiding O(n) scans of large exception text on disabled
paths. Also normalize the remaining .ToString() calls to ?.ToString() for
uniform null tolerance. Behavior-preserving: both logger adapters already gate
their Log body on IsLoggingEnabled.

Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
Copilot-Session: f5fe65a9-3405-47c0-ab3f-8a9a3233af3b
Copilot AI review requested due to automatic review settings July 15, 2026 15:06

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

Copilot reviewed 5 out of 5 changed files in this pull request and generated no new comments.

Copilot AI review requested due to automatic review settings July 15, 2026 16:00

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

Copilot reviewed 5 out of 5 changed files in this pull request and generated no new comments.

Copilot AI review requested due to automatic review settings July 23, 2026 12:37

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

Copilot reviewed 5 out of 5 changed files in this pull request and generated no new comments.

@4gust
Nilesh Choudhary (4gust) merged commit 06ae3da into main Jul 23, 2026
17 checks passed
@4gust
Nilesh Choudhary (4gust) deleted the 4gust-opaque-token-log-scrubber branch July 23, 2026 15:09
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

4 participants