[rchl]

Fixes #2219

get_line() reads lines of files that aren't open in a view through linecache, which decodes as strict UTF-8. On Sublime Text's Python 3.8 plugin host, a file in another encoding (GBK in the issue, or Latin-1, CP1250, ...) raises UnicodeDecodeError. This breaks Find References in the output panel and the rename / workspace edit preview.

Changes:

• Replace get_line() with get_file_lines(window, file_name, rows, strip), which returns the lines of one file at the given rows, in the same order. Lines come from the buffer when the file is open, else from the file on disk, which is read once per call. The references panel and the workspace edit preview call it once per file.
• Files on disk are read with encoding='utf-8-sig', errors='replace'. Invalid bytes show as U+FFFD instead of failing, and the rest of the line is shown correctly. The encoding isn't guessed, since a wrong guess (e.g. Latin-1 for a CP1250 file) would show wrong characters.
• Drop the global linecache, which the references panel cleared wholesale (including entries of other plugins) and the workspace edit preview never cleared, so it could show stale lines from files changed on disk.
• Add a test reading lines from a Latin-1 file.

Assisted-By: Claude Code:claude-opus-5-5

[rchl]

The main reason that linecache is not used is that it's a global cache that can be used by others too (by python's traceback at least) so it's a bit heavy handed to just clear it from LSP.

[rwols]

Why did you choose to use utf-8-sig instead of utf-8?

On this page about the python codecs: https://docs.python.org/3/library/codecs.html

I can read:

Without external information it’s impossible to reliably determine which encoding was used for encoding a string. Each charmap encoding can decode any random byte sequence. However that’s not possible with UTF-8, as UTF-8 byte sequences have a structure that doesn’t allow arbitrary byte sequences. To increase the reliability with which a UTF-8 encoding can be detected, Microsoft invented a variant of UTF-8 (that Python calls "utf-8-sig") for its Notepad program: Before any of the Unicode characters is written to the file, a UTF-8 encoded BOM (which looks like this as a byte sequence: 0xef, 0xbb, 0xbf) is written. As it’s rather improbable that any charmap encoded file starts with these byte values, this increases the probability that a utf-8-sig encoding can be correctly guessed from the byte sequence. So here the BOM is not used to be able to determine the byte order used for generating the byte sequence, but as a signature that helps in guessing the encoding.

So. OK. Using utf-8-sig is more lenient than using utf-8. Because it would consume an optional byte-order-marker (BOM).

Note that all of this does not actually solve the issue of Chinese encodings like GBK. But these changes are a good start to show ??? everywhere for GBK encodings.

[rwols]

In the ideal case, we would be able to detect the encoding using some new ST API function like sublime.detect_encoding(file_path)...

[rchl]

Yes, it's about removing BOM.
It also matches the old behaviour. The linecache code reads files through tokenize.open, which also removes a byte order mark.

[predragnikolic]

We can get the current file encoding with view.encoding()

ST doesn't provide a way to get the encoding of a non open file, but we could reuse the encoding of the current opened view/file who made the request.