UnicodeDecodeError: 'utf-8' codec can't decode byte 0xff in position 0: invalid start byte

The exact error string

with open('export.csv') as f:        # defaults to encoding='utf-8'
    data = f.read()

Traceback (most recent call last):
  File "app.py", line 2, in <module>
    data = f.read()
UnicodeDecodeError: 'utf-8' codec can't decode byte 0xff in position 0: invalid start byte

The same error appears with other bytes — 0x80, 0x92 (a Windows "smart quote"), 0xe9 (é in Latin-1) — and a different position. The byte and position are the clue to which cause you have. Note the headline case specifically — 0xff at position 0 — is almost always a UTF-16 byte-order mark, so jump to Cause 2 (BOM); a bad byte mid-file points instead to a wrong encoding like Windows-1252 (Cause 1).

Why UTF-8 decoding fails

Python 3 opens files in text mode as UTF-8 by default. UTF-8 has strict rules about which byte values can start a character, so any file that is not UTF-8 will eventually hit a byte that is illegal as a "start byte" and raise UnicodeDecodeError. The decoder is correct — the assumption that the file is UTF-8 is wrong.

Cause 1: the file is Windows-1252 / Latin-1

Spreadsheets exported from Excel, and a lot of legacy data, are Windows-1252 (a.k.a. cp1252) or ISO-8859-1 (Latin-1). Tell Python the real encoding:

# Western European text from Excel:
with open('export.csv', encoding='cp1252') as f:
    data = f.read()

# Latin-1 never raises (it maps all 256 byte values) — handy, but it will
# produce WRONG characters if the file is actually some other encoding:
with open('export.csv', encoding='latin-1') as f:
    data = f.read()

Cause 2: a byte-order mark (BOM)

If the failing byte is at position 0, suspect a BOM. 0xff 0xfe means the file is UTF-16; 0xef 0xbb 0xbf is a UTF-8 BOM that the plain UTF-8 codec leaves as a stray character:

# 0xff 0xfe at position 0  ->  the file is UTF-16
with open('data.txt', encoding='utf-16') as f:
    data = f.read()

# 0xef 0xbb 0xbf  ->  UTF-8 with BOM; utf-8-sig strips it
with open('data.csv', encoding='utf-8-sig') as f:
    data = f.read()

Cause 3: reading a CSV with pandas

The same fix applies to pandas.read_csv, which also defaults to UTF-8 — pass the encoding the file actually uses (this is by far the most common place the error shows up, since CSV files from Excel are rarely UTF-8):

import pandas as pd

df = pd.read_csv('export.csv', encoding='cp1252')   # or 'latin-1', 'utf-16'

Need the CSV as JSON for an API or config? Once it loads cleanly, convert it in your browser with CSV to JSON — no upload, no server.

Cause 4: the file is binary, not text

If the data is an image, a zip, a Parquet file, or a pickle, it should never be decoded as text. Open it in binary mode and work with bytes:

with open('photo.png', 'rb') as f:   # 'rb' = read binary, no decoding
    raw = f.read()                   # raw is bytes, not str

# To move binary safely through JSON/text, base64-encode it:
import base64
b64 = base64.b64encode(raw).decode('ascii')

To eyeball or sanity-check a base64 value by hand, paste it into the Base64 encoder / decoder. Storing bytes in JSON this way is also the standard answer to the related Python error Object of type bytes is not JSON serializable.

Detecting an unknown encoding

When you genuinely don't know the encoding, detect it — then verify the result, because detection is a heuristic:

# pip install charset-normalizer
from charset_normalizer import from_path

best = from_path('mystery.csv').best()
print(best.encoding)                 # e.g. 'cp1252'
text = str(best)                     # decoded with the detected encoding

A note on errors='ignore' / 'replace'

You will see advice to add errors='replace' or errors='ignore'. These make the exception go away by substituting or dropping the bad bytes — which silently corrupts or loses data. Use them only for best-effort display of already-broken input; for data you care about, find the correct encoding instead.

Debugging checklist

• ✓ Read the failing byte and position — position 0 usually means a BOM
• ✓ 0xff 0xfe at start → encoding='utf-16'; 0xef 0xbb 0xbf → 'utf-8-sig'
• ✓ Excel/legacy text → try encoding='cp1252' (or 'latin-1' as a never-fail fallback)
• ✓ Using pandas? Pass encoding= to read_csv
• ✓ Binary file? Open with 'rb' and don't decode; base64 it to put in JSON
• ✓ Unknown? Detect with charset-normalizer, then verify the text
• ✓ Avoid errors='ignore'/'replace' for data you need intact