Reading & Writing Files

Knowing open(), .read(), and .write() is the starting point, but real programs need a few more practical patterns: reading huge files without running out of memory, understanding the difference between appending and overwriting, handling text encoding correctly across operating systems, and working with binary data. This page walks through each of these, then ties them together in a complete worked example.

Reading Line-by-Line with `for line in f:`

.readlines() reads the entire file into memory and returns every line as a list before you can process any of it. For a file that's a few kilobytes, that's fine. For a multi-gigabyte log file, it can exhaust your available memory. The fix is to iterate over the file object directly — Python reads one line at a time under the hood, so memory usage stays flat no matter how large the file is.

Python

# Loads the whole file into a list first — avoid for large files
with open("access.log", "r", encoding="utf-8") as f:
    lines = f.readlines()
    for line in lines:
        process(line)

# Streams one line at a time — memory usage stays constant
with open("access.log", "r", encoding="utf-8") as f:
    for line in f:
        process(line)

Tip

Both versions produce the same result for small files, but the second form scales to files far larger than your available RAM, since only one line needs to exist in memory at any given moment.

Appending vs. Overwriting

Mode 'w' truncates the file the instant it's opened — any existing content is gone immediately, even before you write anything new. Mode 'a' instead moves to the end of the file and adds new content after what's already there. Mixing these up is a common source of accidentally-deleted data.

Python

with open("log.txt", "w", encoding="utf-8") as f:
    f.write("Line 1\n")
    f.write("Line 2\n")

# 'w' again: the file is wiped out and starts fresh
with open("log.txt", "w", encoding="utf-8") as f:
    f.write("Line A\n")

with open("log.txt", "r", encoding="utf-8") as f:
    print("After overwrite:")
    print(f.read())

# 'a': new content is added after the existing content
with open("log.txt", "a", encoding="utf-8") as f:
    f.write("Line B\n")

with open("log.txt", "r", encoding="utf-8") as f:
    print("After append:")
    print(f.read())

After overwrite:
Line A

After append:
Line A
Line B

Warning

Opening a file with `'w'` when you meant `'a'` will silently destroy the file's previous contents with no warning or confirmation. When in doubt about which mode to use, check whether the file already exists first, or use `'x'` to force an error if it does.

Always Pass `encoding='utf-8'` Explicitly

When you don't pass an encoding argument, Python falls back to the operating system's locale-preferred encoding to decide how bytes on disk map to text characters. That default is not the same everywhere.

Platform	Typical default text encoding
Windows (English/Western)	`cp1252` (or another codepage depending on locale)
Linux	`utf-8`
macOS	`utf-8`

Warning

A file written on Windows without an explicit encoding may use `cp1252`, while the same code run on Linux or macOS writes `utf-8`. If that file contains non-ASCII characters (accented letters, emoji, curly quotes, currency symbols), opening it on a different platform — or even a differently configured machine — can raise `UnicodeDecodeError` or silently produce garbled text. Always pass `encoding='utf-8'` explicitly to `open()` so behavior is identical everywhere.

Python

# Portable and predictable on every platform
with open("notes.txt", "w", encoding="utf-8") as f:
    f.write("café, naïve, 日本語\n")

with open("notes.txt", "r", encoding="utf-8") as f:
    print(f.read())

café, naïve, 日本語

Binary Files

Text mode assumes the file contains characters and decodes bytes accordingly. Some files — images, audio, PDFs, ZIP archives — aren't text at all, and decoding them as text would corrupt the data. For these, open the file in binary mode ('rb', 'wb') and work with bytes objects instead of str. A binary copy of a file is a common example: read raw bytes in, write the same raw bytes out.

Python

with open("photo.jpg", "rb") as source:
    data = source.read()

with open("photo_copy.jpg", "wb") as destination:
    destination.write(data)

print(f"Copied {len(data)} bytes")

Copied 482113 bytes

Note

Never pass `encoding` when opening a file in binary mode — binary mode deals in raw `bytes`, and there's no text to decode.

Worked Example: Filtering a Log File

Here's a complete example that combines line-by-line reading, explicit encoding, and writing: it reads a log file, keeps only the lines that contain "ERROR", and writes just those lines to a new file — all without ever loading the full input file into memory at once.

Python

input_path = "app.log"
output_path = "errors_only.log"

error_count = 0

with open(input_path, "r", encoding="utf-8") as infile, open(
    output_path, "w", encoding="utf-8"
) as outfile:
    for line in infile:
        if "ERROR" in line:
            outfile.write(line)
            error_count += 1

print(f"Wrote {error_count} error lines to {output_path}")

Wrote 3 error lines to errors_only.log

Tip

Opening two files in a single `with` statement (separated by a comma) is perfectly valid and keeps both of them properly closed — even if one line of processing raises an exception partway through the loop.

for line in f: streams a file one line at a time and scales to files of any size.
'w' truncates immediately on open; 'a' preserves existing content and appends.
Always pass encoding='utf-8' explicitly — the OS default is not guaranteed and varies between Windows and Linux/macOS.
Use binary mode ('rb'/'wb') for non-text files like images, and never combine it with encoding.