Files Streams

Master efficient file handling, streaming patterns, and processing massive datasets that don't fit in memory.

Learn Python, JavaScript, Java and more with free interactive lessons, real projects and AI-powered help. Beginner-friendly.

Part of the free Python course at LearnCodingFast — hands-on lessons with examples you run in your browser, plus practice exercises and a quick quiz.

This lesson teaches professional data handling techniques used in:

1. Python's File I/O Model (Text vs Binary, Buffered vs Unbuffered)

When you open a file, Python uses a layered I/O system:

Buffering matters: Python reads data in chunks internally to reduce syscalls, making it ideal for large datasets.

2. Reading Files the Right Way (Avoid .read() on Large Files)

This loads the entire file into memory, which is disastrous for large logs.

3. Using Buffered Streams Explicitly

For very large binary files (videos, models, audio):

4. Writing Files Efficiently (Avoid Tiny Writes)

5. Working With CSV Files at Scale

6. Handling JSON Without Crashing RAM

7. Working With Very Large Binary Files

8. Memory Mapping (mmap) — Fast Random Access to Huge Files

9. Chunk Processing for Massive Datasets

10. Using Iterators & Generators in Data Pipelines

11. Working With Compressed Data (ZIP, GZIP)

12. Handling Data That Doesn't Fit In RAM

13. Avoiding the Most Common File-I/O Mistakes

14. Best Practices Recap

1. Understanding Buffered I/O Depth

2. Random Access With Seek

3. Streaming API Data

4. Chunked CSV with Pandas

5. Producer/Consumer with Backpressure

6. Live Log Tailing

7. Temporary Files

8. Multiprocessing for Parallel Files

9. Binary Struct Parsing

10. Composable Pipeline Functions

11. Avoiding Common Streaming Pitfalls

1. Streaming Compressed Files

2. Zero-Copy with Memoryview

3. Async File I/O

4. Multiprocessing with imap

5. Rolling Window Processing

6. Endless Streaming Pipelines

7. Atomic File Writes

8. Common Pitfalls in Large-Dataset Work

🎓 Final Summary

You've now mastered professional file handling and large-scale data processing in Python.

📋 Quick Reference — Files & Streams

You can now handle files of any size efficiently, parse CSV/JSON/binary formats, and stream large datasets without memory issues.

Up next: Advanced Collections — master Counter, deque, ChainMap, itertools, and functools.

Practice quiz

Why avoid f.read() on a very large file?

  • It is slower to type
  • It corrupts the file
  • It loads the entire file into memory, which can exhaust RAM
  • It only reads the first line

Answer: It loads the entire file into memory, which can exhaust RAM. f.read() pulls the whole file into RAM at once — disastrous for multi-GB files. Stream line by line instead.

What is the memory-efficient way to process a huge text file line by line?

  • for line in f:
  • data = f.read().split()
  • rows = list(f)
  • f.readlines()

Answer: for line in f:. Iterating 'for line in f:' reads one line at a time, keeping memory usage tiny regardless of file size.

Why is 'with open(path) as f:' preferred for file handling?

  • It is faster to parse
  • It enables binary mode
  • It compresses the file
  • It automatically closes the file even if an error occurs

Answer: It automatically closes the file even if an error occurs. The with statement (context manager) guarantees the file is closed when the block exits, even on exceptions.

What does the file mode 'rb' mean?

  • Read backwards
  • Read binary
  • Replace bytes
  • Read buffered

Answer: Read binary. 'r' is read and 'b' is binary, so 'rb' opens a file for reading in binary mode — used for non-text data like videos.

Why should you NOT do 'rows = list(reader)' on a large CSV?

  • It loads every row into memory at once
  • list is not callable
  • csv.reader has no list support
  • It sorts the rows

Answer: It loads every row into memory at once. Wrapping the reader in list() materializes every row in RAM. Iterate the reader row by row to stay memory-efficient.

What advantage does mmap (memory mapping) give for huge files?

  • It compresses the file
  • It encrypts the data
  • Instant random access to any position with near-zero RAM usage
  • It sorts the bytes

Answer: Instant random access to any position with near-zero RAM usage. Memory mapping lets the OS provide fast random access to any offset without loading the whole file into RAM.

In the lesson, what does f.seek(116) do to a file object before f.read(6)?

  • Reads 116 bytes
  • Moves the read position to byte offset 116
  • Deletes 116 bytes
  • Skips 6 lines

Answer: Moves the read position to byte offset 116. seek(116) moves the file pointer to byte offset 116, so the following read(6) starts from there (reading 'TARGET' in the example).

Why use a bounded queue (Queue(maxsize=5)) in a producer/consumer pipeline?

  • To sort items
  • To run faster
  • To deduplicate items
  • To apply backpressure and limit memory use

Answer: To apply backpressure and limit memory use. A bounded queue blocks the producer when full, applying backpressure so memory stays under control.

What does a deque(maxlen=N) do as you append beyond N items?

  • Raises an error
  • Automatically drops items from the opposite end, keeping the last N
  • Ignores new items
  • Doubles in size

Answer: Automatically drops items from the opposite end, keeping the last N. A deque with maxlen automatically discards the oldest item when full — perfect for rolling/sliding-window processing.

What is the benefit of an atomic write (write to temp file, then os.replace)?

  • Faster writes
  • It compresses the output
  • It prevents file corruption by swapping in the complete file in one step
  • It appends instead of overwriting

Answer: It prevents file corruption by swapping in the complete file in one step. Writing to a temp file and atomically renaming means readers never see a half-written file — the replace is all-or-nothing.