File Handling

Learning Objectives

  • Understand the necessity of file handling for persistent data storage.
  • Distinguish between text and binary file types.
  • Explain the File I/O lifecycle: Open, Read/Write, and Close.
  • Explain Python file opening modes (r, w, a, b) and absolute vs. relative paths.
  • Understand the concepts of serialization and deserialization (JSON, XML, Binary).
  • Understand the importance of file permissions and safe exception handling in file operations.
  • Learn about Context Managers for automatic resource cleanup.

A guide to file paths and streams, Python file modes, resource cleanup, buffering, serialization, and permissions, with language-specific behavior identified where it matters.

File Handling

File handling (or file I/O) is the process of creating, opening, reading, writing, and closing files through an operating system or runtime. Files commonly provide non-volatile storage, but writing or closing a file does not by itself guarantee that data has reached durable physical media; guarantees depend on the API, operating system, file system, and storage device.

1. The Need for File Handling

A program's ordinary in-memory state is usually lost when its process ends. To retain data across runs, programs commonly write it to a file or another persistent service. Persistence guarantees vary: a successful write may first reach runtime or operating-system buffers, and durability may require additional API calls and storage guarantees.

Primary File Types

  • Text formats (.txt, .csv, .json, .xml): These formats define textual characters and structure. On storage, they are still bytes encoded using a character encoding such as UTF-8. A text API decodes bytes into characters when reading and encodes characters when writing.

  • Binary formats (.bin, .dat, .jpg, .exe): These formats define how bytes represent data, often with application-specific fields. Binary files are also bytes on storage; they are not necessarily a copy of an in-memory layout, and their processing speed or size depends on the format, data, and task.

Text and binary formats both store bytes

Text mode applies character encoding and decoding. Binary mode exposes bytes for the program to interpret. A binary format is not automatically a memory dump, and neither mode alone determines size or speed.

The bytes 41 42 can be decoded as the text AB with UTF-8, while a binary format interprets bytes according to its own field rules.

2. The File I/O Lifecycle

A file API locates a resource using a path or another identifier. Absolute and relative paths are common for local files, but the current working directory, path rules, and available storage are platform-dependent.

File Paths

  • Absolute Path: The complete, unambiguous path from the root of the file system to the exact file (e.g., C:\Users\John\Documents\data.txt on Windows, or /home/john/documents/data.txt on Linux).

  • Relative Path: The path to the file relative to the directory where the program is currently running (the Current Working Directory). E.g., ../data/data.txt (meaning go up one folder, into data, and open data.txt).

File APIs commonly expose a lifecycle such as opening a resource, performing reads or writes, and releasing it. Exact behavior and terminology vary by language and operating system. A library may buffer operations or implement locking separately.

Stream

A stream is an API abstraction for sequential input or output. A file stream may expose bytes or decoded text; the API and mode determine which. Examples include C++ ifstream/ofstream and Python file objects.

Standard File I/O Steps

  1. Open: The program asks a runtime or operating system for a file resource using an API-specific mode. Access checks and locking behavior depend on the platform and API.
  2. Read/Write: The API transfers or buffers data between the program and the file, according to its text or binary behavior.
  3. Process: The program manipulates the data while it resides in memory.
  4. Close: The program releases the resource. Many buffered APIs flush their user-space buffers while closing, but closing does not universally guarantee physical-media durability or release a platform-wide lock.

2.1 File Opening Modes

Mode letters are defined by individual APIs. The table below describes Python's built-in open() modes and is not a universal cross-language mode table. Python defaults to text mode; append b for bytes and + for updating a stream that can both read and write.

Common Access Modes

  • Read (r): In Python, opens an existing file for reading and starts at the beginning. Opening fails if the file does not exist or cannot be accessed.

  • Write (w): In Python, creates a file if needed and truncates an existing file when it is opened.

  • Append (a): In Python, creates a file if needed and directs writes to the end, preserving existing content.

  • Binary (b): A Python mode modifier (for example, rb or wb) that makes reads and writes use bytes without text encoding or newline translation.

Python read, write, and append modes have different effects

These are Python built-in open() behaviors. The b marker changes the data interface, while r, w, and a control access and file-creation behavior. Other languages may use different mode names and rules.

Python r reads an existing file, w creates or truncates, a creates or adds at the end, and b changes text to byte handling.

The Buffer Concept

A runtime may buffer writes in user space; the operating system and storage device may also cache data. In Python, call f.flush() to push the file object's buffer toward the operating system, then call os.fsync(f.fileno()) while the file descriptor is still open to request synchronization. Closing a buffered file also flushes its object buffer, but write, flush, fsync, and close guarantees depend on the operating system and storage stack. A crash can lose data that remains in a volatile buffer.

The simulation below illustrates overwrite, append, and close as changes to a displayed document state.

File Operations

data.txt
CLOSED
Initial data...

Simulation scope

This component stores its sample text in React state inside the page. It does not open a disk file, exercise Python file modes, implement operating-system buffering, enforce permissions, or demonstrate durable storage.

Flush, synchronize, then close

For a Python buffered file, f.flush() sends buffered data toward the operating system; os.fsync(f.fileno()) requests synchronization while the descriptor is open. Close the file afterward. Device and platform behavior determine the final durability guarantee.

Application data moves from a Python file buffer through the operating-system cache toward a device; flush occurs before fsync, and the file closes after synchronization.

2.2 Context Managers

Resource-management constructs help ensure a file is released on both normal and exceptional exits. They handle cleanup; they do not guarantee that every write is durable.

Automatic Resource Cleanup

  • Python with statement: A file opened by with open("file.txt", "w") as f: is closed when the block exits, including when an exception propagates out of it. Closing a buffered file flushes its Python-level buffer, but is not a general durability guarantee.

  • Java try-with-resources: Resources that implement AutoCloseable are closed when the statement exits, including exceptional exits, subject to Java's resource-management rules.

Managed file resources close on normal and exceptional exits

Python with and Java try-with-resources invoke cleanup when control leaves the managed block. The figure shows cleanup only; application-specific error handling and durable-write guarantees remain separate concerns.

Both normal completion and an exception leaving a Python with block lead to the file resource being closed.

3. Serialization and Deserialization

To save a structured value for later use or transmission, a program encodes it in a format that another operation can decode. Writing a process-specific memory address does not recreate the referenced object in another run.

Serialization

Serialization converts data into a representation such as text or bytes for storage or transmission. Deserialization parses that representation into program data. A successful parse does not by itself establish that the data is valid, trusted, or safe for the application's intended use.

Common Serialization Formats

  • JSON (JavaScript Object Notation): A language-independent text format representing objects, arrays, strings, numbers, booleans, and null. For example, {"name": "Hero", "health": 100} is a JSON object. RFC 8259 specifies the format.

  • XML (eXtensible Markup Language): A markup format for structured documents, using elements and attributes (for example, <player health="100"/>). Its representation and capabilities differ from JSON; verbosity and processing cost depend on the document and use. W3C XML 1.0.

  • CSV (Comma-Separated Values): A text convention for rows and fields, often used for tabular data. Quoting, delimiters, line endings, and implementation details require a defined dialect; commas in a field are commonly quoted. RFC 4180 documents a common format.

  • Binary encodings: Formats such as Protocol Buffers encode structured values as bytes, typically with a schema. Python's Pickle is a Python-specific serialization protocol. Binary encodings may suit particular size or speed constraints, but are not universally faster or smaller than text formats.

  • Trust boundary: Never unpickle data from an untrusted or tampered source; Python documents that unpickling can execute arbitrary code. Validate parsed values against the application's expected schema and constraints.

Serialization round trips still need validation

Serialization and parsing transform representations. Validation checks whether parsed values meet the receiving application's schema and constraints. Python Pickle requires a trusted source because loading untrusted data can execute code.

Structured values are encoded to text or bytes, stored or transmitted, decoded back to values, and then checked against application rules.

4. File Permissions and Exception Handling

Operating systems control access, but their permission models differ. The following read, write, and execute bits describe common POSIX permissions; Windows commonly uses access-control lists, and applications may add their own checks.

Standard File Permissions (POSIX)

  • Read (r): A process whose credentials match the applicable POSIX permission class may read the file's contents.

  • Write (w): The permission bit grants write access to the file. Removing a directory entry is a separate operation generally governed by permissions on the containing directory, with additional rules such as sticky directories.

  • Execute (x): For ordinary files, the bit can permit execution when the file type and system support it. Directory execute permission has a different role: it permits searching/traversing that directory.

Robust File Handling

File operations can fail because a path is missing, access is denied, a device or network becomes unavailable, or the data is invalid. Handle errors using the language's conventions where recovery is useful: for example, Python try/except, C++ exceptions and RAII, or Rust Result. Do not catch an error only to hide it; report, retry, or recover when the application can do so safely.

Key Takeaways
  • Files commonly retain data beyond a process, but write, flush, and close guarantees depend on the API and storage stack.
  • Text APIs encode and decode characters; binary APIs expose bytes for the program to interpret. Both store bytes.
  • Paths, modes, locking, and permission behavior vary across platforms and APIs.
  • In Python, w creates or truncates, a creates or appends, and b selects byte handling; mode letters are not universal.
  • Managed resources such as Python with and Java try-with-resources close files on block exit, including exceptional exit.
  • Runtime flush and close operations move data through buffers; durable storage may require further platform-specific steps.
  • Serialization and parsing transform representations; applications still need to validate parsed data. Never unpickle untrusted data.
  • Handle file failures using the language's error model and recover or report them when appropriate.

Selected References