Lecture 02: System Calls and Operating System Structures

1. What are System Calls?
2. System Call Lifecycle and Examples
3. SOS Structures: Monolithic, Layered, Microkernel
4. Performance and Modularity Trade-offs
5. Summary and Q&A

Slide Deck

Key Takeaways

• System Calls serve as the secure interface between user applications and the OS kernel, enabling controlled access to hardware and services.
• OS Structure directly impacts performance, modularity, and fault isolation. The choice of architecture influences how the OS handles complexity and failures.
• Real-World Usage:
  • Linux uses a hybrid monolithic model — fast but extensible.
  • Microkernels (e.g., QNX, MINIX) are favored in embedded and safety-critical systems for their reliability and modularity.

Notes for Digging Deeper: Mastering the Linux System Call Interface

Introduction: The Ring 3 to Ring 0 Boundary

A system call (syscall) is the fundamental programmatic interface between a user-space application and the Linux kernel. Because user applications execute in a restricted environment (Ring 3 on x86 architectures), they cannot directly access hardware, manage physical memory, or spawn processes. To perform these privileged operations, the application must issue a system call, temporarily yielding control to the kernel (Ring 0) to execute the request securely.

Understanding Protection Rings: Protection rings are a hardware-enforced security mechanism built directly into the CPU to protect critical system functions from faults and malicious behavior. On x86 architectures, there are traditionally four hierarchical levels of privilege, numbered from 0 to 3. Ring 0 is the innermost layer possessing absolute, unrestricted access to the CPU, memory, and hardware; this is where the operating system kernel operates. Ring 3 is the outermost layer with the fewest privileges, serving as the sandbox where everyday user applications run. (Note: Rings 1 and 2 were historically intended for custom device drivers but are generally bypassed and unused by modern operating systems like Linux and Windows). This strict architectural separation guarantees that if a Ring 3 application (like a web browser or a student’s C program) crashes or misbehaves, it cannot bring down the entire Ring 0 operating system with it.

Below are the core concepts every systems programmer must master to interact with the Linux kernel effectively, efficiently, and securely.

The Syscall Flow and the libc Wrapper

User programs rarely trigger system calls directly using assembly language. Instead, they rely on the C Standard Library (glibc), which acts as an intermediary.

1. Invocation: The application calls a standard library function (e.g., open()).
2. Trap Generation: The glibc wrapper sets up the necessary CPU registers and executes a software interrupt or a dedicated architecture instruction (e.g., syscall on x86_64, int 0x80 on older 32-bit systems, or svc on ARM).
3. Context Switch: The CPU switches from user mode to kernel mode.
4. Execution & Return: The kernel verifies the request, performs the privileged operation, and returns the result (and control) back to user space.

Syscall Numbering and Architecture Dependence

The kernel does not identify system calls by string names; it uses a unique integer index mapped to a kernel function table (the syscall table). This numbering is strictly architecture-dependent. For example, open() is syscall 2 on x86_64, syscall 5 on x86 (32-bit), and syscall 56 on ARM64.

Modern Engineering Note: Modern architectures like ARM64 and RISC-V actually omitted the legacy open() system call entirely to save space and enforce security, relying exclusively on the newer, directory-relative openat() syscall.

The Calling Convention and the errno Illusion

To pass arguments into the kernel, the glibc wrapper places them into specific CPU registers before triggering the syscall instruction. On x86_64, the convention uses:

• RAX: Syscall Number
• RDI, RSI, RDX, R10, R8, R9: Arguments 1 through 6.

The errno Illusion: The Linux kernel knows nothing about the C variable errno. If a syscall fails, the kernel returns a negative error code directly in the RAX register (e.g., -ENOENT). The glibc wrapper intercepts this, negates it, stores the positive error code in the thread-local errno variable, and returns -1 to the user’s C code.

Code Example: Bypassing libc wrappers You can invoke the kernel directly using the syscall() function, bypassing standard library wrappers:

#include <unistd.h>
#include <sys/syscall.h>

int main() {
    // Equivalent to write(1, "Hello Kernel\n", 13);
    // Directly invoking syscall number 1 (SYS_write on x86_64)
    syscall(SYS_write, 1, "Hello Kernel\n", 13);
    return 0;
}

The Data Boundary (Transferring Memory)

The kernel cannot simply “trust” pointers passed from user space, as the user might pass a pointer to protected kernel memory or unmapped pages. Data transfer mechanisms include:

• Copying: The kernel uses specific routines (copy_from_user() and copy_to_user()) to safely validate and copy data across the boundary (used in read() and write()).
• Mapping: Memory is shared directly between user space and the kernel or hardware (used in mmap()), avoiding copy overhead.
• Pinning: Pages are locked into physical RAM during Direct I/O (O_DIRECT), preventing the OS from swapping them out while a disk controller reads them.

Blocking vs. Non-Blocking Syscalls

By default, I/O system calls are blocking. If you call read() on a network socket and no data is available, the kernel removes your thread from the CPU run queue and puts it to sleep until data arrives.

Non-Blocking calls instruct the kernel to return immediately if the operation cannot be completed instantly, typically returning -1 and setting errno to EAGAIN or EWOULDBLOCK.

Code Example: Setting Non-Blocking Mode

#include <fcntl.h>
#include <stdio.h>

void set_nonblocking(int fd) {
    int flags = fcntl(fd, F_GETFL, 0);
    if (flags == -1) return;
    
    // Append O_NONBLOCK to existing flags
    fcntl(fd, F_SETFL, flags | O_NONBLOCK);
}

Interrupted Syscalls and Restartability

If a process is blocked in a system call (e.g., waiting for terminal input) and a signal arrives (like SIGALRM or SIGCHLD), the kernel interrupts the syscall to deliver the signal.

• If the signal handler returns, the system call does not resume automatically. It fails, returning -1 with errno set to EINTR (Interrupted System Call).
• Restartable: Systems programmers can configure signals with the SA_RESTART flag via sigaction(). This tells the kernel to automatically restart certain interruptible syscalls (like read() or wait()) after the signal handler finishes.

Code Example: The Robust I/O Loop A seasoned systems programmer never assumes a single read() or write() will succeed uninterrupted. You must wrap them in an EINTR retry loop:

#include <unistd.h>
#include <errno.h>

ssize_t robust_read(int fd, void *buf, size_t count) {
    ssize_t bytes_read;
    while (1) {
        bytes_read = read(fd, buf, count);
        // If the read was interrupted by a signal, try again
        if (bytes_read == -1 && errno == EINTR) {
            continue; 
        }
        return bytes_read; // Return success or a legitimate error
    }
}

Multiplexing Syscalls (The Evolution of I/O)

When writing high-performance servers (like Nginx or Redis), a single thread must manage tens of thousands of open network sockets without blocking on any of them.

• Legacy (Avoid): select() and poll(). These algorithms require the kernel to iterate through every single file descriptor on every call (an $O(N)$ operation), causing severe CPU bottlenecks at scale.
• Modern (Standard): epoll(). This is an event-driven $O(1)$ interface. The kernel maintains a red-black tree of file descriptors and only notifies the user application about the specific sockets that have data ready.
• Cutting-Edge (Future): io_uring. A revolutionary asynchronous interface utilizing shared ring-buffers mapped directly between user and kernel space, eliminating syscall overhead almost entirely.

Security and TOCTOU Race Conditions

Because system calls traverse security boundaries, they are highly vulnerable to manipulation.

• TOCTOU (Time-of-Check to Time-of-Use): A classic vulnerability where a program checks a file’s permissions, but an attacker swaps the file for a malicious symlink milliseconds before the program actually opens it.
• The Mitigation: Never use path names twice. Open a directory file descriptor, and use the *at family of system calls (like openat(), fstatat()) to ensure atomic operations relative to a locked directory structure.

Code Example: Preventing TOCTOU with openat()

#include <fcntl.h>
#include <unistd.h>
#include <sys/stat.h>

void secure_file_creation(const char *dir_path, const char *filename) {
    // Open the directory itself
    int dir_fd = open(dir_path, O_RDONLY | O_DIRECTORY);
    
    // Safely create a file relative to the opened directory descriptor
    // Prevents attackers from modifying the path resolution between checking and opening
    int file_fd = openat(dir_fd, filename, O_CREAT | O_WRONLY | O_EXCL, 0600);
    
    close(file_fd);
    close(dir_fd);
}

Required Reading and External Resources

To deepen your understanding of the kernel boundary, consult the following authoritative resources:

• Linux Programmer’s Manual:

• man 2 syscalls (The definitive list of all Linux system calls).

• man 2 intro (Introduction to system calls and error handling).

• man 7 signal (Comprehensive guide on signal interruptions and SA_RESTART).

• OSTEP (Operating Systems: Three Easy Pieces):

• Read Chapter 4: The Abstraction: The Process for a theoretical breakdown of kernel/user execution states.

• Linux Kernel Documentation:

• Adding a New System Call - Read the kernel documentation to understand exactly how the syscall table is populated and maintained in the Linux source tree.