System Programming Essentials with Go: System calls, networking, efficiency, and security practices with practical projects in Golang

Cover....1

Title Page....2

Copyright and Credits....3

Contributors....5

Table of Contents....8

Preface....16

Part 1: Introduction....22

Chapter 1: Why Go?....24

Choosing Go....25

Concurrency and goroutines....25

Concurrency....26

Goroutines....26

CSP-inspired model....26

Share by communication....27

Interacting with the OS....27

Tooling....28

go build....28

go test....28

go run....29

go vet....30

go fmt....30

Cross-platform development with Go....31

Summary....33

Chapter 2: Refreshing Concurrency and Parallelism....34

Technical requirements....34

Understanding goroutines....34

WaitGroup....36

Changing shared state....37

Managing data races....41

Atomic operations....42

Mutexes....43

Making sense of channels....45

How to use channels....45

An unbuffered channel....45

Buffered channels....51

The guarantee of delivery....53

Latency....53

State and signaling....55

State....55

Signaling....56

Choosing your synchronization mechanism....57

Summary....57

Part 2: Interation with the OS....58

Chapter 3: Understanding System Calls....60

Technical requirements....60

Introduction to system calls....61

The catalog of services and identification....61

Information exchange....62

The syscall package....62

A closer look at the os and x/sys packages....63

x/sys package – low-level system calls....63

Operating system functionality....65

Portability....67

Everyday system calls....68

Tracing system calls....68

Tracing specific system calls....69

Developing and testing a CLI program....70

Standard streams....70

File descriptors....72

Creating a CLI application....73

Redirections and standard streams....75

Making it testable....75

Summary....80

Chapter 4: File and Directory Operations....82

Technical requirements....82

Identifying unsafe file and directory permissions....82

Files and permissions....84

Scanning directories in Go....87

Understanding file paths....87

Using the path/filepath package....87

Traversing directories....89

Symbolic links and unlinking files....92

Symbolic links – the shortcut of the file world....92

Unlinking files – the great escape act....93

Calculating directory size....96

Finding duplicate files....98

Optimizing filesystem operations....100

Summary....102

Chapter 5: Working with System Events....104

Managing system events....104

What are signals?....104

The os/signal package....105

Task scheduling in Go....107

Why schedule?....107

Basic scheduling....108

Handling timer signals....110

File monitoring....111

Inotify....111

fsnotify....114

File rotation....115

Process management....120

Execution and timeouts....120

Execute and control process execution time....120

Building a distributed lock manager in Go....122

Summary....124

Chapter 6: Understanding Pipes in Inter-Process Communication....126

Technical requirements....126

What are pipes in IPC?....127

Why are pipes important?....127

Pipes in Golang....129

The mechanics of anonymous pipes....129

Navigating named pipes (Mkfifo())....134

Best practices – guidelines for using pipes....137

Efficient data handling....137

Error handling and resource management....139

Security considerations....140

Performance optimization....141

Developing a log processing tool....141

Summary....143

Chapter 7: Unix Sockets....144

Introduction to Unix sockets....144

Creating a Unix socket....145

Going a little deeper into socket creation....148

Creating the client....148

Inspecting the socket with lsof....150

Building a chat server....150

The complete chat client....157

Serving HTTP under UNIX domain sockets....158

Client....160

HTTP request line....162

HTTP request header....163

Empty line signifying end of headers....163

The textproto package....163

Performance....164

Other common use cases....164

Summary....165

Part 3: Performance....166

Chapter 8: Memory Management....168

Technical requirements....168

Garbage collection....168

Stack and heap allocation....169

The GC algorithm....170

GOGC....171

GC pacer....172

GODEBUG....173

Memory ballast....175

GOMEMLIMIT....176

Memory arenas....177

Using memory arenas....178

Summary....180

Chapter 9: Analyzing Performance....182

Escape analysis....182

Stack and pointers....183

Pointers....184

Stack....186

Heap....186

How can we analyze?....187

Benchmarking your code....191

Writing your first benchmark....192

Memory allocations....194

Common pitfalls....197

CPU profiling....200

Memory profiling....205

Profiling memory over time....206

Preparing to explore the trade-offs....207

Summary....209

Part 4: Connected Apps....210

Chapter 10: Networking....212

The net package....212

TCP sockets....214

HTTP servers and clients....217

HTTP verbs....218

HTTP status codes....219

Putting it all together....219

Securing the connection....221

Certificates....222

Advanced networking....228

UDP versus TCP....228

Summary....240

Chapter 11: Telemetry....242

Technical requirements....242

Logs....243

Zap versus slog....245

Logging for debugging or monitoring?....246

What to log?....249

What not to log?....250

Traces....251

Effective tracing....254

Distributed tracing....255

Metrics....256

What metric should we use?....260

The OTel project....262

OTel....263

Summary....266

Chapter 12: Distributing Your Apps....268

Technical requirements....268

Go Modules....268

The routine using modules....270

CI....278

Caching....279

Static analysis....281

Releasing your application....282

Summary....284

Part 5: Going Beyond....286

Chapter 13: Capstone Project – Distributed Cache....288

Technical requirements....288

Understanding distributed caching....289

System requirements....289

Requirements....290

Design and trade-offs....292

Creating the project....292

Thread safety....293

Choosing the right approach....294

Adding thread safety....295

The interface....295

TCP....296

HTTP....296

Others....297

Eviction policies....300

Sharding....314

Summary....324

Chapter 14: Effective Coding Practices....326

Technical requirements....326

Reusing resources....326

Using sync.Pool in a network server....329

Using sync.Pool for JSON marshaling....330

Executing tasks once....333

singleflight....335

Effective memory mapping....338

API usage....340

Advanced usage with protection and mapping flags....340

Avoiding common performance pitfalls....341

Leaking with time.After....342

Defer in for loops....344

Maps management....345

Resource management....346

Handling HTTP bodies....348

Channel mismanagement....349

Summary....351

Chapter 15: Stay Sharp with System Programming....352

Real-world applications....352

Dropbox’s leap of faith....352

HashiCorp – Go from day one....353

Grafana Labs – visualizing success with Go....354

Docker – building a container revolution with Go....355

SoundCloud – from Ruby to Go....356

Navigating the system programming landscape....357

Go release notes and blog....357

Community....357

Contribution....358

Experimentation....358

Resources for continued learning....358

Advanced Programming in the UNIX Environment by W. Richard Stevens....359

Learn C Programming - Second Edition: A beginner’s guide to learning the most powerful and general-purpose programming language with ease....359

Linux Kernel Programming - Second Edition: A comprehensive and practical guide to kernel internals, writing modules, and kernel synchronization....360

Linux System Programming Techniques: Become a proficient Linux system programmer using expert recipes and techniques....360

Operating Systems: Design and Implementation by Andrew S. Tanenbaum....361

Unix Network Programming by W. Richard Stevens....361

Linux System Programming Techniques: Become a proficient Linux system programmer using expert recipes and techniques....362

Mastering Embedded Linux Programming - Third Edition: Create fast and reliable embedded solutions with Linux 5.4 and the Yocto Project 3.1 (Dunfell)....363

Modern Operating Systems by Andrew S. Tanenbaum....363

The Art of UNIX Programming by Eric S. Raymond....364

Your system programming journey....365

Appendix ....366

Hardware Automation....366

Automation in system programming....366

USB....367

Application....367

The goal....368

The /proc/mounts file....370

Reading the files on the flash drive....371

Partitions versus blocks versus devices versus disks....374

Open source to the rescue!....375

Interacting with USB events....379

Bluetooth....384

Detecting the smartwatch....385

Locking the screen....390

XDG dilemma....391

The Wayland conundrum....391

Summary....391

Index....394

Go beyond web development to learn system programming, building secure, concurrent, and efficient applications with Go's unique system programming capabilities

Key Features

• Get a deep understanding of how Go simplifies system-level memory management and concurrency
• Gain expert guidance on essential topics like file operations, process management, and network programming
• Learn cross-platform system programming and how to build applications that interact directly with the OS

Alex Rios, a seasoned Go developer and active community builder, shares his 15 years of expertise in designing large-scale systems through this book. It masterfully cuts through complexity, enabling you to build efficient and secure applications with Go's streamlined syntax and powerful concurrency features.

In this book, you’ll learn how Go, unlike traditional system programming languages (C/C++), lets you focus on the problem by prioritizing readability and elevating developer experience with features like automatic garbage collection and built-in concurrency primitives, which remove the burden of low-level memory management and intricate synchronization.

Through hands-on projects, you'll master core concepts like file I/O, process management, and inter-process communication to automate tasks and interact with your system efficiently. You'll delve into network programming in Go, equipping yourself with the skills to build robust, distributed applications. This book goes beyond the basics by exploring modern practices like logging and tracing for comprehensive application monitoring, and advance to distributed system design using Go to prepare you to tackle complex architectures.

By the end of this book, you'll emerge as a confident Go system programmer, ready to craft high-performance, secure applications for the modern world.

What you will learn

• Understand the fundamentals of system programming using Go
• Grasp the concepts of goroutines, channels, data races, and managing concurrency in Go
• Manage file operations and inter-process communication (IPC)
• Handle USB drives and Bluetooth devices and monitor peripheral events for hardware automation
• Familiarize yourself with the basics of network programming and its application in Go
• Implement logging, tracing, and other telemetry practices
• Construct distributed cache and approach distributed systems using Go

Who this book is for

This book is for software engineers looking to expand their understanding of system programming concepts. Professionals with a coding foundation seeking profound knowledge of system-level operations will also greatly benefit. Additionally, individuals interested in advancing their system programming skills, whether experienced developers or those transitioning to the field, will find this book indispensable.