Let's Go Further: Advanced patterns for building, managing and deploying RESTful JSON APIs and web applications in Go

Contents....2

Introduction....6

Conventions....7

About the author....8

Copyright and disclaimer....8

Prerequisites....9

Background knowledge....9

Go 1.16....9

Other software....9

Getting Started....11

Project Setup and Skeleton Structure....12

Generating the skeleton directory structure....13

Hello world!....14

A Basic HTTP Server....16

Creating the healthcheck handler....17

Demonstration....18

Additional Information....20

API versioning....20

API Endpoints and RESTful Routing....21

Choosing a router....22

Encapsulating the API routes....23

Adding the new handler functions....25

Creating a helper to read ID parameters....28

Additional Information....29

Conflicting routes....29

Customizing httprouter behavior....30

Sending JSON Responses....31

Fixed-Format JSON....33

Additional Information....36

JSON charset....36

JSON Encoding....38

Creating a writeJSON helper method....40

Additional Information....42

How different Go types are encoded....42

Using json.Encoder....43

Performance of json.Encoder and json.Marshal....44

Additional JSON encoding nuances....45

Encoding Structs....46

Changing keys in the JSON object....49

Hiding struct fields in the JSON object....50

Additional Information....52

The string struct tag directive....52

Formatting and Enveloping Responses....54

Relative performance....55

Enveloping responses....56

Additional Information....59

Response structure....59

Advanced JSON Customization....60

Customizing the Runtime field....61

Additional Information....64

Alternative #1 - Customizing the Movie struct....64

Alternative #2 - Embedding an alias....66

Sending Error Messages....68

Routing errors....70

Additional Information....72

System-generated error responses....72

Parsing JSON Requests....74

JSON Decoding....76

Zero values....78

Additional Information....79

Supported destination types....79

Using the json.Unmarshal function....79

Additional JSON decoding nuances....80

Managing Bad Requests....81

Triaging the Decode error....82

Making a bad request helper....85

Additional Information....86

Panicking vs returning errors....86

Restricting Inputs....88

Custom JSON Decoding....92

The json.Unmarshaler interface....92

Validating JSON Input....96

Creating a validator package....96

Performing validation checks....98

Making validation rules reusable....100

Database Setup and Configuration....104

Setting up PostgreSQL....105

Installing PostgreSQL....105

Connecting to the PostgreSQL interactive terminal....105

Creating databases, users, and extensions....107

Connecting as the new user....108

Additional Information....108

Optimizing PostgreSQL settings....108

Connecting to PostgreSQL....110

Establishing a connection pool....110

Decoupling the DSN....113

Additional Information....115

Using the DSN with psql....115

Configuring the Database Connection Pool....116

Configuring the pool....116

The SetMaxOpenConns method....116

The SetMaxIdleConns method....117

The SetConnMaxLifetime method....118

The SetConnMaxIdleTime method....119

Putting it into practice....119

Configuring the connection pool....120

SQL Migrations....123

An Overview of SQL Migrations....124

Installing the migrate tool....124

Working with SQL Migrations....126

Executing the migrations....129

Additional Information....130

Migrating to a specific version....130

Executing down migrations....130

Fixing errors in SQL migrations....131

Remote migration files....132

Running migrations on application startup....132

CRUD Operations....134

Setting up the Movie Model....135

Additional Information....138

Mocking models....138

Creating a New Movie....141

Executing the SQL query....141

Hooking it up to our API handler....143

Creating additional records....145

Additional Information....147

$N notation....147

Executing multiple statements....147

Fetching a Movie....149

Updating the API handler....150

Additional Information....152

Why not use an unsigned integer for the movie ID?....152

Updating a Movie....154

Executing the SQL query....154

Creating the API handler....155

Using the new endpoint....158

Deleting a Movie....160

Adding the new endpoint....161

Advanced CRUD Operations....164

Handling Partial Updates....165

Performing the partial update....166

Demonstration....168

Additional Information....169

Null values in JSON....169

Optimistic Concurrency Control....170

Preventing the data race....171

Implementing optimistic locking....172

Additional Information....176

Round-trip locking....176

Locking on other fields or types....176

Managing SQL Query Timeouts....178

Mimicking a long-running query....178

Adding a query timeout....180

Timeouts outside of PostgreSQL....183

Updating our database model....184

Additional Information....186

Using the request context....186

Filtering, Sorting, and Pagination....188

Parsing Query String Parameters....189

Creating helper functions....190

Adding the API handler and route....191

Creating a Filters struct....193

Validating Query String Parameters....196

Listing Data....199

Updating the application....199

Filtering Lists....204

Dynamic filtering in the SQL query....204

Full-Text Search....209

Adding indexes....211

Additional Information....212

Non-simple configuration and more information....212

Using STRPOS and ILIKE....213

Sorting Lists....215

Implementing sorting....216

Paginating Lists....221

The LIMIT and OFFSET clauses....221

Updating the database model....223

Returning Pagination Metadata....226

Calculating the total records....226

Updating the code....227

Structured Logging and Error Handling....233

Structured JSON Log Entries....234

Creating a custom logger....235

Additional Information....240

Integration with the http.Server error log....241

Third-party logging packages....241

Panic Recovery....243

Additional Information....245

Panic recovery in other goroutines....245

Rate Limiting....247

Global Rate Limiting....248

Enforcing a global rate limit....249

IP-based Rate Limiting....253

Deleting old limiters....254

Additional Information....257

Distributed applications....257

Configuring the Rate Limiters....258

Graceful Shutdown....262

Sending Shutdown Signals....263

Intercepting Shutdown Signals....266

Catching SIGINT and SIGTERM signals....268

Executing the Shutdown....271

User Model Setup and Registration....275

Setting up the Users Database Table....276

Setting up the Users Model....278

Adding Validation Checks....280

Creating the UserModel....282

Registering a User....286

Additional Information....290

Email case-sensitivity....290

User enumeration....291

Sending Emails....293

SMTP Server Setup....294

Setting up Mailtrap....294

Creating Email Templates....298

Sending a Welcome Email....300

Creating an email helper....300

Using embedded file systems....302

Using our mail helper....303

Checking the email in Mailtrap....306

Additional Information....307

Retrying email send attempts....307

Sending Background Emails....309

Recovering panics....311

Using a helper function....312

Graceful Shutdown of Background Tasks....316

An introduction to sync.WaitGroup....316

Fixing our application....317

User Activation....322

Setting up the Tokens Database Table....324

Creating Secure Activation Tokens....326

Creating the TokenModel and Validation Checks....328

Additional Information....330

The math/rand package....330

Sending Activation Tokens....331

Additional Information....336

A standalone endpoint for generating tokens....336

Activating a User....337

Creating the activateUserHandler....337

The UserModel.GetForToken method....340

Additional Information....344

Web application workflow....344

SQL query timing attack....346

Authentication....347

Authentication Options....348

HTTP basic authentication....348

Token authentication....349

Stateful token authentication....350

Stateless token authentication....350

API-key authentication....352

OAuth 2.0 / OpenID Connect....352

What authentication approach should I use?....353

Generating Authentication Tokens....355

Building the endpoint....356

Additional Information....360

The Authorization header....360

Authenticating Requests....362

Creating the anonymous user....362

Reading and writing to the request context....363

Creating the authentication middleware....365

Demonstration....368

Permission-based Authorization....370

Requiring User Activation....371

Demonstration....373

Splitting up the middleware....375

Additional Information....376

In-handler checks....376

Setting up the Permissions Database Table....378

Relationship between permissions and users....378

Creating the SQL migrations....380

Setting up the Permissions Model....382

Checking Permissions....385

Demonstration....387

Granting Permissions....390

Updating the permissions model....390

Updating the registration handler....391

Cross Origin Requests....394

An Overview of CORS....395

Demonstrating the Same-Origin Policy....397

Demonstration....399

Simple CORS Requests....403

Restricting Origins....406

Supporting multiple dynamic origins....406

Additional Information....411

Partial origin matches....411

The null origin....412

Authentication and CORS....412

Preflight CORS Requests....413

Demonstrating a preflight request....413

Responding to preflight requests....418

Updating our middleware....419

Additional Information....421

Caching preflight responses....421

Preflight wildcards....422

Metrics....424

Exposing Metrics with Expvar....425

Creating Custom Metrics....429

Dynamic metrics....431

Additional Information....437

Protecting the metrics endpoint....437

Removing default metrics....437

Request-level Metrics....438

Additional Information....444

Calculating additional metrics....444

Recording HTTP Status Codes....446

Additional Information....450

Visualizing and analyzing metrics....450

Building, Versioning and Quality Control....452

Creating and Using Makefiles....453

A simple makefile....453

Environment variables....454

Passing arguments....455

Namespacing targets....457

Prerequisite targets and asking for confirmation....458

Displaying help information....460

Phony targets....461

Managing Environment Variables....465

Using a .envrc file....466

Quality Controlling Code....468

Additional Information....471

Testing....471

Module Proxies and Vendoring....473

Module proxies....473

Vendoring....475

Vendoring new dependencies....480

Additional Information....483

The ./… pattern....483

Building Binaries....485

Reducing binary size....486

Cross-compilation....487

Additional Information....488

Build caching....488

Managing and Automating Version Numbers....490

Displaying the version number....490

Including the build time....492

Automated version numbering with Git....495

Using Git tags....498

Deployment and Hosting....500

Creating a Digital Ocean Droplet....502

Creating a SSH key....503

Adding the SSH key to Digital Ocean....505

Creating a droplet....507

Server Configuration and Installing Software....515

Connecting as the greenlight user....518

Connecting to the droplet....521

Additional Information....522

Future changes to the droplet configuration....522

Deployment and Executing Migrations....523

Running the API....526

Running the API as a Background Service....529

Restarting after reboot....532

Disable port 4000....533

Additional Information....534

Listening on a restricted port....534

Viewing logs....534

Configuring the SMTP provider....535

Additional unit file options....535

Using Caddy as a Reverse Proxy....536

Blocking access to application metrics....538

Using a domain name....541

Enabling HTTPS....543

Appendices....546

Managing Password Resets....547

Additional Information....553

Web application workflow....553

Creating Additional Activation Tokens....554

Authentication with JSON Web Tokens....559

JSON Encoding Nuances....568

Nil and empty slices are encoded differently....568

Using omitempty on a zero-valued struct doesn’t work....568

Using omitempty on a zero-value time.Time doesn’t work....569

Non-ASCII punctuation characters aren’t supported in struct tags....569

Integer, time.Time and net.IP values can be used as map keys....570

Angle brackets and ampersands in strings are escaped....570

Trailing zeroes are removed from floats....571

Working with pre-computed JSON....571

The MarshalText fallback....572

The receiver matters when using MarshalJSON....573

JSON Decoding Nuances....574

Decoding into Go arrays....574

Partial JSON decoding....574

Decoding into interface{} types....575

Decoding a JSON number to an interface{}....576

Struct tag directives....578

Request Context Timeouts....579

This book is written as a follow up to Let’s Go, and we’ll leverage a lot of the information and code patterns from that book again here.

If you’ve already read and enjoyed Let’s Go, then this book should be a good fit for you and the ideal next step in your learning. If you haven’t, then I highly recommend starting with Let’s Go first — especially if you’re a newcomer to Go.

You can read this as a standalone book, but please be aware that it is somewhat advanced — it doesn’t explain the fundamentals in detail, and some topics (like testing) don’t feature at all because they were covered heavily in the previous book. But if you’re comfortable using Go and already have a decent amount of experience behind you, then this book may also be a good fit for you. Feel free to jump straight in.