Building Programming Language Interpreters: A bottom-up approach to runtimes, execution, and implementation in C++

Cover....1

Title Page....2

Copyright and Credits....3

Dedication....4

Contributors....5

Table of Contents....8

Preface....16

Free Benefits with Your Book....23

Part 1: Modeling the Programming Language Runtime Environment....26

Chapter 1: Defining the Scope....28

Technical requirements....29

Why do we keep creating new languages?....29

Domain-specific languages....31

Compilers and interpreters....31

The exercise we will complete in this book....32

How will the interpreter be integrated?....34

Summary....38

Chapter 2: The Blurred Lines Between Native Code, Virtual Machines, and Interpreters....40

Modern ISAs are virtual machines of sorts....41

Native programming languages also have a virtual machine model....45

Interpreters as virtual machines....46

Abstraction of complexity versus execution overhead....47

How JIT compilers change the performance calculation....50

Deciding which model our language will use....51

Summary....52

Chapter 3: Instructions, Concurrency, Inputs, and Outputs....54

Instructions as building blocks....55

Operator stack versus registers....56

Interpreter stack versus language stack....58

Interruptions to the control flow....61

Continuations across native and interpreted code....62

Concurrency model....64

Deciding on the execution model....67

Summary....68

Chapter 4: Native Types, User Types, and Extension Points....70

Different types of type systems....70

Static typing versus dynamic typing....71

Strong typing versus weak typing....74

Language types versus user-defined types....75

Nominal typing versus structural typing....76

Duck typing....78

Approaches to modeling the type system....80

The native types for your language....80

How the users will declare their own types....83

Interaction with native extensions....83

Summary....85

Chapter 5: Putting It All Together: Making Trade-Off Decisions....88

Designing the execution model....88

Interacting with inputs and outputs....89

Identifying continuations that are ready to execute....90

Performing interpreted operations....91

Making native callbacks....92

Concurrency in an embedded language....93

Memory management....95

Memory access patterns....95

Managing mutability....96

Data ownership....98

Life cycle management....99

Finalizing my design....99

Summary....100

Part 2: Modeling the Programming Language Syntax....102

Chapter 6: Review of Programming Language Paradigms....104

Imperative programming....105

Functional programming....106

Declarative programming....109

Logic programming....111

Choosing a paradigm for our language....114

Summary....115

Chapter 7: Values, Containers, and the Language Meta-Model....118

Variables and values....118

Values and containers....120

Mutability....122

Copies, references, and shared values....123

The language meta-model....124

Summary....126

Chapter 8: Lexical Scopes....128

The lexical pad....128

Function lexical scopes....129

Block lexical scopes....130

Closure lexical scopes....131

Global lexical scopes....132

Summary....133

Chapter 9: Putting It All Together and Creating a Coherent Vision....136

The shape of a declarative language....136

Specifying the empty program....137

Hello World!....138

Hello who?....140

Immutability, containers, and values....141

Value types....143

Container types....144

Variables and references....145

The final language design....146

Summary....151

Part 3: Implementing the Interpreter Runtime....152

Chapter 10: Initialization and Entry Point....154

The scaffolding....154

The global interpreter state....155

The interpreted program and the interpreter....155

The operation tree and the mutable interpreter state....156

Executing the operations in the tree....160

Making it friendly to be embedded....165

Refactoring for multiple types of operations....169

Designing the interface for IO....173

Summary....176

Chapter 11: Execution Frames, the Stack, and Continuations....178

Code as a value....179

Invoking a callable value....182

Making a function....186

Calling a function....195

The end-to-end example....198

Summary....200

Chapter 12: Running and Testing Language Operators....202

Identifying and implementing operators....208

IO operators....209

List operators....215

Testing the operators....221

Finalizing the integration....225

Summary....239

Part 4: Interpreting Source Code....242

Chapter 13: Lexing: Turning Text into a Stream of Tokens....244

Identifying token types and their data structures....245

Specifying the rules of the lexer....248

Evaluating different lexer libraries....249

Getting a stream of tokens....250

Testing the tokenizer....253

Summary....254

Chapter 14: Parsing: Turning a Stream of Tokens into a Parse Tree....256

Types of parse nodes and their data structures....257

Specifying the grammar for the parser....259

Evaluating different parser libraries....261

Building a grammar engine in C....263

Getting the parse tree....272

Summary....276

Chapter 15: Analyzing: Turning a Parse Tree into an Abstract Syntax Tree....278

Difference between a parse tree and an abstract syntax tree....279

Modeling the node types and their data structures....279

Transforming one tree into another....286

Summary....298

Chapter 16: Generating: Turning an Abstract Syntax Tree into Instructions....300

Matching AST nodes to interpreter operations....301

Introducing the TransitionLookAhead operation....301

Introducing StateMachineOperation....306

Generating the code for StateMachineOperation....314

Generating the entire program....321

Integrating into the interpreter....328

Executing code in our programming language for the first time....330

Summary....332

Chapter 17: Proving That It Works....334

Revisiting our goals....334

Domain-specific language (DSL) for specifying network protocols....335

Synchronous request–response protocols....335

Transfer of control at multiple points....335

Transferring values to the native language....336

Using captured values within the protocol....336

Checking our acceptance criteria....337

Initialize the interpreter once....337

Make the interpreter drive the interaction with the network....337

Transfer control at specific points....338

Working through an example....338

Understanding SMTP....339

Expressing SMTP in the DSL....340

Implementing an SMTP server....347

Implementing callbacks....351

Was it worth it?....355

Summary....356

Chapter 18: Unlock Your Exclusive Benefits....358

Other Books You May Enjoy....364

Index....368

Explore why you might build a new programming language, which aspects influence runtime and language design choices, and how to implement a working first-version interpreter for that language in C++.

Free with your book: DRM-free PDF version + access to Packt's next-gen Reader*

Key Features

• Design a domain-specific language to solve focused problems and reduce complexity and bugs
• Follow a bottom-up approach, from runtime design to interpreter implementation
• Build an interpreter from scratch as a functional, minimum viable product

Book Description

Designing a custom programming language can be the most effective way to solve certain types of problems—especially when precision, safety, or domain-specific expressiveness matters. This book guides you through the full process of designing and implementing your own programming language and interpreter, from language design to execution, using modern C++.

You’ll start by exploring when and why building a domain-specific language is worth it, and how to design one to fit a specific problem domain. Along the way, you’ll examine real-world interpreter architectures and see how their design decisions affect language behavior, capabilities, and runtime trade-offs.

The book then walks through the entire process of interpreter implementation: defining syntax, building a lexer and parser, designing an abstract syntax tree, generating executable instructions, and implementing a runtime. All examples are in modern C++, with a focus on clean architecture and real-world usability.

By the end, you’ll have a fully working interpreter for a domain-specific language designed to handle network protocols—plus the knowledge and tools to design your own programming language from scratch.

*Email sign-up and proof of purchase required

What you will learn

• Design a domain-specific language and interpreter from scratch
• Write an interpreter that can be embedded into existing environments
• Understand how runtime shapes language execution and interpreter design
• Reason about language design and runtime trade-offs
• Define and implement the execution model of an interpreted language
• Implement a lexer, parser, analyzer, and instruction emitter in C++

Who this book is for

This book is tailored for intermediate to advanced software developers, particularly those interested in language design and implementation. It's ideal for programmers seeking to expand their skill set and tackle complex problems efficiently. Professionals working in roles such as software engineers, language designers, or system architects will benefit from the practical insights and hands-on experience provided in the book. Good understanding of C++ programming and basic understanding of language design concepts are recommended to fully grasp the content.