Prompt Engineering for Generative AI: Future-Proof Inputs for Reliable AI Outputs

Preface

Software Requirements for This Book

Conventions Used in This Book

Using Code Examples

O’Reilly Online Learning

How to Contact Us

Acknowledgments

1. The Five Principles of Prompting

Overview of the Five Principles of Prompting

1. Give Direction

2. Specify Format

3. Provide Examples

4. Evaluate Quality

5. Divide Labor

Summary

2. Introduction to Large Language Models for Text Generation

What Are Text Generation Models?

Vector Representations: The Numerical Essence of Language

Transformer Architecture: Orchestrating Contextual Relationships

Probabilistic Text Generation: The Decision Mechanism

Historical Underpinnings: The Rise of Transformer Architectures

OpenAI’s Generative Pretrained Transformers

GPT-3.5-turbo and ChatGPT

GPT-4

Google’s Gemini

Meta’s Llama and Open Source

Leveraging Quantization and LoRA

Mistral

Anthropic: Claude

GPT-4V(ision)

Model Comparison

Summary

3. Standard Practices for Text Generation with ChatGPT

Generating Lists

Hierarchical List Generation

When to Avoid Using Regular Expressions

Generating JSON

YAML

Filtering YAML Payloads

Handling Invalid Payloads in YAML

Diverse Format Generation with ChatGPT

Mock CSV Data

Explain It like I’m Five

Universal Translation Through LLMs

Ask for Context

Text Style Unbundling

Identifying the Desired Textual Features

Generating New Content with the Extracted Features

Extracting Specific Textual Features with LLMs

Summarization

Summarizing Given Context Window Limitations

Chunking Text

Benefits of Chunking Text

Scenarios for Chunking Text

Poor Chunking Example

Chunking Strategies

Sentence Detection Using SpaCy

Building a Simple Chunking Algorithm in Python

Sliding Window Chunking

Text Chunking Packages

Text Chunking with Tiktoken

Encodings

Understanding the Tokenization of Strings

Estimating Token Usage for Chat API Calls

Sentiment Analysis

Techniques for Improving Sentiment Analysis

Limitations and Challenges in Sentiment Analysis

Least to Most

Planning the Architecture

Coding Individual Functions

Adding Tests

Benefits of the Least to Most Technique

Challenges with the Least to Most Technique

Role Prompting

Benefits of Role Prompting

Challenges of Role Prompting

When to Use Role Prompting

GPT Prompting Tactics

Avoiding Hallucinations with Reference

Give GPTs “Thinking Time”

The Inner Monologue Tactic

Self-Eval LLM Responses

Classification with LLMs

Building a Classification Model

Majority Vote for Classification

Criteria Evaluation

Meta Prompting

Summary

4. Advanced Techniques for Text Generation with LangChain

Introduction to LangChain

Environment Setup

Chat Models

Streaming Chat Models

Creating Multiple LLM Generations

LangChain Prompt Templates

LangChain Expression Language (LCEL)

Using PromptTemplate with Chat Models

Output Parsers

LangChain Evals

OpenAI Function Calling

Parallel Function Calling

Function Calling in LangChain

Extracting Data with LangChain

Query Planning

Creating Few-Shot Prompt Templates

Fixed-Length Few-Shot Examples

Formatting the Examples

Selecting Few-Shot Examples by Length

Limitations with Few-Shot Examples

Saving and Loading LLM Prompts

Data Connection

Document Loaders

Text Splitters

Text Splitting by Length and Token Size

Text Splitting with Recursive Character Splitting

Task Decomposition

Prompt Chaining

Sequential Chain

itemgetter and Dictionary Key Extraction

Structuring LCEL Chains

Document Chains

Stuff

Refine

Map Reduce

Map Re-rank

Summary

5. Vector Databases with FAISS and Pinecone

Retrieval Augmented Generation (RAG)

Introducing Embeddings

Document Loading

Memory Retrieval with FAISS

RAG with LangChain

Hosted Vector Databases with Pinecone

Self-Querying

Alternative Retrieval Mechanisms

Summary

6. Autonomous Agents with Memory and Tools

Chain-of-Thought

Agents

Reason and Act (ReAct)

Reason and Act Implementation

Using Tools

Using LLMs as an API (OpenAI Functions)

Comparing OpenAI Functions and ReAct

Use Cases for OpenAI Functions

ReAct

Use Cases for ReAct

Agent Toolkits

Customizing Standard Agents

Custom Agents in LCEL

Understanding and Using Memory

Long-Term Memory

Short-Term Memory

Short-Term Memory in QA Conversation Agents

Memory in LangChain

Preserving the State

Querying the State

ConversationBufferMemory

Other Popular Memory Types in LangChain

ConversationBufferWindowMemory

ConversationSummaryMemory

ConversationSummaryBufferMemory

ConversationTokenBufferMemory

OpenAI Functions Agent with Memory

Advanced Agent Frameworks

Plan-and-Execute Agents

Tree of Thoughts

Callbacks

Global (Constructor) Callbacks

Request-Specific Callbacks

The Verbose Argument

When to Use Which?

Token Counting with LangChain

Summary

7. Introduction to Diffusion Models for Image Generation

OpenAI DALL-E

Midjourney

Stable Diffusion

Google Gemini

Text to Video

Model Comparison

Summary

8. Standard Practices for Image Generation with Midjourney

Format Modifiers

Art Style Modifiers

Reverse Engineering Prompts

Quality Boosters

Negative Prompts

Weighted Terms

Prompting with an Image

Inpainting

Outpainting

Consistent Characters

Prompt Rewriting

Meme Unbundling

Meme Mapping

Prompt Analysis

Summary

9. Advanced Techniques for Image Generation with Stable Diffusion

Running Stable Diffusion

AUTOMATIC1111 Web User Interface

Img2Img

Upscaling Images

Interrogate CLIP

SD Inpainting and Outpainting

ControlNet

Segment Anything Model (SAM)

DreamBooth Fine-Tuning

Stable Diffusion XL Refiner

Summary

10. Building AI-Powered Applications

AI Blog Writing

Topic Research

Expert Interview

Generate Outline

Text Generation

Writing Style

Title Optimization

AI Blog Images

User Interface

Summary

Index

About the Authors

Large language models (LLMs) and diffusion models such as ChatGPT and Stable Diffusion have unprecedented potential. Because they have been trained on all the public text and images on the internet, they can make useful contributions to a wide variety of tasks. And with the barrier to entry greatly reduced today, practically any developer can harness LLMs and diffusion models to tackle problems previously unsuitable for automation.

With this book, you'll gain a solid foundation in generative AI, including how to apply these models in practice. When first integrating LLMs and diffusion models into their workflows, most developers struggle to coax reliable enough results from them to use in automated systems. Authors James Phoenix and Mike Taylor show you how a set of principles called prompt engineering can enable you to work effectively with AI.

Learn how to empower AI to work for you. This book explains:

• The structure of the interaction chain of your program's AI model and the fine-grained steps in between
• How AI model requests arise from transforming the application problem into a document completion problem in the model training domain
• The influence of LLM and diffusion model architecture—and how to best interact with it
• How these principles apply in practice in the domains of natural language processing, text and image generation, and code