UX for Enterprise ChatGPT Solutions: A practical guide to designing enterprise-grade LLMs

Preface xvii
Part 1: UX Foundation for Enterprise ChatGPT
1
Recognizing the Power of Design in ChatGPT....3
Technical requirements....4
Approach 1 – The no-code approach....5
Approach 2 – code with Node.JS, Python, or curl....5
Traversing the history of conversational AI....5
The importance of UX design
for ChatGPT....10
Understanding the science and art of UX design....11
The science of design....12
The art of design....14
It takes a village to create superb UX....19
Setting up a customized model....21
Summary....24
References....25
2
Conducting Effective User Research....27
Surveying UX research methods....27
Understanding user needs analysis....29
Surveys for conversational AI....33
Survey checklist....34
Case study on an effective survey....40
Designing insightful interviews....44
Defining research objectives....45
Selecting participants....45
Develop a structured interview program....46
Pilot the interview process and program....46
Conduct the structured interviews....47
Record and document findings....48
Data analysis....48
Report findings....49
Summary of the interview process....49
Getting started with
conversational analysis....50
Table of Contents
Tagging a log file should focus on
each interaction....53
Define success and failure categories....55
Trying conversational analysis....60
Exploring the examples from the case study....61
Generate enhancements and bugs from
groups of issues....64
Score results....64
Results....65
Summary....66
References....66
3
Identifying Optimal Use Cases for ChatGPT....67
Understanding use case basics....68
Use case or user stories....68
Establishing a baseline with ChatGPT....69
Example use case for a ChatGPT
instance – patching software....71
Creating a user story from a use case....76
Prioritizing ChatGPT opportunities from the
use case....77
Aligning LLMs with user goals....79
Applications of ChatGPT....80
Examples of generative AI outside of chat....82
Avoiding ChatGPT limitations,
biases, and inappropriate responses....83
Lack of real-time information....83
Complex or specialized topics....84
Long-form content generation....84
Long-term memory....84
Sensitive information....85
Biased thinking....85
Emotion and empathy....86
Ethical and moral guidance....86
Critical decision making....86
Programming and debugging....87
Translation accuracy....87
Educational substitution....88
Don’t force-fit a solution....88
Summary....88
References....89
4
Scoring Stories....91
Prioritizing the backlog....91
WSJF....92
User Needs Scoring....95
Scoring enterprise solutions....96
Examples of scoring....103
Putting a backlog into order....109
Patching case study revisited....110
Extending tracking tools with scoring....111
Try the User Needs Scoring method....111
Creating more complex
scoring methods....112
Working with multiple backlogs in Agile....113
Real-world hiccups with scoring....115
I know Agile, and this is not WSJF....115
The use of simple numbers one to four....116
Weighting factors....116
Severity seems complicated to judge....117
The cost is so high that we can’t ever get the
work done....117
Grouping issues into bugs to protect the quality....118
How to work WSJF into the organization....118
Summary....119
References....119
5
Defining the Desired Experience....121
Designing chat experiences....121
Chat-only experiences....122
Integrating ChatGPT into an existing
chat experience....124
Enabling components for a chat experience....125
Designing hybrid UI/chat experiences....126
Chat window size and location....133
Tables....134
Forms....137
Charts....140
Graphics and images....141
Buttons, menus, and choice lists....143
Links....145
Creating voice-only experiences....147
Designing a recommender and
behind-the-scenes experiences....150
Overarching considerations....152
Accessibility....152
Internationalization....154
Trust....169
Security....172
Summary....173
References....173
Part 2: Designing
6
Gathering Data – Content is King....177
What is in a ChatGPT
foundational model....178
Incorporating enterprise data
using RAG....179
Understanding RAG....179
Limitations of ChatGPT and RAG....180
Building a demo with enterprise data....184
Cleaning data....188
Other considerations for creating a quality
data pipeline....208
Resources for RAG....215
Community resources....223
Summary....226
References....226
7
Prompt Engineering....227
Giving context through
prompt engineering....227
Prompt 101....228
Designing instructions....229
Basic strategies....231
Quick tricks to always keep in mind....235
A/B testing....237
Prompt engineering techniques....237
Self-consistency....237
General knowledge prompting....239
Prompt chaining....240
Program-aided language models....242
Few-shot prompting....244
Andrew Ng’s agentic approach....245
Reflection....246
Tool use....247
Planning....248
Multi-agent collaboration....248
Advanced techniques....250
Summary....260
References....260
8
Fine-Tuning....261
Fine-tuning 101....261
Prompt engineering or fine-tuning? Where to
spend resources....262
Token costs do matter....262
Creating fine-tuned models....264
Fine-tuning for style and tone....265
Using the fine-tuned model....272
Fine-tuning for structuring output....277
Generating data should still need a
check and balance....279
Fine-tuning for function and tool calling....284
Fine-tuning tips....285
Wove case study, continued....288
Prompt engineering....288
Fine-Tuning for Wove....289
Summary....294
References....294
Part 3: Care and Feeding
9
Guidelines and Heuristics....297
Applying guidelines to design....298
Adapting heuristic analysis for
conversational UIs....299
1 – Visibility of system status....302
2 – Match between a system and the real world....304
3 – User control and freedom....305
4 – Consistency and standards....308
5 – Error prevention....310
6 – Recognition rather than recall....312
7 – Flexibility and efficiency of use....315
8 – Aesthetic and minimalist design....316
9 – Help users recognize, diagnose, and
recover from errors....317
10 – Help and documentation....317
Is there an 11th possible heuristic?....319
Building conversational guidelines....320
Web guidelines....321
A sample guideline set for hybrid
chat/GUI experiences....321
Some specific style and tone guidelines
with examples....322
Flow order can reduce interactions....332
Case study....340
Handling errors – repair and disfluencies....342
Summary....345
References....345
10
Monitoring and Evaluation....347
Evaluate using RAGAs....347
The RAGAs process....348
Synthesizing data....349
Evaluation metrics....350
User experience metrics....357
Other metrics....359
Monitoring and classifying the types of
hallucination errors....359
OpenAI’s case study on quality and
how to measure it....363
Systematic testing processes....364
Testing matrix approach....368
Improving retrieval....372
The wide range of LLM evaluation metrics....372
Monitor with usability metrics....374
Net Promoter Score (NPS)....375
SUS....378
Refine with heuristic evaluation....380
Summary....380
References....380
11
Process....381
Incorporating design thinking
into development....381
Find a sponsor....383
Find the right tools and integrate
Generative AI....384
Be religious… at first....384
Avoid “unknown unknowns”....385
Always evolve and improve....385
Agile does not mean “no requirements”....385
Team composition and location matters....386
Manage Work in Progress (WIP) and
technical debt....386
Focus on customer value....387
Incorporate the design process
into the dev process....387
Designing a content improvement
life cycle....390
Inputs for conversational AIs....391
Inputs for recommender UIs....391
Inputs for backend AIs....391
Monitoring Monday....392
Analysis Tuesday (and Wednesday’s workup)....393
Treatment Thursday and fault-finding Friday....393
What doesn’t fit into a week is still important....394
Conclusion....398
References....399
12
Conclusion....401
Applying learnings to the
new frontier....401
Double-checking what feels right....402
Set clear goals....403
Know your processes....403
Know the data....404
Align and be accountable....404
Prioritize thoughtfully....405
Automate with intention....405
Building processes
that fit the solution....405
Wrapping up the journey....406
References....408
Index....409
Other Books You May Enjoy....420

Many enterprises grapple with new technology, often hopping on the bandwagon only to abandon it when challenges emerge. This book is your guide to seamlessly integrating ChatGPT into enterprise solutions with a UX-centered approach.

UX for Enterprise ChatGPT Solutions empowers you to master effective use case design and adapt UX guidelines through an engaging learning experience. Discover how to prepare your content for success by tailoring interactions to match your audience’s voice, style, and tone using prompt-engineering and fine-tuning. For UX professionals, this book is the key to anchoring your expertise in this evolving field. Writers, researchers, product managers, and linguists will learn to make insightful design decisions. You’ll explore use cases like ChatGPT-powered chat and recommendation engines, while uncovering the AI magic behind the scenes. The book introduces a care and feeding model, enabling you to leverage feedback and monitoring to iterate and refine any Large Language Model solution. Packed with hundreds of tips and tricks, this guide will help you build a continuous improvement cycle suited for AI solutions.

By the end, you’ll know how to craft powerful, accurate, responsive, and brand-consistent generative AI experiences, revolutionizing your organization’s use of ChatGPT.

What you will learn

• Align with user needs by applying design thinking to tailor ChatGPT to meet customer expectations

• Harness user research to enhance chatbots and recommendation engines

• Track quality metrics and learn methods to evaluate and monitor ChatGPT's quality and usability

• Establish and maintain a uniform style and tone with prompt engineering and fine-tuning

• Apply proven heuristics by monitoring and assessing the UX for conversational experiences with trusted methods

• Refine continuously by implementing an ongoing process for chatbot care and feeding

Who this book is for

This book is for user experience designers, product managers, and product owners of business and enterprise ChatGPT solutions who are interested in learning how to design and implement ChatGPT-4 solutions for enterprise needs. You should have a basic-to-intermediate level of understanding in UI/UX design concepts and fundamental knowledge of ChatGPT-4 and its capabilities.