Deep Belief Nets in C++ and CUDA C: Volume 3: Convolutional Nets

Table of Contents....4

About the Author....7

About the Technical Reviewer....8

Introduction....9

Chapter 1: Feedforward Networks....11

Review of Multiple-Layer Feedforward Networks....11

Wide vs. Deep Nets....14

Locally Connected Layers....16

Rows, Columns, and Slices....17

Convolutional Layers....18

Half-Width and Padding....19

Striding and a Useful Formula....22

Pooling Layers....24

Pooling Types....24

The Output Layer....25

SoftMax Outputs....25

Back Propagation of Errors for the Gradient....28

Chapter 2: Programming Algorithms....33

Model Declarations....34

Order of Weights and Gradient....35

Initializations in the Model Constructor....36

Finding All Activations....39

Activating a Fully Connected Layer....40

Activating a Locally Connected Layer....41

Activating a Convolutional Layer....44

Activating a Pooling Layer....46

Evaluating the Criterion....49

Evaluating the Gradient....52

Gradient for a Fully Connected Layer....56

Gradient for a Locally Connected Layer....58

Gradient for a Convolutional Layer....61

Gradient for a Pooled Layer (Not!)....62

Backpropagating Delta from a Nonpooled Layer....63

Backpropagating Delta from a Pooled Layer....66

Multithreading Gradient Computation....68

Memory Allocation for Threading....73

Chapter 3: CUDA Code....76

Weight Layout in the CUDA Implementation....77

Global Variables on the Device....78

Initialization....80

Copying Weights to the Device....81

Activating the Output Layer....88

Activating Locally Connected and Convolutional Layers....90

Using Shared Memory to Speed Computation....97

Device Code....102

Launch Code....108

Activating a Pooled Layer....110

SoftMax and Log Likelihood by Reduction....114

Computing Delta for the Output Layer....118

Backpropagating from a Fully Connected Layer....120

Backpropagating from Convolutional and Local Layers....122

Backpropagating from a Pooling Layer....128

Gradient of a Fully Connected Layer....131

Gradient of a Locally Connected or Convolutional Layer....133

Flattening the Convolutional Gradient....138

Launch Code for the Gradient....140

Fetching the Gradient....144

Putting It All Together....150

Chapter 4: CONVNET Manual....155

Menu Options....155

File Menu....155

Test Menu....157

Display Menu....158

Read Control File....158

Making and Reading Image Data....159

Reading a Time Series as Images....159

Model Architecture....163

Training Parameters....164

Operations....167

Display Options....168

Display Training Images....168

Display Filter Images....168

Display Activation Images....169

Example of Displays....170

The CONVNET.LOG file....174

Printed Weights....177

The CUDA.LOG File....180

Index....181

This book is a continuation of Volumes 1 and 2 of this series. Numerous references are made to material in the prior volumes, especially in regard to coding threaded operation and CUDA implementations. For this reason, it is strongly suggested that you be at least somewhat familiar with the material in Volumes 1 and 2. Volume 1 is especially important, as it is there that much of the philosophy behind multithreading and CUDA hardware accommodation appears.

All techniques presented in this book are given modest mathematical justification, including the equations relevant to algorithms. However, it is not necessary for you to understand the mathematics behind these algorithms. Therefore, no mathematical background beyond basic algebra is necessary.

The two main purposes of this book are to present important convolutional net algorithms in thorough detail and to guide programmers in the correct and efficient programming of these algorithms. For implementations that do not use CUDA processing, the language used here is what is sometimes called enhanced C, which is basically C that additionally employs some of the most useful aspects of C++ without getting into the full C++ paradigm. Strict C (except for CUDA extensions) is used for the CUDA algorithms. Thus, you should ideally be familiar with C and C++, although my hope is that the algorithms are presented sufficiently clearly that they can be easily implemented in any language.