LING 590 F26 Week 9, 11/06/26. Intro to neural nets.

Key points from 3Blue1Brown videos

But what is a neural network?

• "Plain vanilla" neural network: called multilayer perceptron (MLP), a simple kind of architecture for a deep neural network (DNN), which has many layers (like 5 or more). MLPs typically are on the shallow side, e.g., just a few layers.
• In the example here (see screenshot below), we've got a total of 4 layers.

• Input layer: in the example here, we have an input vector of length 784, because each image is 28 x 28 pixels. The feature extracted from the image is the brightness value in grayscale, ranging from 0 (black) to 1 (white).

• Output layer: this is the result of the classification task. Since there are 10 categories (10 possible digits), the output vector is of length 10. It tells us which digit is the most likely class for the image in the input. Typically there is a probability distribution over the 10 categories, and the one with the highest probability is the class that the neural net classifies the input image as.

• Hidden layers : the layers in between the input and output layers. These are what make this a multilayer perceptron, as opposed to a perceptron, which has no hidden layers. These layers are also what makes the MLP a non-linear classifier as opposed to a linear classifier like a perceptron.

• Computing with neural nets is all about setting weights. Pre-trained neural net models have the weights for each neuron already set based on training data and then can be used off-the-shelf.

  • We are yet again working with linear combinations like we have been talking about for basis vectors and basis functions, see screenshots from 3Blue1Brown example below, for computing the weight for one neuron in the first hidden layer based on weighting the input vector.

• Activation functions The video and notes discuss two activation functions used to transform the output from the linear combinations to an activation for a neuron: a sigmoidal (the logistic) function \(\sigma(x)\), and also ReLUs (Rectified Linear Units). Wrapping up the weighted sums in these activation functions is what makes the classifier nonlinear.

\[ \sigma(x) = \frac{1}{1 + e^{-x}} \]

\[ \text{ReLU}(a) = \max{(0,a)} \]

• The logistic function is commonly used as the activation function for the output layer, since we want a probability distribution over the classes. This tells us how likely our input is each one of the possible digits.

• ReLUs are typically used for the hidden layers, since sigmoidal activation functions are very flat at extremes, which can lead to very slow learning (since jiggling around weights has minimal effect). Also super fast to compute.

• Finally, the bias term is introduced, which is just an additive constant that you can think of as setting a nonzero, positive activation threshold for the neuron

Perceptron demo

• The simplest kind of neural network only has an input layer and an output layer. No hidden layers (layers in between). This is called a perceptron, which is a linear classifier.

• One other parameter to introduce here, called the learning rate, which often gets the symbol eta \(\eta\), which scales how big a step is taken when trying a new hypothesis decision boundary.

• We can take a look at this perceptron demo by Jay Alammar

• We'll then take a look at a demo, to see how learning works in a perceptron in an R notebook, see code/perceptron-demo.Rmd.