Show description

The squared loss function given by å ( y - yˆ i =1 ) 2 should be used for regression problems. The output layer in this case will have a single unit. Types of Units/Activation Functions/Layers We will now look at a number of Units/Activation Functions/Layers commonly used for Neural Networks. Let’s start by enumerating a few properties of interest for activation functions. 1. In theory, when an activation function is non-linear, a two-layer Neural Network can approximate any function (given a sufficient number of units in the hidden layer).

This loss function should typically be used when the Neural Network is designed to predict the probability of the outcome. In such cases, the output layer has a single unit with a suitable sigmoid as the activation function. 2. The Cross entropy function given by the expression n -å yi log f ( xi ,q ) i =1 is the recommended loss function for multi-classification. This loss function should typically be used with the Neural Network and is designed to predict the probability of the outcomes of each of the classes.

Let us again use the idea behind Maximum Likelihood, which is to find a θ that maximizes P ( D | q ) . Assuming a Multinomial distribution and given that each of the examples {(x1, y1), (x2, y2), … (xn, yn)} are independent, we have the following expression: P ( D |q ) = n! n1 ! × n2 ! ××× nk ! n Õ f ( x ,q ) i =1 yi i 23 Chapter 3 ■ Feed Forward Neural Networks We can take a logarithm operation on both sides to arrive at the following: n log P ( D | q ) = log n ! - log n1 ! × n2 ! ××× nk ! + log Õ f ( xi , q ) yi i =1 This can be simplified to the following: n log P ( D | q ) = log n !