AskSia

Plus

Problem 1. (25 points) 1. Consider a neural network with three hidden layers: $...

Dec 18, 2024

Solution by Steps

Question 1: Neural Network

step 1

The neural network consists of 5 input units, 3 units in the first hidden layer, 4 units in the second hidden layer, 2 units in the third hidden layer, and 1 output unit. The total number of neurons is

5 + 3 + 4 + 2 + 1 = 15

step 2

The explicit expression for the output function

f(X)

using ReLU activation can be written as:

f(X) = W_4 \cdot \text{ReLU}(W_3 \cdot \text{ReLU}(W_2 \cdot \text{ReLU}(W_1 \cdot X + b_1) + b_2) + b_3) + b_4

where

W_i

are the weight matrices and

b_i

are the bias vectors for each layer

step 3

Assigning hypothetical values: Let

W_1 = \begin{bmatrix} 0.1 &amp; 0.2 &amp; 0.3 &amp; 0.4 &amp; 0.5 \\ 0.1 &amp; 0.2 &amp; 0.3 &amp; 0.4 &amp; 0.5 \\ 0.1 &amp; 0.2 &amp; 0.3 &amp; 0.4 &amp; 0.5 \end{bmatrix}

b_1 = \begin{bmatrix} 0.1 \\ 0.2 \\ 0.3 \end{bmatrix}

, and similarly for other layers. For input

X = [1, 0, -1, 2, 3]^{\top}

, compute

f(X)

step 4

The total number of parameters is calculated as follows:

\text{Total Parameters} = (5 \times 3 + 3) + (3 \times 4 + 4) + (4 \times 2 + 2) + (2 \times 1 + 1) = 15 + 16 + 10 + 3 = 44

Question 2: Convolutional Neural Network (CNN)

step 1

The first convolutional layer has 5 filters of size

3 \times 3

with no padding, resulting in an output size of

26 \times 26

(since

28 - 3 + 1 = 26

). The second layer has 10 filters of size

3 \times 3

applied to the

26 \times 26

feature maps, resulting in an output size of

24 \times 24

step 2

The total number of parameters in the CNN is calculated as follows: For the first layer:

(3 \times 3 \times 1 \times 5) + 5 = 45 + 5 = 50

For the second layer:

(3 \times 3 \times 5 \times 10) + 10 = 135 + 10 = 145

Total parameters =

50 + 145 = 195

step 3

The CNN can be represented as a fully connected feedforward neural network by treating each pixel in the input image as an individual input unit. Each connection from the input to the hidden layer corresponds to a weight in the fully connected network. The constraints arise from the local connectivity and shared weights in convolutional layers

step 4

If there were no constraints, the number of parameters in the fully connected network would be:

28 \times 28 \times 195 = 156840

Answer

Total parameters in the neural network: 44; Total parameters in the CNN: 195; Fully connected network parameters: 156840

Key Concept

Understanding the structure and parameters of neural networks and CNNs

Explanation

The calculations show how to derive the number of parameters and the structure of both types of networks, highlighting the differences in connectivity and parameterization.