a

Lorem ipsum dolor sit amet, consectetur adicing elit ut ullamcorper. leo, eget euismod orci. Cum sociis natoque penati bus et magnis dis.Proin gravida nibh vel velit auctor aliquet. Leo, eget euismod orci. Cum sociis natoque penati bus et magnis dis.Proin gravida nibh vel velit auctor aliquet.

  /  Project   /  Blog: Artificial Neural Networks

Blog: Artificial Neural Networks


One of the fundamental premises of Artificial Intelligence is to build machines that can perform tasks that require human intelligence.

Go to the profile of Dammnn

The human brain is amazing at learning new things. Why not use the model of the human brain to build a machine? An artificial neural network is a model designed to simulate the learning process of the human brain.

Artificial neural networks are designed such that they can identify the underlying patterns in data and learn from them. They can be used for various tasks such as classification, regression, segmentation, and so on. We need to convert any given data into numerical form before feeding it into the neural network. For example, we deal with many different types of data including visual, textual, time-series, and so on. We need to figure out how to represent problems in a way that can be understood by artificial neural networks.

The human learning process is hierarchical. We have various stages in our brain’s neural network and each stage corresponds to a different granularity. Some stages learn simple things and some stages learn more complex things. Let’s consider an example of visually recognizing an object. When we look at a box, the first stage identifies simple things like corners and edges. The next stage identifies the generic shape and the stage after that identifies what kind of object it is. This process differs for different tasks, but you get the idea! By building this hierarchy, our human brain quickly separates the concepts and identifies the given object.

To simulate the learning process of the human brain, an artificial neural network is built using layers of neurons. These neurons are inspired by the biological neurons we discussed in the previous paragraph. Each layer in an artificial neural network is a set of independent neurons. Each neuron in a layer is connected to neurons in the adjacent layer.

If we are dealing with N-dimensional input data, then the input layer will consist of N neurons. If we have M distinct classes in our training data, then the output layer will consist of M neurons. The layers between the input and output layers are called hidden layers. A simple neural network will consist of a couple of layers and a deep neural network will consist of many layers.

Consider the case where we want to use a neural network to classify the given data. The first step is to collect the appropriate training data and label it. Each neuron acts as a simple function and the neural network trains itself until the error goes below a certain value. The error is basically the difference between the predicted output and the actual output. Based on how big the error is, the neural network adjusts itself and retrains until it gets closer to the solution. You can learn more about neural networks at http://pages.cs.wisc.edu/~bolo/shipyard/neural/local.html .

We will be using a library called NeuroLab in this chapter. You can find more about it at https://pythonhosted.org/neurolab . You can install it by running the following command on your Terminal:

$ pip3 install neurolab

Once you have installed it, you can proceed to the next section.

Building a Perceptron based classifier

A Perceptron is the building block of an artificial neural network. It is a single neuron that takes inputs, performs computation on them, and then produces an output. It uses a simple linear function to make the decision. Let’s say we are dealing with an N-dimension input data point. A Perceptron computes the weighted summation of those N numbers and it then adds a constant to produce the output. The constant is called the bias of the neuron. It is remarkable to note that these simple Perceptrons are used to design very complex deep neural networks. Let’s see how to build a Perceptron based classifier using NeuroLab.

Create a new Python file and import the following packages:

import numpy as np 
import matplotlib.pyplot as plt
import neurolab as nl

Load the input data from the text file data_perceptron.txt provided to you. Each line contains space separated numbers where the first two numbers are the features and the last number is the label:

# Load input data 
text = np.loadtxt('data_perceptron.txt')

Separate the text into datapoints and labels:

# Separate datapoints and labels 
data = text[:, :2]
labels = text[:, 2].reshape((text.shape[0], 1))

Plot the datapoints:

# Plot input data 
plt.figure()
plt.scatter(data[:,0], data[:,1])
plt.xlabel('Dimension 1')
plt.ylabel('Dimension 2')
plt.title('Input data')

Define the maximum and minimum values that each dimension can take:

# Define minimum and maximum values for each dimension 
dim1_min, dim1_max, dim2_min, dim2_max = 0, 1, 0, 1

Since the data is separated into two classes, we just need one bit to represent the output. So the output layer will contain a single neuron.

# Number of neurons in the output layer 
num_output = labels.shape[1]

We have a dataset where the datapoints are two-dimensional. Let’s define a Perceptron with two input neurons, where we assign one neuron for each dimension.

# Define a perceptron with 2 input neurons (because we 
# have 2 dimensions in the input data)
dim1 = [dim1_min, dim1_max]
dim2 = [dim2_min, dim2_max]
perceptron = nl.net.newp([dim1, dim2], num_output)

Train the perceptron with the training data:

# Train the perceptron using the data 
error_progress = perceptron.train(data, labels, epochs=100, show=20, lr=0.03)

Plot the training progress using the error metric:

# Plot the training progress 
plt.figure()
plt.plot(error_progress)
plt.xlabel('Number of epochs')
plt.ylabel('Training error')
plt.title('Training error progress')
plt.grid()

plt.show()

The full code is given in the file perceptron_classifier.py. If you run the code, you will get two output screenshot. The first screenshot indicates the input data points:

The second screenshot represents the training progress using the error metric:

We can observe from the preceding screenshot, the error goes down to 0 at the end of the fourth epoch.

Constructing a single layer neural network

A perceptron is a good start, but it cannot do much. The next step is to have a set of neurons act as a unit to see what we can achieve. Let’s create a single neural network that consists of independent neurons acting on input data to produce the output.

Create a new Python file and import the following packages:

import numpy as np 
import matplotlib.pyplot as plt
import neurolab as nl

We will use the input data from the file data_simple_nn.txt provided to you. Each line in this file contains four numbers. The first two numbers form the datapoint and the last two numbers are the labels. Why do we need to assign two numbers for labels? Because we have four distinct classes in our dataset, so we need two bits to represent them. Let us go ahead and load the data:

# Load input data 
text = np.loadtxt('data_simple_nn.txt')

Separate the data into datapoints and labels:

# Separate it into datapoints and labels 
data = text[:, 0:2]
labels = text[:, 2:]

Plot the input data:

# Plot input data 
plt.figure()
plt.scatter(data[:,0], data[:,1])
plt.xlabel('Dimension 1')
plt.ylabel('Dimension 2')
plt.title('Input data')

Extract the minimum and maximum values for each dimension (we don’t need to hardcode it like we did in the previous section):

# Minimum and maximum values for each dimension 
dim1_min, dim1_max = data[:,0].min(), data[:,0].max()
dim2_min, dim2_max = data[:,1].min(), data[:,1].max()

Define the number of neurons in the output layer:

# Define the number of neurons in the output layer 
num_output = labels.shape[1]

Define a single layer neural network using the above parameters:

# Define a single-layer neural network 
dim1 = [dim1_min, dim1_max]
dim2 = [dim2_min, dim2_max]
nn = nl.net.newp([dim1, dim2], num_output)

Train the neural network using training data:

# Train the neural network 
error_progress = nn.train(data, labels, epochs=100, show=20, lr=0.03)

Plot the training progress:

# Plot the training progress 
plt.figure()
plt.plot(error_progress)
plt.xlabel('Number of epochs')
plt.ylabel('Training error')
plt.title('Training error progress')
plt.grid()

plt.show()

Define some sample test datapoints and run the network on those points:

# Run the classifier on test datapoints 
print('\nTest results:')
data_test = [[0.4, 4.3], [4.4, 0.6], [4.7, 8.1]]
for item in data_test:
print(item, '-->', nn.sim([item])[0])

The full code is given in the file simple_neural_network.py. If you run the code, you will get two screenshot. The first screenshot represents the input datapoints:

The second screenshot shows the training progress:

Once you close the graphs, you will see the following printed on your Terminal:

If you locate these test data points on a 2D graph, you can visually verify that the predicted outputs are correct.

Constructing a multilayer neural network

In order to enable higher accuracy, we need to give more freedom to the neural network. This means that a neural network needs more than one layer to extract the underlying patterns in the training data. Let’s create a multilayer neural network to achieve that.

Create a new Python file and import the following packages:

import numpy as np 
import matplotlib.pyplot as plt
import neurolab as nl

In the previous two sections, we saw how to use a neural network as a classifier. In this section, we will see how to use a multilayer neural network as a regressor. Generate some sample data points based on the equation y = 3x² + 5 and then normalize the points:

# Generate some training data 
min_val = -15
max_val = 15
num_points = 130
x = np.linspace(min_val, max_val, num_points)
y = 3 * np.square(x) + 5
y /= np.linalg.norm(y)

Reshape the above variables to create a training dataset:

# Create data and labels 
data = x.reshape(num_points, 1)
labels = y.reshape(num_points, 1)

Plot the input data:

# Plot input data 
plt.figure()
plt.scatter(data, labels)
plt.xlabel('Dimension 1')
plt.ylabel('Dimension 2')
plt.title('Input data')

Define a multilayer neural network with two hidden layers. You are free to design a neural network any way you want. For this case, let’s have 10 neurons in the first layer and 6 neurons in the second layer. Our task is to predict the value, so the output layer will contain a single neuron:

# Define a multilayer neural network with 2 hidden layers; 
# First hidden layer consists of 10 neurons
# Second hidden layer consists of 6 neurons
# Output layer consists of 1 neuron
nn = nl.net.newff([[min_val, max_val]], [10, 6, 1])

Set the training algorithm to gradient descent:

# Set the training algorithm to gradient descent 
nn.trainf = nl.train.train_gd

Train the neural network using the training data that was generated:

# Train the neural network 
error_progress = nn.train(data, labels, epochs=2000, show=100, goal=0.01)

Run the neural network on the training datapoints:

# Run the neural network on training datapoints 
output = nn.sim(data)
y_pred = output.reshape(num_points)

Plot the training progress:

# Plot training error 
plt.figure()
plt.plot(error_progress)
plt.xlabel('Number of epochs')
plt.ylabel('Error')
plt.title('Training error progress')

Plot the predicted output:

# Plot the output 
x_dense = np.linspace(min_val, max_val, num_points * 2)
y_dense_pred = nn.sim(x_dense.reshape(x_dense.size,1)).reshape(x_dense.size)

plt.figure()
plt.plot(x_dense, y_dense_pred, '-', x, y, '.', x, y_pred, 'p')
plt.title('Actual vs predicted')

plt.show()

The full code is given in the file multilayer_neural_network.py. If you run the code, you will get three screenshot. The first screenshot shows the input data:

The second screenshot shows the training progress:

The third screenshot shows the predicted output overlaid on top of input data:

The predicted output seems to follow the general trend. If you continue to train the network and reduce the error, you will see that the predicted output will match the input curve even more accurately.

You will see the following printed on your Terminal:

Building a vector quantizer

Vector Quantization is a quantization technique where the input data is represented by a fixed number of representative points. It is the N-dimensional equivalent of rounding off a number. This technique is commonly used in multiple fields such as image recognition, semantic analysis, and data science. Let’s see how to use artificial neural networks to build a vector quantizer.

Create a new Python file and import the following packages:

import numpy as np 
import matplotlib.pyplot as plt
import neurolab as nl

Load the input data from the file data_vector_quantization.txt. Each line in this file contains six numbers. The first two numbers form the datapoint and the last four numbers form a one-hot encoded label. There are four classes overall.

# Load input data 
text = np.loadtxt('data_vector_quantization.txt')

Separate the text into data and labels:

# Separate it into data and labels 
data = text[:, 0:2]
labels = text[:, 2:]

Define a neural network with two layers where we have 10 neurons in the input layer and 4 neurons in the output layer:

# Define a neural network with 2 layers: 
# 10 neurons in input layer and 4 neurons in output layer
num_input_neurons = 10
num_output_neurons = 4
weights = [1/num_output_neurons] * num_output_neurons
nn = nl.net.newlvq(nl.tool.minmax(data), num_input_neurons, weights)

Train the neural network using the training data:

# Train the neural network 
_ = nn.train(data, labels, epochs=500, goal=-1)

In order to visualize the output clusters, let’s create a grid of points:

# Create the input grid 
xx, yy = np.meshgrid(np.arange(0, 10, 0.2), np.arange(0, 10, 0.2))
xx.shape = xx.size, 1
yy.shape = yy.size, 1
grid_xy = np.concatenate((xx, yy), axis=1)

Evaluate the grid of points using the neural network:

# Evaluate the input grid of points 
grid_eval = nn.sim(grid_xy)

Extract the four classes:

# Define the 4 classes 
class_1 = data[labels[:,0] == 1]
class_2 = data[labels[:,1] == 1]
class_3 = data[labels[:,2] == 1]
class_4 = data[labels[:,3] == 1]

Extract the grids corresponding to those four classes:

# Define X-Y grids for all the 4 classes 
grid_1 = grid_xy[grid_eval[:,0] == 1]
grid_2 = grid_xy[grid_eval[:,1] == 1]
grid_3 = grid_xy[grid_eval[:,2] == 1]
grid_4 = grid_xy[grid_eval[:,3] == 1]

Plot the outputs:

# Plot the outputs 
plt.plot(class_1[:,0], class_1[:,1], 'ko',
class_2[:,0], class_2[:,1], 'ko',
class_3[:,0], class_3[:,1], 'ko',
class_4[:,0], class_4[:,1], 'ko')
plt.plot(grid_1[:,0], grid_1[:,1], 'm.',
grid_2[:,0], grid_2[:,1], 'bx',
grid_3[:,0], grid_3[:,1], 'c^',
grid_4[:,0], grid_4[:,1], 'y+')
plt.axis([0, 10, 0, 10])
plt.xlabel('Dimension 1')
plt.ylabel('Dimension 2')
plt.title('Vector quantization')

plt.show()

The full code is given in the file vector_quantizer.py. If you run the code, you will get the following sccreenshot that shows the input data points and the boundaries between clusters:

You will see the following printed on your Terminal:

Analyzing sequential data using recurrent neural networks

We have been dealing with static data so far. Artificial neural networks are good at building models for sequential data too. In particular, recurrent neural networks are great at modeling sequential data. Perhaps time-series data is the most commonly occurring form of sequential data in our world. You can learn more about recurrent neural networks at http://www.wildml.com/2015/09/recurrent-neural-networks-tutorial-part-1-introduction-to-rnns. When we are working with time-series data, we cannot just use generic learning models. We need to characterize the temporal dependencies in our data so that we can build a robust model. Let’s see how to build it.

Create a new python file and import the following packages:

import numpy as np 
import matplotlib.pyplot as plt
import neurolab as nl

Define a function to generate the waveforms. Start by defining four sine waves:

def get_data(num_points): 
# Create sine waveforms
wave_1 = 0.5 * np.sin(np.arange(0, num_points))
wave_2 = 3.6 * np.sin(np.arange(0, num_points))
wave_3 = 1.1 * np.sin(np.arange(0, num_points))
wave_4 = 4.7 * np.sin(np.arange(0, num_points))

Create varying amplitudes for the overall waveform:

# Create varying amplitudes 
amp_1 = np.ones(num_points)
amp_2 = 2.1 + np.zeros(num_points)
amp_3 = 3.2 * np.ones(num_points)
amp_4 = 0.8 + np.zeros(num_points)

Create the overall waveform:

wave = np.array([wave_1, wave_2, wave_3, wave_4]).reshape(num_points * 4, 1) 
amp = np.array([[amp_1, amp_2, amp_3, amp_4]]).reshape(num_points * 4, 1)

return wave, amp

Define a function to visualize the output of the neural network:

# Visualize the output 
def visualize_output(nn, num_points_test):
wave, amp = get_data(num_points_test)
output = nn.sim(wave)
plt.plot(amp.reshape(num_points_test * 4))
plt.plot(output.reshape(num_points_test * 4))

Define the main function and create a waveform:

if __name__=='__main__': 
# Create some sample data
num_points = 40
wave, amp = get_data(num_points)

Create a recurrent neural network with two layers:

# Create a recurrent neural network with 2 layers 
nn = nl.net.newelm([[-2, 2]], [10, 1], [nl.trans.TanSig(), nl.trans.PureLin()])

Set the initializer functions for each layer:

# Set the init functions for each layer 
nn.layers[0].initf = nl.init.InitRand([-0.1, 0.1], 'wb')
nn.layers[1].initf = nl.init.InitRand([-0.1, 0.1], 'wb')
nn.init()

Train the neural network:

# Train the recurrent neural network 
error_progress = nn.train(wave, amp, epochs=1200, show=100, goal=0.01)

Run the data through the network:

# Run the training data through the network 
output = nn.sim(wave)

Plot the output:

# Plot the results 
plt.subplot(211)
plt.plot(error_progress)
plt.xlabel('Number of epochs')
plt.ylabel('Error (MSE)')

plt.subplot(212)
plt.plot(amp.reshape(num_points * 4))
plt.plot(output.reshape(num_points * 4))
plt.legend(['Original', 'Predicted'])

Test the performance of the neural network on unknown test data:

# Testing the network performance on unknown data 
plt.figure()

plt.subplot(211)
visualize_output(nn, 82)
plt.xlim([0, 300])

plt.subplot(212)
visualize_output(nn, 49)
plt.xlim([0, 300])

plt.show()

The full code is given in the file recurrent_neural_network.py. If you run the code, you will see two output figures. The upper half of the first screenshot shows the training progress and the lower half shows the predicted output overlaid on top of the input waveform:

The upper half of the second screenshot shows how the neural network simulates the waveform even though we increase the length of the waveform. The lower half of the screenshot shows the same for decreased length.

You will see the following printed on your Terminal:

Visualizing characters in an Optical Character Recognition database

Artificial neural networks can use optical character recognition. It is perhaps one of the most commonly sited examples. Optical Character Recognition (OCR) is the process of recognizing handwritten characters in images. Before we jump into building that model, we need to familiarize ourselves with the dataset. We will be using the dataset available at http://ai.stanford.edu/~btaskar/ocr . You will be downloading a file called letter.data. For convenience, this file has been provided to you in the code bundle. Let’s see how to load that data and visualize the characters.

Create a new python file and import the following packages:

import os 
import sys

import cv2
import numpy as np

Define the input file containing the OCR data:

# Define the input file 
input_file = 'letter.data'

Define the visualization and other parameters required to load the data from that file:

# Define the visualization parameters 
img_resize_factor = 12
start = 6
end = -1
height, width = 16, 8

Iterate through the lines of that file until the user presses the Esc key. Each line in that file is tab separated. Read each line and scale it up to 255:

# Iterate until the user presses the Esc key 
with open(input_file, 'r') as f:
for line in f.readlines():
# Read the data
data = np.array([255 * float(x) for x in line.split('\t')[start:end]])

Reshape the 1D array into a 2D image:

# Reshape the data into a 2D image 
img = np.reshape(data, (height, width))

Scale the image for visualization:

# Scale the image 
img_scaled = cv2.resize(img, None, fx=img_resize_factor, fy=img_resize_factor)

Display the image:

# Display the image 
cv2.imshow('Image', img_scaled)

Check if the user has pressed the Esc key. If so, exit the loop:

# Check if the user pressed the Esc key 
c = cv2.waitKey()
if c == 27:
break

The full code is given in the file character_visualizer.py. If you run the code, you will get an output screenshot displaying a character. You can keep pressing the space bar to see more characters. An o looks like this:

An i looks like this:

Building an Optical Character Recognition engine

Now that we have learned how to work with this data, let’s build an optical character recognition system using artificial neural networks.

Create a new python file and import the following packages:

import numpy as np 
import neurolab as nl

Define the input file:

# Define the input file 
input_file = 'letter.data'

Define the number of datapoints that will be loaded:

# Define the number of datapoints to 
# be loaded from the input file
num_datapoints = 50

Define the string containing all the distinct characters:

# String containing all the distinct characters 
orig_labels = 'omandig'

Extract the number of distinct classes:

# Compute the number of distinct characters 
num_orig_labels = len(orig_labels)

Define the train and test split. We will use 90% for training and 10% for testing:

# Define the training and testing parameters 
num_train = int(0.9 * num_datapoints)
num_test = num_datapoints - num_train

Define the dataset extraction parameters:

# Define the dataset extraction parameters 
start = 6
end = -1

Create the dataset:

# Creating the dataset 
data = []
labels = []
with open(input_file, 'r') as f:
for line in f.readlines():
# Split the current line tabwise
list_vals = line.split('\t')

If the label is not in our list of labels, we should skip it:

# Check if the label is in our ground truth 
# labels. If not, we should skip it.
if list_vals[1] not in orig_labels:
continue

Extract the current label and append it to the main list:

# Extract the current label and append it 
# to the main list
label = np.zeros((num_orig_labels, 1))
label[orig_labels.index(list_vals[1])] = 1
labels.append(label)

Extract the character vector and append it to the main list:

# Extract the character vector and append it to the main list 
cur_char = np.array([float(x) for x in list_vals[start:end]])
data.append(cur_char)

Exit the loop once we have created the dataset:

# Exit the loop once the required dataset has been created 
if len(data) >= num_datapoints:
break

Convert the lists into numpy arrays:

# Convert the data and labels to numpy arrays 
data = np.asfarray(data)
labels = np.array(labels).reshape(num_datapoints, num_orig_labels)

Extract the number of dimensions:

# Extract the number of dimensions 
num_dims = len(data[0])

Create a feedforward neural network and set the training algorithm to gradient descent:

# Create a feedforward neural network 
nn = nl.net.newff([[0, 1] for _ in range(len(data[0]))],
[128, 16, num_orig_labels])

# Set the training algorithm to gradient descent
nn.trainf = nl.train.train_gd

Train the neural network:

# Train the network 
error_progress = nn.train(data[:num_train,:], labels[:num_train,:],
epochs=10000, show=100, goal=0.01)

Predict the output for test data:

# Predict the output for test inputs 
print('\nTesting on unknown data:')
predicted_test = nn.sim(data[num_train:, :])
for i in range(num_test):
print('\nOriginal:', orig_labels[np.argmax(labels[i])])
print('Predicted:', orig_labels[np.argmax(predicted_test[i])])

The full code is given in the file ocr.py. If you run the code, you will see the following on your Terminal:

It will keep going until 10,000 epochs. Once it’s done, you will see the following on your Terminal:

As we can see in the preceding screenshot, it gets three of them right. If you use a bigger dataset and train longer, then you will get higher accuracy.

Source: Artificial Intelligence on Medium

(Visited 94 times, 1 visits today)
Post a Comment

Newsletter