In the first example, we demonstrate learning a nonlinear function. Let
f(x1,x2)=x1sin(3x2)cos(2x1x2). |
We create a neural network with three hidden layers and train it to predict the value f(x1,x2)
given the input vector x=(x1,x2) in the square S=[−1,1]2.
We use Adam with the default parameters and set the
weight decay factor to 10−4. The block size is set to 100 (we will use full-batch gradient descent,
meaning that the network will process training samples in bulks of 100 samples at once).
The network has 30+20+10 neurons in hidden layers (not counting the bias neurons).
We store the network topology in t.
Input:
t:=[2,30,20,10,1]:; |
net:=neural_network(t,momentum=adaptive,weight_decay=1e-4,block_size=100)
|
Output:
"a neural network with input of size 2 and output of size 1" |
Next we create 5000 training samples in S by using the uniform random variable U(−1,1). Input:
f(x):=x[0]*sin(3*x[1])*cos(2*x[0]*x[1]) |
data:=ranm(5000,2,uniformd(-1,1)):; res:=apply(f,data):;
|
Now we have data points in data and the corresponding function values in res.
In a similar manner, we create a collection of another 100 samples, which will be kept
unseen by the network and used for testing its accuracy. Input:
test_data:=ranm(100,2,uniformd(-1,1)):; test_res:=apply(f,test_data):;
Now we train the network using 2500 epochs. We test the accuracy in intervals of 250 epochs.
Input:
for epoch from 1 to 2500 do |
net:=train(net,data,res); |
if irem(epoch,250)==0 then |
print(mean(net(test_data,test_res))); |
fi; |
od:;
|
Possible output:
0.00211048030912 |
0.000199757341385 |
8.70954607301e-05 |
6.21486919568e-05 |
5.22746108944e-05 |
5.0011469063e-05 |
4.91138941048e-05 |
4.81631000381e-05 |
4.86611973063e-05 |
4.79773288935e-05 |
|
Evaluation time: 16.85
|
Note that half-MSE is used as the error function by default (this is a regression model).
Now we generate a random point x0 in S and compute the predicted and exact
value of f(x0).
Input:
x0:=ranv(2,uniformd(-1,1))
Possible output:
⎡
⎣ | −0.402978600934,−0.836934269406 | ⎤
⎦ |
Input:
net(x0), f(x0)
Possible output:
0.18592080555,0.185619807512 |
To plot the learned surface, use the command:
plot3d(quote(net([x1,x2])),x1=-1..1,x2=-1..1)