(REALLY) simple neural network program

Question

This is a simple program to create neural networks. It only includes weighting of connections and activation values for the neurons. It doesn't include any learning feature of any kind, and it is really just a first attempt at creating something resembling a neural network.

Main.java

package uniliniarnetwork;
public class Main {
public static void main(String[] args) {
 Neuron inputNode = new Neuron(0,1,"InputNode",1);
 Neuron hiddenNode_One = new Neuron(0.5f,0.9f,"HiddenNode_One",1);
 Neuron hiddenNode_Two = new Neuron(0.5f,0.9f,"HiddenNode_Two",1);
 Neuron hiddenNode_Three = new Neuron(0.5f,0.9f,"HiddenNode_Three",1);
 Neuron outputNode = new Neuron(0,0.9f,"OutputNode",3);
 inputNode.connect(hiddenNode_One);
 inputNode.connect(hiddenNode_Two);
 inputNode.connect(hiddenNode_Three);
 hiddenNode_One.connect(outputNode);
 hiddenNode_Two.connect(outputNode);
 hiddenNode_Three.connect(outputNode);
 inputNode.input(1);
}
}

Neuron.java

package uniliniarnetwork;
import java.util.ArrayList;
public class Neuron {
private float activationValue, weight;
private String neuronName;
private ArrayList<Neuron> outputs = new ArrayList<Neuron>();
private float[] inputs;
int inputCounter = 0, nInputs;
public Neuron(float activationValue, float weight, String neuronName, int nInputs){
 this.activationValue = activationValue;
 this.weight = weight;
 this.neuronName = neuronName;
 this.nInputs = nInputs;
 inputs = new float[nInputs];
}
public void connect(Neuron neuron){
 outputs.add(neuron);
}
public void input(float inputValue){
 inputs[inputCounter] = inputValue;
 inputCounter++;
 if(inputCounter == nInputs){
 fire();
 }
}
public void fire(){
 float sum = 0;
 for(int i = 0; i < nInputs; i++){
 sum+=inputs[i];
 }
 float signal = sum*weight;
 if(signal > activationValue){
 for(int i = 0; i < outputs.size(); i++){
 outputs.get(i).input(signal);
 }
 } else{
 for(int i = 0; i < outputs.size(); i++){
 outputs.get(i).input(0);
 }
 }
 System.out.println(neuronName + ":" + signal);
}
}

Basically main creates Neuron objects from the Neuron class which have an activationValue and a weight value. I know it's not perfect and it doesn't include most of the most important features of a neural network.

My question is whether this is a good place to build from towards more advanced neural networks, for example those which can classify images. It would be greatly appreciated if you could explain in terms a high-school student (with only a rudimentary understanding of calculus) would understand.

• πάντα ῥεῖ: "it doesn't include any learning feature of any kind" TL;DR; what's the actual point of it then? Building a dead brain?
• Adam Fraser: it doesn't really have a point. its only purpose is to serve as a learning tool to see what needs to change in order to progress to real or even useful neural networks.
• Thomas Wagenaar: @πάνταῥεῖ did he say he's not going to implement any learning feature? If you would have read his question you might have noticed that this might be his place to start building more advanced nerual networks. And maybe he doesn't need to incoperate a learning feature at all, maybe he wants to copy existing networks by directly copying their weights and biases.
• holroy: Don't be to eager to accept answers, as that is a turn-off for other reviewers: In general, up vote answers you find helpful as they come along, and after a few days choose the most helpful answer to you. And possibly change the accepted answer if a new answers comes along later on which is even more helpful or better suited in your particular case.

Answer 1

You don't typically use a Neuron data structure for Neural Nets, instead you use matrices and vectors (for the inputs, weights, and outputs). Moreover, activation functions are done using Sigmoid, Hyperbolic Tangent, or comparable functions. Below is a very simple python example using Numpy for the matrix multiplication; it also has the back propagation algorithm in place for it to learn.

import numpy as np
import scipy.special
# simple neural network class
# it has one input layer, one output layer, and a single hidden layer
# nodes are connected to all subsequent nodes where such is possible
class NeuralNet:
# constructor
# each parameter is a number representing the number of given objects
def __init__(self, input_nodes, output_nodes, hidden_nodes, learning_rate):
 self.input_nodes = input_nodes
 self.hidden_nodes = hidden_nodes
 self.output_nodes = output_nodes
 self.lr = learning_rate
 # set the weights to random values within a gaussian distribution
 # this way we get different weights, but none which bias or saturate the system
 self.wih = \
 np.random.normal(0.0, pow(self.hidden_nodes, -0.5), (self.hidden_nodes, self.input_nodes))
 self.who = \
 np.random.normal(0.0, pow(self.output_nodes, -0.5), (self.output_nodes, self.hidden_nodes))
 self.activation_function = lambda x: \
 scipy.special.expit(x) # activation is a sigmoid
 pass
# one iteration of training given inputs and desired targets
def train(self, inputs_list, targets_list):
 # inputs and targets
 targets = np.array(targets_list, ndmin=2).T
 inputs = np.array(inputs_list, ndmin=2).T
 # outputs
 hidden_outputs = self.activation_function(np.dot(self.wih, inputs))
 outputs = self.activation_function(np.dot(self.who, hidden_outputs))
 # error
 output_errors = (targets-outputs)
 # back-propagated hidden layer error
 hidden_errors = np.dot(self.who.T, output_errors)
 self.who += \
 self.lr * np.dot((output_errors * outputs * (1.0 - outputs)), np.transpose(hidden_outputs))
 self.wih += \
 self.lr * np.dot((hidden_errors * hidden_outputs * (1.0 - hidden_outputs)), (np.transpose(inputs)))
 pass
# function that will query the neural net with an input list,
# returning the outputs for the given inputs
def query(self, inputs_list):
 # we convert our list of inputs into a matrix
 inputs = np.array(inputs_list, ndmin=2).T
 # outputs of hidden layer are as follows:
 # dot product of input matrix by first layer weights matrix
 # passed through our sigmoid activation lambda function
 hidden_outputs = self.activation_function(np.dot(self.wih, inputs))
 # final outputs' code is more or less the same as hidden outputs'
 final_outputs = self.activation_function(np.dot(self.who, hidden_outputs))
 return final_outputs

• holroy: Could you enhance on how this is an answer to the OP, and a review of his code/question?
• Adam Fraser: thanks this is really helpful and i think i know what i should be working on, probably my understanding of maths more than anything.
• 18AdrianoH: We do things using matrices and vectors because its generally faster and easier to use.
• 18AdrianoH: The foward feeding part is pretty simple because It will just multiple each neuron's output by a weight and when it goes into the next neuron that will pass it through an activation function with its other inputs (which is really just used to keep the sum if inputs within a set of bounds and to keep changes reasonable).
• 18AdrianoH: The back propagation algorithm just looks at the derivative (slope) of the cost function at each point (how off we are from the "right answer"; in the layers which are not output this is derived from neurons' contribution to the final output due to weights) and just moves down the slope, like we would walk down a hill (since we defined this function to determine how "wrong" we were, if we just walk down it we are less "wrong").

Answer 2

I'm not really into Neural Networks, but I do read code and I do have some suggesting for what to do next related to this code.

Code and Style Comments

• Number in variable names, not good – Even though you spelt out the number, still having numbers in a variable names indicates that you might be doing something wrong. I'm thinking of the hiddenNode_One& co.

• Please fix your indentation – This excerpt is taken directly from your code:

  public class Neuron {
  private float activationValue, weight;
  private String neuronName;
  private ArrayList<Neuron> outputs = new ArrayList<Neuron>();
  private float[] inputs;
  int inputCounter = 0, nInputs;
  public Neuron(float activationValue, float weight, String neuronName, int nInputs){
   this.activationValue = activationValue;
  

  This looks like a list of variable declaraction outside of classes and methods, and only with a second look it's possible to see that you are actually declaring a class Neuron, its class variables, and a class constructor of public Neutron(). And finally you indent for the method definition. Here is the same code with better indentation:

  public class Neuron {
   private float activationValue, weight;
   private String neuronName;
   private ArrayList<Neuron> outputs = new ArrayList<Neuron>();
   private float[] inputs;
   int inputCounter = 0, nInputs;
   public Neuron(float activationValue, float weight, String neuronName, int nInputs){
   this.activationValue = activationValue;
  

  This looks a lot better, and it's easier to follow the program flow.

• Is it wise to lock the number of inputs? – In your constructor you lock the number of inputs for any given Neuron, which might lead to confusing situations later on if you want to reorder your network.

• Feature: To many input()'s will trigger index error – Since you keep increasing the inputCounter whenever this method is called, you'll run out of array elements to update, and will finally trigger an index error.

  In the same code, you'll only fire() once when the inputCounter exactly equals the preset number of inputs. So if the input of a Neuron changes afterwards, the network is never updated. Neither of these seems to be correct.

• Always generate input() even when not over the activation value? – In light of preceding point, it seems kind of strange to do the input(0) sequence, as that might for some networks trigger the index error before it should. Wouldn't it be better to leave out the no input part?

  That is, unless that is a requirement due to the value swapping from over and under the activationValue during the lifetime of the network, in which case you really need to address the case of connecting inputs to the Neuron receiving the input.

• Why delay the summation of inputs, and the signal value – In my mind it would be better to update a sum and the signal value whenever receiving an input, instead of when doing the fire().

  Imaging having a network of thousands of nodes, and you having to calculate this for each fire event you trigger, even though the input possibly didn't change? To me it makes more sense to do this within the input() having a sum variable, and to have a class variable (or method) holding (or returning) the sum * weight.

  Do however know that this would require proper knowledge of which input is being updated, so that if you receive a secondary input on that Neuron you'll correctly adjust the sum, and not blatantly keep adding the new value.

• Order of calculation and output – Most likely your System.out.println(neuronName + ":" + signal) should be replaced with a call to some logger, but I would also put this one in front of the signal > activationValue loops. As it is in a bigger network you would get the output of the latter nodes, before the first nodes. For the output to make sense, it would be wiser to have the output directly after the calculation of signal.

Some General Thoughts on Neural Networks

In my mind a neural network is something dynamic, which could/should easily be able to change and reconnect towards other neurons (or nodes). So here are some concerns I have related to your code:

• No visualization of the neural network – It's not possible to visualize your network. I think it would've been nice to have a way to visualize it, so that you can see how it is connected with which triggers and so on.
• No connection of the input towards the neurons – There is no connection between the neurons connected to another neuron and the input. In other words, if the hiddenNode_Two decides to change its input, and thusly it fires an output, you don't have a way to know which of the inputs of the outputNode actually changed. This doesn't seem correct.

• No dynamic in neuron connection – You've got no methods to remove and change your network. In addition you've limited the neuron at construction time to a given set of inputs. Thusly disallowing a re-ordering of your network based on any future learning.

  Imaging having a network doing something related to alphabetic calculation. At the start you could start out with a few nodes, but after a while you see the need to split the a-e node as it is overworked. Your code as it stands, would require a full reconstruction, and not a simple replacement of that node with a few extra nodes.

One way to implement this interconnection, and related sums and signal values, could be to introduce a static list of Neurons, and extend each Neuron with dynamic lists of indexes into this list to describe its inputs and outputs.

This would allow for methods identifying all the inputs or outputs, by checking whether the input or output list for a given neuron is empty or not. You could also make static methods traversing based on input nodes describing the paths to new nodes (possibly with circular node connection detection) until you reaches the output nodes.