For instance, a custom loss function would greatly improve training times when using reinforcement learning.

OMG 🤣 Soooo... A lot more digging, and I found out that this was already solved 🥰

btw the work is being done in this branch. Already have a working implementation for NeuralNetworkGPU, but I still need to implement it for NeuralNetwork/CPU-based nets. As of now, much is left to be desired, but it works 🥰 Currently implementing it for the CPU before adding a few parameters to the loss function to make it far more useful for training optimization. The end-goal API I'm thinking of looks like this (InputType is defined by NeuralNetwork and NeuralNetworkGPU):

function loss(
 this: IKernelFunctionThis,
 actual: KernelOutput, // The actual output data of the neural network during this run.
 expected: KernelOutput, // The output data expected to be produced by the neural network given the provided input data.
 inputs: InputType, // The input data provided to entire neural network (the data passed into the `NeuralNetwork.run(input)` method).
 state: KernelOutput[][], // The current training state of the neural network; see below for more information.
) {
 return expected[this.thread.x] - actual[this.thread.x];
}

Where:

• this: IKernelFunctionThis is the GPU.js kernel function's this value.
• actual: KernelOutput is the current actual output data produced by running the neural network.
• expected: KernelOutput is the output data expected to be produced when running the neural network if it were perfectly trained and optimized.
• inputs: InputType is the input data that was provided to the neural network this run.
• state: KernelOutput[][] is a state matrix. To better understand the state matrix, consider this pseudo-indexer: property = KernelOutput[layer][neuron] Each value in the first order of the array corresponds to a layer of the neural network. Each value in the second order corresponds to a neuron in that layer. Lastly, each value returned when indexing the second order of the state matrix corresponds to that neuron's state tensor. In this context, the neuron's state tensor can be defined as: "a tensor containing information specific to any particular neuron within an artificial neural network." While that definition is quite ad-hoc seeing as I just came up with it, I have to admit that it is some of my finest improv 😂

The aspect of the state parameter of the loss function that excites me the most is that it paves the way for more implementing far more advanced training algorithms than the library currently supports, and with GPU-acceleration at that!! The possibilities are even more limitless with the CPU algorithms, but performance limits their applications at the moment (no where near too limited to be useful though!!! So many applications 🥰).

An example designed to optimize training for XOR might look like:

function loss(
 this: IKernelFunctionThis,
 actual: KernelOutput, // The actual output data of the neural network during this run.
 expected: KernelOutput, // The output data expected to be produced by the neural network given the provided input data.
 inputs: InputType, // The input data provided to entire neural network (the data passed into the `NeuralNetwork.run(input)` method).
 state: KernelOutput[][], // The current training state of the neural network; see above for more information.
) {
 // Perform a standard loss function as our foundation to build upon.
 let neuronLoss = expected[this.thread.x] - actual[this.thread.x];
 // if ( o == i0 ^ i1 ) then return 10% of the loss value.
 // Otherwise, return the full loss value.
 if (Math.round(actual[0]) === Math.round(inputs[0]) ^ Math.round(inputs[1])) neuronLoss *= 0.1;
 return neuronLoss;
}

which allows for you to train a neural network to predict XOR calculations without ever even having been fed a data set. Just stream random data as training input, allow it to produce random output at first, and train on the error from the calculation. Instead of telling the network, "Hey! There's a bounty reward for whoever can solve for Y given X," you can instead define a rule to determine what is or is not good behavior, or even create a hybrid loss function. The above kernel function is a simple example of a hybrid loss function that I kind of just tossed together to train a standard feed-forward network to solve for XOR by using the standard XOR training set. In my experience, this greatly improves training times when done correctly.

Sending tons of positivity and good vibes your way friend 💕

I just realized that a KernelOutput[][] likely isn't possible given that GPU.js can currently only handle arrays up to rank 3. Perhaps a single network-wide KernelOutput then. It's already up to the loss function implementer to consider the neural network's topology/shape, and simply using KernelOutput allows the loss function implementer to have greater control over the training algorithm 😊

Okay, this is honestly the coolest feature I've ever added to anything. This has me so excited!!! Like, to give you an idea of why I'm so excited: A custom loss function made training an autoencoder much faster, and within a single NeuralNetwork or NeuralNetworkGPU meaning that the current autoencoder class, AE could be split into Autoencoder and AutoencoderGPU where each just directly inherits from NeuralNetwork and NeuralNetworkGPU respectively, making them play much more nicely with the existing API 🥰 Okie, PR time 😊 💞