Learn more about Russian war crimes in Ukraine.

What is the simplest neural network? One neuron

The basis of neural networks is the neuron. A neuron has some inputs, and based on those, either fires or doesn’t fire. We can interpret the decision to fire as a binary classification of the inputs into two classes.

To neuron whether to fire, the neuron follows a simple procedure: take each input, multiply it by a weight, sum those products, then fire if the sum is above a threshold. The specific behavior of a neuron is given by its weights and its threshold.

The following C code shows two neurons at work. One classifies numbers according to whether they are even or odd. Another neuron classifies numbers according to whether they are greater than five. Each is defined by specific weights and threshold.

The neuron only understands inputs as a vector of numbers, so we have to encode our inputs. To encode the natural numbers, I’ve encoded them in binary.

#include <stdbool.h>
#include <stdio.h>

bool neuron(float weights[4], float threshold, float input[4]) {
  float sum = 0;
  for (size_t i = 0; i < 4; i++) {
    sum += input[i] * weights[i];
  }
  return threshold < sum;
}

float is_odd_weights[]      = { 0, 0, 0, 1 };
float real_number_weights[] = { 8, 4, 2, 1 };

float four[]  = {0, 1, 0, 0};
float five[]  = {0, 1, 0, 1};
float six[]   = {0, 1, 1, 0};
float seven[] = {0, 1, 1, 1};

int main() {
  printf("4 is odd: %d\n", neuron(is_odd_weights, 0, four));
  printf("5 is odd: %d\n", neuron(is_odd_weights, 0, five));
  printf("6 is odd: %d\n", neuron(is_odd_weights, 0, six));
  printf("7 is odd: %d\n", neuron(is_odd_weights, 0, seven));

  printf("4 is greater than 5: %d\n", neuron(real_number_weights, 5, four));
  printf("5 is greater than 5: %d\n", neuron(real_number_weights, 5, five));
  printf("6 is greater than 5: %d\n", neuron(real_number_weights, 5, six));
  printf("7 is greater than 5: %d\n", neuron(real_number_weights, 5, seven));

  return 0;
}

What kinds of classifications can this pattern make? For example, it understands divisibility by two, but could we find weights which classify according to divisibility by three? Find out in a future episode.

What can computers do? What are the limits of mathematics? And just how busy can a busy beaver be? This year, I’m writing Busy Beavers, a unique interactive book on computability theory. You and I will take a practical and modern approach to answering these questions — or at least learning why some questions are unanswerable!

It’s only $19, and you can get 50% off if you find the discount code ... Not quite. Hackers use the console!

After months of secret toil, I and Andrew Carr released Everyday Data Science, a unique interactive online course! You’ll make the perfect glass of lemonade using Thompson sampling. You’ll lose weight with differential equations. And you might just qualify for the Olympics with a bit of statistics!

It’s $29, but you can get 50% off if you find the discount code ... Not quite. Hackers use the console!

More by Jim

Tagged . All content copyright James Fisher 2017. This post is not associated with my employer. Found an error? Edit this page.