No ratings yet.

brein-chip-ai-neuromorphic-1232x856

GPT-3 is a promising new program developed by OpenAI, a startup founded by (who else than) Elon Musk. GPT-3 is a linguistic neural network, which in essence is nothing more or less than a stochastic language model that predicts the most likely word or token to follow a prompted sequence, just like GPT-2 is. However, what truly sets GPT-3 apart is its scale, containing 175 billion parameters in contrast to its GPT-2’s 1.5 billion parameters (arXiv, 2020). This gives rise to new and surprising possibilities, which may possibly have implications for the skills that you are developing right now.

Whereas you may think that the scale of a language model merely improves the accuracy of its language interpretation and prediction, which would come down to an embellished auto-completer, GPT-3 can do more. It can actually generate differently formatted responses to whatever prompts it is given. This means that it does not only produce text that seems written by humans, but can give surprisingly accurate answers to questions, as long as they are given a few examples and a prompt. This gives rise to “few-shot” learning, something that its predecessor(s) weren’t capable of. Without being fine-tuned to perform a specific task, GPT-3 may perform these tasks as effectively as the traditional domain-specific programs that were written to perform these tasks (ZDNet, 2020). Such a general utility, low effort, natural language-prompted program provides interesting opportunities.

Training and fine-tuning language models to perform specific tasks are strenuous endeavours that require lots of data, human capital, and patience. With the scale of GPT-3, such training is taken out of the equation; the newest language model by OpenAI requires no gradient updates or fine-tuning of the model. This means that the GPT-3 can be deployed and grows more effective as it accesses more data.

Whereas GPT-3 is only accessible to a group of Beta users, it could potentially spur another wave of automation. Customer support or content writing are the most obvious tasks that can be automated, but GPT-3 can do more than that. The linguistic neural network can actually infer and generate code in different programming languages, to create functional scripts (DaleonAI, 2020). Just consider the applications thought of by the following Twitter users:

@sharifshameem with…
- a layout generator that generates JSX code: https://twitter.com/sharifshameem/status/1282676454690451457
- a todo list app by simply describing how it should work:
  https://twitter.com/sharifshameem/status/1284421499915403264
@mattshumer_ with…
- an automatically built machine learning model, by simply describing the dataset and required output to GPT-3:
  https://twitter.com/mattshumer_/status/1287125015528341506

Does this make you second-guess whether choosing the BIM: Data Science track is still the right choice? Well, my guess would be not to overthink it. There are still a number of limitations to GPT-3 that limit it from being viable for enterprise adoption (ZDnet, 2020).

GPT-3 …

becomes repetitive and loses coherence throughout longer texts.
does not possess common sense; it is not a ‘reasoning’ machine, although some answers may lead you to think it is. It is a stochastic linguistic model at its core, and cannot reason through logic.
has variable output quality; some answers are amazingly accurate, whereas others are complete nonsense.
operates on ‘garbage in, garbage out’. Some prompts will be more effective at retrieving high-quality answers than others.
amplifies human biases and can sneak in racism, sexism other types of discrimination and prejudice throughout its prompted answers.
drives out creative answers that are sometimes needed; the program only follows the ‘fat tail’ of the probability distribution, due to its stochastic nature.
cannot be tuned to company-specific data, and is difficult to specialize for an industry.
1. Some start-ups, such as Sapling, may bridge this gap. However, we don’t know whether it is even possible as of yet.
may lead people to infringe copyrights; GPT-3 scrapes its data from different online sources, and does not consider whether these are copyrighted when producing its answers.
is a black box for its users. This becomes particularly worrying when considering its variable output quality, amplification of human biases, copyright issues.
requires tremendous computational power and investments to run. It is in no way commercially viable as of yet.

So should you then not care at all? Well, not really.

This could be a new chapter in machine learning. Consider the synergies of linguistic neural networks with neuromorphic systems (link to my other blog discussing this: https://digitalstrategy.rsm.nl//2020/09/29/biomimicry-from-neural-networks-to-neural-architecture/) integrating audio and image processing in future versions of GPT-x, and running these algorithms at the edge rather than the cloud.

Who knows what our future may hold. Perhaps in a decade or two, we will all be working with virtual employee assistants that integrate data flows from all of your enterprise systems, that can query whatever system you want through conversational prompts, or generate its own code based on our description of the end product. Imagine asking your Siri-on-steroids to optimize your scheduling, setting agendas for meetings, interpreting data for decision-making, and performing your customer due diligence work.

However, we are not there yet by far. For now, learning how to program and analyse data still seems like a great choice. Just make sure to keep an eye out for future developments.

References
ZDNet. (2020) https://www.zdnet.com/article/what-is-gpt-3-everything-business-needs-to-know-about-openais-breakthrough-ai-language-program/ [Accessed October 6, 2020]
arXiv. (2020) https://arxiv.org/abs/2005.14165 [Accessed October 6, 2020]
DaleonAI. (2020) https://daleonai.com/gpt3-explained-fast [Accessed October 6, 2020]

5/5 (3)

Biomimicry: From Neural Networks to Neural Architecture

Biomimicry is not new; the philosophy is that nature has already solved some of the puzzling problems that we humans are facing today. Just looking at a few examples—bird beaks as inspiration for trains, gecko toes for adhesives, whales for wind turbines, and spiders for protective glass (Interesting Engineering, 2018)—leads us to indeed conclude that nature can help us solve difficult problems. However, what about our own nature; what problems could human biology inspire us to solve?

Well, there is one problem that we are facing in the field of computer science, which I am fairly sure you have heard of: the nearing ‘death’ of Moore’s law. Gordon E. Moore predicted, in 1965, that “the number of transistors per silicon chip doubles every year” (Brittanica, n.d.). However, we are nearing the limits of physics when it comes to scaling down transistors and packing them closer together on a traditional computer chip (Technology Review, 2020); going any denser would cause overheating issues.

There are plenty of problems that require vastly more computational power than we possess today, for example in the fields of physics, chemistry and biology, but also in more socio-technical contexts. Our traditional computer chips, built on Von Neumann architecture, do not pack the power to solve these and more problems; traditional chips even struggle with tasks as image and audio processing. Perhaps the biggest flaw is the infamous ‘Von Neumann bottleneck’.

This, amongst other reasons, has been inspiring researchers to pursue a different type of architecture, one that is more energy efficient, packs more processing power, and get rids of a particular bottleneck between processing and memory retrieval (but more on that below). One promising field of research is that of neuromorphic architecture: a design that mimics the architecture of the human brain.

Traditional chips

Von Neumann architectures – i.a. what your laptop and mobile phone is built on – have computer chips with master clocks that, at each tick, evaluate a binary input and pass on a binary output through the logic gates in the transistors of the chip. The processor can only be on or off and often stands idle while waiting to fetch information from the separated memory, which makes them very energy inefficient and gives rise to the ‘Von Neumann bottleneck’. The latter comes down to the problem that, no matter how much processing power grows, if ‘transfer rates’ (the rates at which memory is retrieved) stay the same, the latency will not improve (All About Circuits, 2020). This clock-driven, binary architecture stands in stark contrast to the architecture of neuromorphic chips.

Neuromorphic chips

Neuromorphic chips contain a spiking neural network, the artificial neurons within which are only activated when signals reach an activation threshold, remaining at a low power use-baseline otherwise. These signals, which are electric pulses, are fired when sensory input changes. The implications of a signal depends on the number of spikes within a certain period of time, as well as the design of that specific chip. These signals are gradient rather than binary, which means that they – via weighted values – can transfer more information per signal than a bit can. Overall, the design lends itself excellently for processing sensor data, including speech, image and radar inputs. We currently see that such sensory inputs are processed on neural networks, but on traditional architecture. The hardware of neuromorphic chips, as the name may give away, resembles neural networks, which adds the benefit of running AI models at vastly higher speeds than C/GPUs can (IEEE, 2017).

The artificial neurons of a neuromorphic chip, or synaptic cores, operate in parallel. This means that multiple neurons can be activated and activate other neurons at the same time (Psychology Today, 2019). This makes neuromorphic chips incredibly scalable—since you can increase the amount of artificial neurons—as well as fault-tolerant, since neurons can find other synaptic routes (via other neurons) when another neuron breaks. This mimics neuroplasticity in the human brain.

Another quintessential aspect of neuromorphic chips is that memory and computation are tightly coupled. Whereas traditional chips require external memory for non-volatility, this type of memory is inherent to the design of neuromorphic chips (IBM, n.d.). The artificial neurons within a chip are connected by memristors that resemble artificial synapses. These memristors pack non-volatile memory, because they ‘remember’ the electric charge that has previously flown through it, as well the direction in which it has been sent (MIT, 2020). This non-volatility means that memristors retain their information even after the device is shut off.

Players to watch

The neuromorphic computing industry is consolidated and built upon traditional computer engineering capabilities. In my view, there are three chips to watch in the field of neuromorphic computing: BrainChip’s Akida, IBM’s TrueNorth, and Intel’s Loihi.

BrainChip’s Akida consists of 80 neuromorphic processing units that amount to 1.2 million neurons and 10 billion synapses (Nextplatform, 2020).
IBM’s TrueNorth consists of 4096 cores that amount to 1 million neurons and 250 million synapses (CACM, 2020).
Intel’s Pohoiki integrates 768 Loihi chips and amounts to 100 million neurons (Intel, 2020).

While Intel’s Pohoiki—a neuromorphic systemic aggregation of Loihi chips—is still in the research phase and only oriented to researchers, its 100 million neurons make it the most advanced neuromorphic system as of today (InsideHPC, 2020). It can do specific tasks up to 1,000 times faster and 10,000 times more efficiently than conventional processors (Intel, 2020). In terms of the amount of neurons inside, Intel’s Pohoiki resembles a small mammal. In addition, Intel (2020) claims that the neuromorphic system does not only fulfil AI purposes, but a wide range of computationally difficult problems.

Practical considerations

Neuromorphic chips are energy efficient, run AI models more efficiently than traditional architectures, are scalable, and reduce latency by tightly couple processing and memory. These properties make neuromorphic chips fit to run AI models at the edge rather than the cloud, which can be valuable for application in (i.a.) autonomous cars, industrial (IoT) environments, smart cities, cybersecurity, embedded video and audio, and in optimization problems such as minimal risk stock portfolios (Nextplatform, 2020; Intel, 2020). In addition, the energy efficient and compact design could enable deep learning to become embedded inside devices such as mobile phones. This could drastically improve natural language processing in day-to-day applications – just imagine Siri actually understanding your question and providing a helpful answer!

However, we are not there yet. There are still plenty of challenges, amongst which is developing the most efficient learning algorithms to be ran on the neuromorphic chips. Neuromorphic chips are still in their infancy, and overcoming technical hurdles will not be the only challenge (Analytics Insight, 2020); ethical concerns surrounding biomimicking hardware already exist, and should be expected to expected to intensify as the technology gains traction and its capabilities grow.

As of now, neuromorphic hardware is not commercially viable yet, but that does not mean we should not pay attention to it.

In the face of all this exciting uncertainty, I will conclude with some food for thought. Please let me know in the comments what your opinion are on (one of) the following three questions:

Do you think neuromorphic chips possess potentially transformative power to the nature of work, or even our day-to-life? Why?
What type of (business applications) do you see for hyper efficient neural network processing at the edge?
Can you think of any problems that we have pushed forward along the uncertain and lengthy path of quantum computing research, that may be solved earlier by neuromorphic computing?

References

Will GPT-3 make your programming skills obsolete?

6

Please rate this

Biomimicry: From Neural Networks to Neural Architecture

29

Biomimicry: From Neural Networks to Neural Architecture

Traditional chips

Neuromorphic chips

Players to watch

Practical considerations

References

Please rate this