Syntiant NDP 101 – Always-On Low Power Speech Recognition

Its a try ….curious if it will work .. fresh from California…

Always-On Low Power Speech Recognition with Syntiant NDP 101

NDP 101 – fresh from Digitkey – what do I want to now?

I will integrate the Syntiant TinyML NDP101 board with my latest project: GoodWatch – a super smart voice controlled watch.

Currently for GoodWatch I am using an ESP32 as MCU for speech recognition. It is working with 10 numbers and two keywords (yes, no). But – to get good results for so many keywords and to get a quick response – I needed to trick a bit: Its only listening to my words. If you want to learn more on ML for Edge-Device, have a look at: Overview of ML on Edge – Devices .You can see how far I got with an ESP32 here:

Goodwatch – eInk Alarm Clock with Voice Recognition

With this project I have two objectives:

  • Wake up the alarm clock with a key word to set-up the Alarm, get it on 100% voice controlled operation
  • Enable speech recognition not only for my voice and reach a highly reliable speech recognition (otherwise setting up the old clock will be faster 🙂

Some facts on the TinyML Development Board

Board Overview

I have ordered the development “Syntiant TinyML”, it is a combination of the NDP101 processor combined with a Cortex-M0+ 32bitARM MCU, a SPH0641LM4H microphone, a LIPO power connector and USB plug- so ready to go, in more detail:

• Neural Decision Processor: NDP101
• Host processor: SAMD21 Cortex-M0+ 32bit low power ARM MCU, including:
    ◦ 256KB flash memory
    ◦ 32KB host processor SRAM
• Board power supply: 5V micro-USB or 3.7V LiPo battery
• 5 Digital I/Os compatible with Arduino MKR series boards
• 1 UART interface (included in the digital I/O Pins)
• 1 I2C interface (included in the digital I/O Pins)
• 2MB on-board serial flash
• 48MHz system clock
• One user defined RGB LED
• uSD card slot (uSD card not included)
• BMI160 6 axis motion sensor
• SPH0641LM4H microphone

The board has two components – a normal low power ARM MCU with USB connection (for communication, control, loading the models…) and a SPI connection to the heard of the system – the NDP101 MCU.

About the NDP1010 MCU: Its main component is a neural network with fixed structure: 3 Dense Layers of 256 neurons and 3 Dropout layers, the input layer must always have 1600 features.

Data recorded from the microphone is directly processed by the Audio Frontent (16 bit) , the “Holding Tank” works as a rolling buffer of up to 3 seconds of speech. The feature extractor performs a pretty much standard MEL (log-mel LMFB) transformation before using the neural network. You can read more about his here.

I guess, the small MCU is used to manage the communication via SPI to the outside world.

Software

The story just started – its available at: https://github.com/happychriss/Goodwatch_NDP101_SpeeechRecognition

This is an ongoing project, I have no clue if it will work out or how long it will take. I will keep posting on the progress – starting the next days with the development set-up and the first steps.

  • Kick-Off: Setup Platformio / CLION to work with EdgeImpulse and TinyML NDP101 (24.11.2021)

    In this short post I summarize my development environment using Platformio & Clion. I have adjusted platformio.ini file to include libraries and files needed – so you can start. EdgeImpulse provides a similar setting that fits when you are working with the Arduino IDE.

    Background

    I am using Platformio as development platform for embedded projects and CLION as my IDE. Both are integrated and together they provide a consistent way for programming, compiling and deployment – independent from the MCU/CPU used. I try to use the Arduino Libraries- as to a certain degree the provide platform independence. In addition many libraries are available for Arduino. To facilitate the Machine Learning I rely on Edgeimpulse – an cloud platform that supports the e2e machine learning cycle – so in a nutshell:

    • Platformio – Platform for embedded development
    • Clion – Integrated Development Environment with Platformio Integration
    • EdgeImpulse – Development platform for machine learning on edge devices

    The magic salt in the soup: EdgeImpulse

    EdgeImpulse takes away the two most complicated parts: The model design/training and the feature extraction. Its a science on its own to run a continuous inversion process on an small MCU (e.g ESP32) that includes recording the voice, running a Fourier transformation, applying a MEL transformation and finally to feed the features into the model. As the feature extraction during training (running on the cloud or on a desktop PC) and later during inversion (running on the ESP) must follow the exact same rules from logic AND timing persective- this adds additional complexity for embedded ML projects. Syntiant has partnered with EdgeImpulse – so the feature extraction and inversion process has been tailored for the NDP 101 – there is no need to deep dive into its architecutre.

    So..what do I get and how do I build actually something?

    Once training is completed (Detailed description from EdgeImpulse) EdgeImpulse gives you two options to download the software on your device: You can directly download a binary image and test your model, or – and here its getting really interesting – to download source code that contains:

    • Feature Extraction SW (DSP…)
    • Trained Model
    • Model Inversion (feed feature into model and run the inversion)
    • Library to integrate the “model decision=inversion result” into the c-code
    • Example program using all above based on Arduino framework – actually showing its working
    • For NDP101 it also contains the libraries to directly communicate to the NDP101 via SPI commands (load model, run model…..). I am still exploring the different commands and options

    SourceCode – Ready To Go Package on GitHub

    My initial template version for this configuration is available on GitHub https://github.com/happychriss/Goodwatch_NDP101_SpeeechRecognition

    To build this I have downloaded the Syntiant/EdgeImpulse repository and adjusted the file structure to the Platformio build system, in addition I have adjusted the platformio.ini file to include needed libraries.

    Please note that you will need to patch one of the libraries, the baseline for my work and also instructions how to patch the local libraries can be found here.

    A nice goody is the script download_model.sh – that connects to EdgeImpulse downloads a *build* model and puts the files in the right folders (./src/edgeimpulse) – so you can build your binary fresh from the model and upload to the board. You have to add a run-configuratio in CLION, so you can start the script directly. The API code for authorization on EdgeImpulse needs to be added to the paramerters.

    This is, how it looks like – when all is set-up. Enjoy developing.