In this short post I summarize my development environment using Platformio & Clion. I have adjusted platformio.ini file to include libraries and files needed – so you can start. EdgeImpulse provides a similar setting that fits when you are working with the Arduino IDE.

Background

I am using Platformio as development platform for embedded projects and CLION as my IDE. Both are integrated and together they provide a consistent way for programming, compiling and deployment – independent from the MCU/CPU used. I try to use the Arduino Libraries- as to a certain degree the provide platform independence. In addition many libraries are available for Arduino. To facilitate the Machine Learning I rely on Edgeimpulse – an cloud platform that supports the e2e machine learning cycle – so in a nutshell:

• Platformio – Platform for embedded development
• Clion – Integrated Development Environment with Platformio Integration
• EdgeImpulse – Development platform for machine learning on edge devices
  

The magic salt in the soup: EdgeImpulse

EdgeImpulse takes away the two most complicated parts: The model design/training and the feature extraction. Its a science on its own to run a continuous inversion process on an small MCU (e.g ESP32) that includes recording the voice, running a Fourier transformation, applying a MEL transformation and finally to feed the features into the model. As the feature extraction during training (running on the cloud or on a desktop PC) and later during inversion (running on the ESP) must follow the exact same rules from logic AND timing persective- this adds additional complexity for embedded ML projects. Syntiant has partnered with EdgeImpulse – so the feature extraction and inversion process has been tailored for the NDP 101 – there is no need to deep dive into its architecutre.

So..what do I get and how do I build actually something?

Once training is completed (Detailed description from EdgeImpulse) EdgeImpulse gives you two options to download the software on your device: You can directly download a binary image and test your model, or – and here its getting really interesting – to download source code that contains:

• Feature Extraction SW (DSP…)
• Trained Model
• Model Inversion (feed feature into model and run the inversion)
• Library to integrate the “model decision=inversion result” into the c-code
• Example program using all above based on Arduino framework – actually showing its working
• For NDP101 it also contains the libraries to directly communicate to the NDP101 via SPI commands (load model, run model…..). I am still exploring the different commands and options

SourceCode – Ready To Go Package on GitHub

My initial template version for this configuration is available on GitHub https://github.com/happychriss/Goodwatch_NDP101_SpeeechRecognition

To build this I have downloaded the Syntiant/EdgeImpulse repository and adjusted the file structure to the Platformio build system, in addition I have adjusted the platformio.ini file to include needed libraries.

Please note that you will need to patch one of the libraries, the baseline for my work and also instructions how to patch the local libraries can be found here.

A nice goody is the script download_model.sh – that connects to EdgeImpulse downloads a *build* model and puts the files in the right folders (./src/edgeimpulse) – so you can build your binary fresh from the model and upload to the board. You have to add a run-configuratio in CLION, so you can start the script directly. The API code for authorization on EdgeImpulse needs to be added to the paramerters.

This is, how it looks like – when all is set-up. Enjoy developing.

NDP 101 – fresh from Digitkey – what do I want to now?

I will integrate the Syntiant TinyML NDP101 board with my latest project: GoodWatch – a super smart voice controlled watch.

Currently for GoodWatch I am using an ESP32 as MCU for speech recognition. It is working with 10 numbers and two keywords (yes, no). But – to get good results for so many keywords and to get a quick response – I needed to trick a bit: Its only listening to my words. If you want to learn more on ML for Edge-Device, have a look at: Overview of ML on Edge – Devices .You can see how far I got with an ESP32 here:Goodwatch: An eInk Alarm Clock That Tries to Understand Me

With this project I have two objectives:

• Wake up the alarm clock with a key word to set-up the Alarm, get it on 100% voice controlled operation
• Enable speech recognition not only for my voice and reach a highly reliable speech recognition (otherwise setting up the old clock will be faster 🙂

Some facts on the TinyML Development Board

Board Overview

I have ordered the development “Syntiant TinyML”, it is a combination of the NDP101 processor combined with a Cortex-M0+ 32bitARM MCU, a SPH0641LM4H microphone, a LIPO power connector and USB plug- so ready to go, in more detail:

• Neural Decision Processor: NDP101
• Host processor: SAMD21 Cortex-M0+ 32bit low power ARM MCU, including:
    ◦ 256KB flash memory
    ◦ 32KB host processor SRAM
• Board power supply: 5V micro-USB or 3.7V LiPo battery
• 5 Digital I/Os compatible with Arduino MKR series boards
• 1 UART interface (included in the digital I/O Pins)
• 1 I2C interface (included in the digital I/O Pins)
• 2MB on-board serial flash
• 48MHz system clock
• One user defined RGB LED
• uSD card slot (uSD card not included)
• BMI160 6 axis motion sensor
• SPH0641LM4H microphone

The board has two components – a normal low power ARM MCU with USB connection (for communication, control, loading the models…) and a SPI connection to the heard of the system – the NDP101 MCU.

About the NDP1010 MCU: Its main component is a neural network with fixed structure: 3 Dense Layers of 256 neurons and 3 Dropout layers, the input layer must always have 1600 features.

Data recorded from the microphone is directly processed by the Audio Frontent (16 bit) , the “Holding Tank” works as a rolling buffer of up to 3 seconds of speech. The feature extractor performs a pretty much standard MEL (log-mel LMFB) transformation before using the neural network. You can read more about his here.

I guess, the small MCU is used to manage the communication via SPI to the outside world.

Software

The story just started – its available at: https://github.com/happychriss/Goodwatch_NDP101_SpeeechRecognition

This is an ongoing project, I have no clue if it will work out or how long it will take. I will keep posting on the progress – starting the next days with the development set-up and the first steps.

[display-posts category=”ndp101 – Embedded ML” include_content=”true” image_size=”medium” include_date=”true” date_format=”(d.m.Y)”]

I built already the MCW – an eInk alarm clock running on an Arduino Pro Mini – and got fascinated by the great look and power-saving of eInk papers. One issue I could not solve really efficient was alarm programming. Always many buttons… so I want to have my clock understanding me – my words in German: Numbers zero to nine and the words “yes=ja” and “no=nein”.

One easy option would be to build in a microphone and let the watch connect to Googles SpeechToText API – getting the text back – and set-up the alarm. But really, who wants to have a direct connection from his sleeping room into the cloud. I could also just use Alexa – not my style.

So, I started 3 projects:

• Teach an ESP32 microcontroller to understand my voice: Edge AI – Embedded Machine Learning
• Build a really smart ePaper/eInk watch without “button” – using voice, movement and light for control,
• !!!!! NEW NEW !!! Use a Syntiant NDP 101 for speech recognition https://44-2.de/syntiant-ndp-101—always-on-low-power-speech-recognition/

 

Hardware…Hardware…Hardware……

The key to success – is picking the right hardware – that was a good lessons learned from my last project. Here is my pick:

eInk Display

• Good Display – E-ink epd display 4.2 inch e-ink raspberry GDEW042T2
  The eInk display supports partial refresh and communication via I2C bus – important not to use to many IO pins from the ESP32.
• DESPI-C02 – a little board to connect the e-Paper with the ESP32 (no connection for power to the FrontLight Panel) and DESPI-C03 (with connection). Caution: The DESPI-C03 allows also to plugin the both FFC cables for the display and the front-light. If you think you can power-both at the same time, you kill your display

eInk FrontLight

If you want to read the time and night – you need light. With eInk that is a bit complicated, you need a special front-light. Its basically a piece of glass with LEDs at the edge. I was thinking to build it by myself. But I think buying it, was a much better idea. Still getting the result I wanted was a bit more complicated:

• Good Display – FrontLight Pannel <- these are the trick and I think it just looks cool at night The FrontLight must be an exact fit to the display. I had quite some emails with Good Display to select this combination and I was very surprised by their great customer service. At the end it came direct from China 🙂
• FFC Adapter RM 2.54 to connect the FrontLight Pannel with the power support (see above)
• HW-045 DC-DC boost voltage regulator: The FrontLigh Pannel has 7 LEDs of 200 Ohm in series , if they need 15mA to shine – you need 21V. Uups – one more device needed, a power booster.
• IRF3708 Transistor N-MOSFET 30V 62A 87W TO220AB – dimming the front-light, of course it needs to adjust to outside light condition. A first idea to use PWM as input for the voltage regulator was a bit naive, so the ESP32 is using the MOSFET to PWM the output of the voltage regulator before powering the display. To keep it running on battery for a long time, each part of hardware needs to be in low power consumption mode when the ESP32 is on sleep. For the amplifier I found the following solution, watch the tiny blue cable – that pulls the amplifier into deep sleep mode also:

Additional Hardware and Sensor

Here comes the rest, I had to do several rounds until I found the best pick – but I am really happy with below selection:

• ESP32 as https://www.wemos.cc/en/latest/d32/d32_pro.html. (320kb RAM, 4MP PSRAM, 16MB Flash).  Two very important things, my one had 16MB Flash (program code, SPIFF), what is great to have when you want to work with Machine Learning and do some more fancy stuff, build in LIPO charger and 4MB PSRAM (that is also needed by the sound library and for ML)
• Adafruit DS3231 – Precision Clock: An alarm clock needs time and alarms, both can be done the DS3231- supported by a little battery, so time and alarms can survive a short power outage. I finally picked the more expensive version from Adafruit, as two much cheaper versions from the internet just kept running to fast – approx 10s per day – no option 🙂 The DS3231 triggers and interrupt every 5min to show the time and of course when there is an alarm.
• CS43434 I2S Digital Microphone – there are several different microphones on the market, I tried them all. At least for that one, I can say its compatible with the ESP32 I2S bus and really records in good quality. My one I got over Tindie.
• PIR HC-SR602 Infrarot sensor: You can see in the video that I disable the alarm with a movement of my hand over the clock, that is a PIR sensor. Amazing – super, super low power consumption and so sensitive. The PIR sensor triggers and interrupt on the ESP32 for wake-up.
• Adafruit I2S 3W Class D Amplifier Breakout – MAX98357A: It needs to play a nice sound when waking up, and the ESP32 has enough computing power to even decode MP3 files or stream from a radio station. Also this section is on hardware – https://github.com/schreibfaul1/ESP32-audioI2S to get sound into the alarm clock with 5 lines of code.
• APDS-9960 – Digital Proximity, Ambient Light, RGB and Gesture Sensor: I wanted to use gestures to wake-up the alarm clock and decide between different operating modes. The APDS-9960 is really smart, it detects different hand-moves and measures distance. In addition it can check the light strength and color. I spent a lot of time, but its still consuming a lot of power also in low-energy mode. The only way to use this efficient: Let the PIR sensor (see above) run always – and then start the measurement. Unfortunately this causes a delay in processing – here we come to my next project: Wake up with my voice.

 

Software

Initially I have been using Arduino Platform to build this project. This platform includes a pre-compiled set of the ESP IDF. The ESP-IDF is Espressif’s official IoT Development Framework for the ESP32. As a consequence any settings used to compile the ESP IDF are fixed and can not be changed.

Boot-Speed is King

When waking up the Goodwatch, I want to see quick reaction – but noticed a 1-2 second break before the ESP32 got active.  The ESP is using this time to validate the Flash and RAM. As the Wemos D1 Pro has 4MP PSRAM and 16MB Flash – this takes a while.With a change in the compiler-options, this check could be disabled.

Arduino as a Component

The trick is to install Arduino as a component to the main framework used for compilation is the ESP IDF and it includes the Arduino libraries as a subfolder.  This provides access to an utility “menueconfig”, that generates a config file, e.g. sdkconfig.lolin_d32_pro.The problem is, that Arduino and ESP-IDF use different build-tool. After a lot of research I managed to get this working.  Speed is KING again, I would say less then 500ms to react on eInk is good.The source code can be found here. I have put some comments in the platformio.ini on the build.It’s all on Github: https://github.com/happychriss/GoodWatch_espidf

Thinking in Circles…

I had some fun on painting the circle-arcs – that brought me back some memories from school with my friends “sinus” and “cosinus”. I am sure there is much more elegant solution – but at one point of time, I was just happy to see the circles. In total I think I found a quite efficient way. Look as this “naughty” peace of code to paint an arc – and the painting:

Time takes Time – and calculating with days and months – even more

Another simple topic – that got me some time to get it right: Calculating with time. My alarm watch has 5 different wake-up alarms, the DS3231 can manage 2 alarms and create an interrupt signal to wake-up the ESP32. One alarm is gone for a wake-up every 5 minutes to paint a new arc. The other one needs to be updated on time – so, you need to sort the alarms to find out the next possible. Now you have alarms on weekdays and on weekend, repeating alarms and one-time alarms – you get the message…..I am not proud of my code here – but sorting was easy at the end.

struct strct_alarm {
    uint8_t i =0;
    bool active = {false};
    bool valid = {false};
    DateTime time;
};

int SortTimes(void const *a, void const *b) {
    const struct strct_alarm *ad = (struct strct_alarm *) a;
    const struct strct_alarm *bd = (struct strct_alarm *) b;
    return (int) (ad->time.secondstime() - bd->time.secondstime());
}

// Calling the qsort
qsort(rtcData.d.alarms, ALARM_NUMBERS_DISPLAY, sizeof(strct_alarm), SortTimes);

Flash – Keep the Memory

The alarm times need to be stored on the ESP32 and should survive a power-outage (similar to the time of the clock). The ESP32 has FLASH memory available, perfect – all what I needed. After fiddling around with CRC values to check if the data in the flash is valid and addresses..I learned about an ESP32 feature called “NVS” on the ESP Documentation. – where it got really easy. It even checks, if the flash is consistent.

   esp_err_t err = nvs_flash_init();
    if (err == ESP_ERR_NVS_NO_FREE_PAGES || err == ESP_ERR_NVS_NEW_VERSION_FOUND) {
        // NVS partition was truncated and needs to be erased Retry nvs_flash_init
        ESP_ERROR_CHECK(nvs_flash_erase());
        err = nvs_flash_init();
    }
    ESP_ERROR_CHECK( err );

// Write:
       err = nvs_open(STORAGE_NAMESPACE, NVS_READWRITE, &my_handle);
        size_t required_size = sizeof(d);
        if (err != ESP_OK) DPL("NVS_OPEN ERROR");
        err = nvs_set_blob(my_handle, STORAGE_NAMESPACE, (uint8_t *) &d, required_size);
        if (err != ESP_OK) DPL("NVS_WRITE ERROR");
        // Commit
        err = nvs_commit(my_handle);
        if (err != ESP_OK) DPL("NVS Commit Error");

// Read: 
       err = nvs_get_blob(my_handle, STORAGE_NAMESPACE, (uint8_t *) this, &required_size);
        if (err != ESP_OK && err != ESP_ERR_NVS_NOT_FOUND) {
            DPL("NVS READ ERROR");
        }

Programming the eInk Display

This would have been the most complicated, but thanks to: https://github.com/ZinggJM/GxEPD2 it was really easy. It was a bit of work to find this library and to get the concept. There is also a support-forum and the author is really helpfull. THANK YOU.

Its worth to note that the display is connected to the ESP32 via a simple adapter DESPI-C02 or DESPI-C023 using an FCC cable, from here its connected to the ESP. Most reasons for error is the wiring and correct association of the PINS – and I mixed up MISO and MOSI — took my so long 🙂

// From my hardware
// EPD Pins - GxEPD2_display_selection_new_style.h using SPI
// EPD Pins****
// CS=5, DC=0, RST=2,BUSY=15;
// SPI: SCK=18, SDI=MISO=23
// BUSY -> 15, RST -> 2, DC -> 0, CS -> 5, CLK -> SCK(18), DIN -> MOSI(23), GND -> GND, 3.3V -> 3.3V

#define EPD_CS 5 // Chip Select -
#define EPD_DC 0
#define EPD_RST 2
#define EPD_BUSY 15

GxEPD2_DISPLAY_CLASS<GxEPD2_DRIVER_CLASS, MAX_HEIGHT(GxEPD2_DRIVER_CLASS)> 
display(GxEPD2_DRIVER_CLASS(/*CS=5*/ EPD_CS, /*DC=*/ EPD_DC, /*RST=*/ EPD_RST, /*BUSY=*/ EPD_BUSY));

15 Seconds to understand the concept!

What I want is:

• A Tiny Touch-Button
• I can just stick on a wall or desk, without additional wiring, so its battery driven
• Can be enabled with a simple-touch, so it needs to wake-up from sleep
• Connects to WiFi and is very flexible, so it needs to have an ESP8266
• !!!!! The final solution will have a small battery having the same size as the button — of course 🙂 !!!!

Why this was a little challenge?

Well…I already got the display running, check out this Post. But it consumes a lot of power when used, so it needs to SLEEP, when nobody is using the display. Now it can stick on a wall, it just waits. Once you need it, you touch the display, it wakes up and it is there – after some time it will fail again into deep sleep. It should last there for some month.

It is easy to put the ESP8266 into deep sleep – -but how to wake it up?

Unfortunately I didn’t find a logic in the display, that would support a “wake-up” signal for the ESP8266. So the approach is to have the ESP8266 always sleeping and use a PIC Low Power Microcontroller to regularly give short time power to the display to check its touch controller. If not touch is reported, the PIC switches off the power and goes back to sleep. If a touch was detected, the PIC will wake-up the ESP8266, give power to the display and enable the backlight. All is back running.

Energy saving !

My initial idea was to use the ESP8266 to regularly wake-up and check the display. Unfortunately the ESP8266 is not able to keep status of its IO (digital IN/OUT) during deep sleep (took me quite some time to find this out), so when in deep sleep -it would leak power to the display (actually even to disable the backlight was not possible)…and *no power saving*.

 

PIC 16LF1824 Low Power Microcontroller

Here comes the PIC into the game:

• PIC is sleeping
• To enable the display, its connected to a transistor that is “grounding” the display to close the power loop.
• Once this is done, it will pull-down the backlight (so it keeps disabled) and it will check the “touch IRQ” pin of the display.
• Only when detecting a touch it will pull down the RST to ground, so the ESP8266 is waking up. The nice thing of a PIC, it keeps the IO status during sleep, it doest need any other hardware to run and it can be programmed easily.

I am posting this extract, to show how easy the PIC can be used for this kind of tasks:

 while (1)
    {
        WDTCONbits.SWDTEN = 0b1;
        SLEEP(); // sleep for watchdog sleep time 1.3 sec
        WDTCONbits.SWDTEN = 0b0;
       
        pen_pressed=false;
      
        //Enable the display (without backlight) to measure if touched 
        IO_RC4_TFT_BKL_OUT_SetLow();
        IO_RC3_TFT_PWR_OUT_SetHigh();  
        
        __delay_ms(40); //time needs the TFT to boot
                
        PEN_value=IO_RC5_TFT_PEN_IN_GetValue()  ;

        // Pen is pressed *** Wake up display
        if (PEN_value==0) {
         
            pen_pressed=true;

            IO_RC0_ESP_RST_OUT_SetLow(); 
            __delay_ms(10);
            IO_RC0_ESP_RST_OUT_SetHigh(); 
            
            // enable TFT backlight
            __delay_ms(600); //give the ESP time to wake up
            IO_RC4_TFT_BKL_OUT_SetHigh();
        }

        // check, if the ESP allows me to sleep again
        if (pen_pressed) {
            do {
                __delay_ms(500);
                SLEEP_value=IO_RC1_ESP_SLEEP_IN_GetValue() ;
            }
            while (SLEEP_value!=0);
        }

        // Go back to sleep
        IO_RC2_TST_OUT_SetLow();
        IO_RC3_TFT_PWR_OUT_SetLow();
        IO_RC4_TFT_BKL_OUT_SetLow() ;
        
    }
}

ESP8266 Programming

On wake up from the PIC, the ESP will take over control of the display and connect via WiFi to the SmartHome Server / DocBox Server – to register. Nothing special about this. But I wanted to be able to set-up a system to display menus and sub-menues, so I came with the following approach using AdafruitGFX-Button library and “button-press-handler” as shown below -after defining the buttons, each button is assigned to a “handler” (https://github.com/happychriss/TouchSwitch/blob/master/src/tft_helper.cpp) that will be called when the button is pressed. If each “sub-menue” is placed in its own c-file, the result is a very modular and structured.

uint main_menu() {

    //****** Paint the Display ************************************************

    DPL("**** Paint Main-Menu Display");

    tft.fillScreen(ST7735_WHITE);

    enum M1 {
        B_SCAN = 0, B_COPY = 1, B_HOME = 2, B_STATUS = 3
    };

    Adafruit_GFX_Button buttons[4];
    buttons[B_SCAN].initButtonUL(&tft, B_X_SPACE, B_Y_SPACE, B_X_WIDTH, B_Y_HEIGHT, C_BUTTON, C_BACK, C_TEXT, "SCAN", 1);
    buttons[B_COPY].initButtonUL(&tft, B_X_SPACE + (X_MAX / 2), B_Y_SPACE, B_X_WIDTH, B_Y_HEIGHT, C_BUTTON, C_BACK, C_TEXT, "COPY", 1);
    buttons[B_HOME].initButtonUL(&tft, B_X_SPACE, (Y_MAX / 2) + B_Y_SPACE, B_X_WIDTH, B_Y_HEIGHT, C_BUTTON, C_BACK, C_TEXT, "STATUS", 1);
    buttons[B_STATUS].initButtonUL(&tft, (X_MAX / 2) + B_X_SPACE, (Y_MAX / 2) + B_Y_SPACE, B_X_WIDTH, B_Y_HEIGHT, C_BUTTON, C_BACK, C_TEXT, "MORE", 1);

    for (int i = B_SCAN; i <= B_STATUS; i++) {
        buttons[i].drawButton();
        buttons[i].press(false);
    }

    global_status = GLOBAL_ACTION_MAIN_MENU;
    struct _button_handler buttonHandler[4];
    buttonHandler[0].button_action = menu_start_scanner;
    buttonHandler[1].button_action = menu_start_copy;
    buttonHandler[2].button_action = menu_start_test;
    buttonHandler[3].button_action = menu_start_more;

    return HandleButtons(buttons, 4, buttonHandler, TOUCH_SECONDS_WAIT);
}

 

 

Building a PCB

Finally I realized that this will be a bit of wiring, so I did my 2nd PCB using Fritzing to layout the PCB and directly ordered it via Aisler, a German company for quick prototyping. I was very positively surprised by the cheap price and the quick delivery. Its so much easier to design the PCB then doing all the wiring.

 

 

 

 

 

All build together – still TINY

In total the system has the following components: TFT Display, ESP8266, PIC and battery – and should still be slime. I played a lot with different layouts – that was the result. Finally (of course) the battery needs to be added. When choosing the ESP, I choosed the Wemos Mini, that comes with an integrated LiPo and LiPo charger.

What is next?

Yes …it needs a case… 🙂