Concept for Fearless and Agile Firmware Development #1

This notes is not about HIL or HDL Simulator

Motivation

In OSS, embedded software development is one area that is pretty alienated. Most of the time it was related to a hardcore low-level programming. And for someone who doesn't know what embedded programming, they will relate it to Arduino which most of the time identical with hardware prototyping and education (though it can be used for professional work). Embedded development takes more time than web development and one of the cause is lack of tools to support agile methodology. Embedded development is part of IoT ecosystem which is pretty lag behind compared to Web development or Data scientist/analysis domain. To improve the IoT ecosystem, there must be a workflow to support full-stack IoT development to stimulate code reuse in many different areas, including embedded development. Also, from time to time, the IoT use case and architecture become more and more unique (maybe because the influence of DLT).

Goals

• provide a toolchain for someone to be able to do full-stack IoT development
• provide an ecosystem module for collaboration across different expertise (web-dev, embedded-dev, and data-sci-fientist)
• provide a way to simulate the external module/peripheral/sensor/transducer, not only the chip or board

Benefit

• minimize guessing when troubleshooting by comparing the simulated firmware (run in the browser) with the real implementation (run in the physical world). Maybe the problem is not in your program, but more like in the hardware you bought 😋
• encourage the community or hardware producer to roughly model their sensor/transducer into a simulated driver
• 🦄 provide a way to code once, run or simulate anywhere
• reuse the driver implementation on the web. Maybe someone wants to create their own machine simulator in a more detailed manner

Approach

In this concept, I'm going to rely on Rust since it supports code reuse on many different domains (base on their homepage). In Rust ecosystem, they separated into 4 different working groups and one of them is rust-embedded. The core idea for this concept is to have another HAL implementation that can be compiled into WebAssembly and run on the simulated hardware environment.

How Rust in embedded development works?

In embedded development, Rust supports two general Embedded Programming:

• Hosted Environments which most of the time use Linux
• Bare Metal Environments which no high-level OS running and hosting our code

All of them was supported by a single HAL (Hardware Abstraction Layer) which in Rust it named as embedded-hal. In a nutshell, it is a set of traits which define implementation contracts between HAL implementations, drivers and applications (or firmware). Those contracts include both capabilities (i.e. if a trait is implemented for a certain type, the HAL implementation provides a certain capability) and methods (i.e. if you can construct a type implementing a trait it is guaranteed that you have the methods specified in the trait available) ^[ref].

^{Fig 1. how embedded-hal worksref}

The main reason for having the embedded-hal traits and crates implementing and using them is to keep complexity in check. There're additional benefits to be had, such as less trial-and-error due to a well-defined and ready-to-use APIs. A HAL implementation provides the interfacing between the hardware and the users of the HAL traits. Typical implementations consist of three parts:

• One or more hardware specific types
• Functions to create and initialize such a type, often providing various configuration options (speed, operation mode, use pins, etc.)
• one or more trait impl of embedded-hal traits for that type

In rust-embedded ecosystem, the utilization of embedded-hal are separated into 2 category:

• A Driver which implements a set of custom functionality for an internal or external component, connected to a peripheral
• The Application (the blue box) which binds the various parts together and ensures that the desired functionality is achieved

[a fun fact]: Arduino also has a term of HAL but it stands for Hardware Abstraction Library which in Rust is called a Driver. For the HAL (standard term) itself, Arduino calls it as Arduino API.

A HAL that compiled into WebAssembly

To achieve the concept of a Driver that can be simulated, a firmware code that can be compile-then-run in the web browser could be the quickest solution since Rust support WebAssembly as a target compilation. To do this, we need to create embedded-hal implementation which compiles into wasm32 (WebAssembly) binary code. Let's say the HAL implementation is called wasim_hal, then the usage would look like:

/// gastly/src/main.rs
#![no_std]

#[cfg(target_os = "linux")]
use linux_embedded_hal as hal;

#[cfg(target_arch = "wasm32")]
use wasim_hal as hal;

use hal::{Delay, I2cdev, println, panic};
use sgp30::Sgp30; // this is a Driver for gas sensor

#[no_mangle]
pub fn setup() {
  let dev = I2cdev::new("/dev/i2c-1").unwrap();
  let address = 0x58;
  let mut sgp = Sgp30::new(dev, address, Delay);

  match sgp.init() {
    Ok(_) => println!("SGP30 Connected"),
    Error(e) => {
      println!("SGP30 selftest: {}", sgp.selftest()?);
      panic!("SGP30 is not connected!");
    },
  }
}

#[no_mangle]
pub fn run() {
  let measurement = sgp.measure()?;
  println!("CO₂eq parts per million: {}", measurement.co2eq_ppm);
  println!("TVOC parts per billion: {}", measurement.tvoc_ppb);
  Delay.delay_ms(1000u16 - 12);
}

#[cfg(not(target_arch = "wasm32"))]
fn main() -> ! {
  setup();
  loop {
    run();
  }
}

By using conditional compilation macro, those codes can be compiled either targeting Hosted Environments (e.g raspi) and Simulated Environments which leverage WebAssembly and run on the web browser. In the Simulated Environments, we can call the compilation result of that code like:

// svelte root component
import bytes from 'gastly.wasm';

let interval;
const importObj = {
  global: {
    println: str => console.log(str),
    panic(str) {
      console.error(str);
      clearInterval(interval);
    }
  }
}

export default {
  async oncreate() {
    const {exports: simulation} =  await WebAssembly.instantiate(bytes, importObj);
    simulation.setup();

    // run simulation with frequency 5Hz
    interval = setInterval(simulation.run, 200/*ms*/);
  }
}

A driver boilerplate that can be used on 3 domain

Having a runtime for running the simulated firmware is not enough. To encourage the use of the Driver in other domain and have a working Driver that have a visual simulated component, we need to provide a boilerplate. For example, assuming we have a working cli app called hwsim, we could generate the skeleton code to support the Driver in 3 different domain by executing:

$ hwsim create:driver dynamixel-xl320

will generate:

dynamixel-xl320/
├── Cargo.toml
├── package.json
├── pyproject.toml
├── README.md
├── src
│   ├── component.html
│   ├── driver.rs
│   └── widget.py
└── tests
    ├── component.spec.js
    ├── driver.rs
    └── test_widget.py

^{Fig 2. project that generated by the cli}

Each file in the src folder have a different role:

• component.html is a svelte component which model and visualize how the Driver works
• driver.rs is a Driver implementation of specific peripherals that inherit embedded-hal traits
• widget.py is a Jupyter notebook widget implementation which can be used to help to do data exploration

I will mention why I consider svelte later in the other post

The trickiest part of this boilerplate is how to support one python file (widget.py) because if we look at the IPywidget documentation, at least it needs both javascript and python implementation. The cli will play a huge role in this case. The cli should be able to compile component.html as a library then wrap it as an IPywidget.

To support an existing Driver that rust-embedded ecosystem already have, the cli can exclude driver.rs. Though if we look at Fig 3 (subject to change), there is a possibility that we still need to compile the existing driver into WebAssembly code. For this case, the authors can provide a thin wrapper of the Driver they want to implement which compiled into WebAssembly.

^{Fig 3. using the driver for various purpose}

There is also a possibility for the Driver to be used in an edge computing scenario. High chance that the edge service/device will be implemented using Python because most of the computing SDK like google assistant or tensorflow have first-class support for Python. To be able to integrate the Driver implementation in the edge service/device implementation, we need to provide a way to compile it as a python module as shown in Fig 4 then publish it to PyPi alongside with the widget.

^{Fig 4. using the driver on edge computing case}

FAQ

Why build another simulation?

Well, most of the good simulation software I ever know is proprietary which still depends on the OS (e.g only available on Windows). Most of the Open Source or browser base simulator is kinda....umm...only good for teaching? I feel like it's not suitable for professional work, especially when you need to collaborate with others department and expertise.

Why not using an emulator like QEMU or others?

The answer is simple. It's an emulator and the nature of emulator is it's slow, especially at the startup time. Though I would love to leverage QEMU if it can be extended or run inside the web browser.

Why not base it on Arduino API?

I would love too since emscripten is much more mature than rustwasm but it's harder and time-consuming because of the C++ ecosystem. The C++ ecosystem is lack of good dependency management, hard to cross-compile, hard to configure, and so on. In short, the amount of code reuse in C++ ecosystem (across different kind of projects) is pretty low 😢

Why Rust?

A bit opinionated but what I like about Rust is their ecosystem. I think building a framework/tools on top of a great ecosystem would be a good start. Beside, WebAssembly support in Rust is much more mature than any other language out there (with C++ as the exception). For other reason, you can see it here

Why WebAssembly?

Good question! Maybe I'm a bit influenced by many smart-contract implementations out there which use WebAssembly. The main benefit of WebAssembly is portability. Hopefully, some part of the driver implementation can be used in another non-hardware project.

Why not base it on Javascript/Python like micro-python or Johny-Five?

Most of the time for firmware development, runtime/interpreter base language is a big no (unless you really really need that). Although it can speed up the development, the memory that you must sacrifice make it less appealing. This is also a problem if you want to have a custom OTA implementation (e.g update firmware will trigger something) or utilize RTOS (though I'm still doubting this concept can support RTOS).

What hardware it can/can't support?

Since Rust depend on LLVM, any hardware with architecture chip that has LLVM backend can be supported which exclude:

• Xtensa base hardware (i.e esp32 and esp8266)
• AVR base hardware (i.e Arduino UNO/Mega 😢)

Though I'm still not sure if Rust code that uses C can still be compiled into WebAssembly.

Interesting References

• rust-embedded 2019 wishlist
• rust-embedded weekly driver initiative
• Binaryen as a Qemu backend
• A bare metal physical implementation of WebAssembly
• CI for Embedded Systems
• What is embedded-hal?
• Video on how embedded-hal works
• rust-embedded: Test runner that leverage QEMU

_{This article is revision and continuation of the previous concept}

---

If you have questions and others, you can find me on: |Patreon|Youtube|Github|Twitter|