|
|
Subscribe / Log in / New account

A look at the ESP8266 for IoT

By John Coggeshall
June 15, 2020

The Internet of Things (IoT) world is filled with countless microprocessors. One option we have covered in various ways before is the Arduino ecosystem. In the same vein, we now will look at another interesting segment of that community: The WiFi-enabled Espressif ESP8266 chip.

The ESP8266 chip is one of the more ideal chips for home-grown IoT development. It is cheap, has built-in 2.4GHz WiFi capabilities, has up to 17 different general purpose I/O (GPIO) pins available, and can take advantage of the extensive libraries available in the larger Arduino community. Many different commercial IoT devices use the ESP8266 chip; some devices, like those provided by Sonoff are known to be reprogrammable with custom firmware, while other off-brand devices are also known to be unlocked as well. The downside to the ESP8266 is that the SDK is closed-source; consisting of headers and binary libraries distributed under a non-free license that forbids use on anything but the ESP8266.

The lack of source code for the SDK is regrettable, but there still is an open-source community around the chip worth understanding and exploring. For readers unfamiliar with the ESP8266 chip, we will start with how it fits into the Arduino ecosystem.

Understanding the layers

When getting started with ESP8266 development, it is helpful to understand the various approaches that can be taken to use the chip, and how it fits into the development landscape of the Arduino project.

Many (if not most) ESP8266 chips come standard with the Espressif AT firmware that provides an AT-command interface over serial to make various TCP/IP requests (some documentation of available AT commands can be found here). In fact, it is not uncommon to use both an Arduino device and an ESP8266 chip together. One example would be an Arduino device controlling sensors and motors, while communicating with the ESP8266 over serial to provide network connectivity. For circumstances where the primary firmware is written for an Arduino device, and the ESP8266 simply provides networking capabilities, there may not be any need to re-program the stock ESP8266 at all.

That said, the Arduino board can often be cut out entirely and replaced using only the ESP8266 chip and its own I/O functionality. In these cases, the stock AT firmware on the ESP8266 is replaced with a custom one that accomplishes the same tasks normally done using an Arduino board. To begin we have the Espressif ESP8266 SDK which, as mentioned, is distributed closed-source. The SDK provides the C APIs (PDF) for the building of custom firmware that can replace the default firmware on the chip.

This SDK, as its name implies, is all that is needed to write and deploy code to an ESP8266 using the lowest-level APIs available. However, most developers do not use the Espressif SDK directly in favor of the Arduino Core for ESP8266 (Arduino Core). This LGPL-licensed code base provides the Espressif SDK in a way that is compatible with the Arduino ecosystem, allowing developers to write Arduino-style code for their ESP8266 devices. In this approach, one would add the ESP8266 board definitions to the Arduino IDE, and from there it can be treated like any other Arduino board. It's still using bits from the closed SDK, such as TCP/IP and WiFi stacks, but a lot more of the code base is open and when possible kept in step with other boards in the Arduino ecosystem.

What you get with the Arduino Core for ESP8266

Many efforts around the ESP8266 Arduino Core make it worth the effort to use it over the stock SDK, besides simply being as open source as possible. Arduino Core includes C++ classes for a web server (SSL/TLS available), a filesystem implementation, and both a DNS and mDNS server. Client tools for communicating over TCP/IP, including an implementation for making HTTP requests, are also provided.

The Arduino Core enables access to most of the Arduino project's library of modules. According to the Arduino Core project documentation, most Arduino libraries should work without modification. This includes a VNC client, libraries to implement REST APIs, and MQTT clients. Since the ESP8266 has multiple GPIO pins available, most libraries that interact with other hardware such as temperature sensors or individually-addressable LEDs should also work out of box. These libraries provide the bit-level protocol implementations to communicate with, for example, a temperature sensor — providing the developer with a simple API to access the temperature reading.

Other open-source efforts around ESP8266

Cost and availability of the ESP8266 have made it a favorite in the Arduino community when network connectivity is needed, despite the nature of the chip's SDK. Those same reasons have also driven efforts to reverse-engineer the closed-source SDK and replace it with open-source solutions.

The ESP Open SDK is the most comprehensive attempt at an open-source solution for the ESP8266. This project combines the closed-source SDK with a variety of open-source projects to provide a complete stand-alone SDK including a toolchain for development. While the project appears to have stagnated in recent years, it is still a viable way to build firmware for those who would prefer as much open source as possible. It is also noteworthy that, due to the stagnation, it seems to still rely on older versions of the closed source SDK for the binary components it needs.

Another interesting open source project trying to demystify the software stack of the ESP8266 is an open-source bootloader named rBoot developed by Richard Burton. The bootloader in the ESP8266 is the first code executed when the device is powered on, and is responsible for a number of pre-application tasks, like switching to a new firmware version after an over-the-air (OTA) update.

rBoot is enabled by some linker-hacking at compile time to replace a function in the SDK. By replacing this function, rBoot is able to trick the SDK into loading rBoot instead of eBoot. As is typically the case, rBoot provides not only confidence in the code executing on the chip, but more features when compared to the stock eBoot bootloader. Key features such as multiple firmware images, GPIO-based firmware selection, seamless fallback if a firmware is corrupt, and more robust firmware checksum-validation are all welcome additions. To use this validation feature of rBoot, Burton has also written his own version of an ESP8266 ROM packaging tool called esptool2. This ROM-creation tool enables the checksum-validation found in rBoot by including the .irom0.text section of the ROM in checksum calculations. For those who would like to learn more, Burton's blog is a plentiful source of information on both projects.

The linker-hacking aspect of rBoot can make it challenging to use, however. The biggest barrier is that the most common way to build an ESP8266 firmware is to use the Arduino IDE, which doesn't support rBoot. To use rBoot, a level of comfort with more traditional build tooling such as ld and make is necessary to produce an rBoot-enabled ROM. Once built, you will also need to be familiar with esptool2 to package and then flash that ROM onto the ESP8266 itself.

Getting away from the Arduino IDE is hardly an unwelcome idea in the community, as the shortcomings of the Arduino IDE have not gone unnoticed by seasoned developers. Those who prefer not to be forced into using the Arduino IDE for ESP8266 development will be happy to know about the makeEspArduino project. This project's goal is to remove the dependency on the Arduino IDE entirely for ESP8266 development, without losing access to the vast array of Arduino-compatible libraries available. Essentially, it is a complicated makefile system that does all the heavy lifting when it comes to making sure everything is compiled and built correctly. It parses the same board definition files used by Arduino IDE when building the project, which is helpful as they are updated frequently. With a little understanding of build tools and makefiles, it is reasonable to modify makeEspArduino to use rBoot instead of the stock SDK bootloader.

Keeping in line with getting away from the Arduino IDE is the Arduino CLI project. This Go-based command-line tool effectively replaces other important bits of the Arduino IDE, such as the library and board managers, entirely allowing the developer to install and use Arduino library dependencies for their projects directly from the shell. When coupled with a project like makeEspArduino, it completely removes the dependency on the Arduino IDE when building ESP8266 projects.

Other considerations and wrapping up

For developers who are looking to use something other than the C++ development framework used in Arduino environments, there are multiple other potential language options available. There is the MicroPython project, Lua provided by NodeMcu, and even, JavaScript for the ESP8266 provided by the Espruino project. All these projects by necessity still require the binary blobs that ship with the official SDK, but all may be useful if there is a desire to use a different language than C++.

With so many language choices and reasonable GPIO abilities, it is unsurprising there are many ESP8266-based projects to be found. For users of Home Assistant smart hub (see our previous introduction), esphomelib integrates complete with Home Assistant and claims to enable people to build sensors and IoT devices for their hub without touching a single line of code. There are also projects like Tasmota designed to take the difficulty out of designing an ESP8266 firmware for many common devices types.

The bottom line when it comes to building network-aware devices using the ESP8266 chip is that there are a lot of options. From straight use of the official SDK, to the Arduino Core, to another language entirely like Python or JavaScript, the chip is remarkably accessible — especially considering it is fundamentally closed-source. Moreover, the chip comes in many different form factors, each with a different price point and set of capabilities, allowing it to fit into almost any project. These qualities make it a prime candidate for any hacker interested in implementing their own IoT devices. Coupled with an open-source smart hub like Home Assistant, it makes the idea of a (largely) open-source smart home more attainable.



to post comments

A look at the ESP8266 for IoT

Posted Jun 15, 2020 17:40 UTC (Mon) by Cyberax (✭ supporter ✭, #52523) [Link] (5 responses)

I really dislike WiFi-based devices for home automation. Does anybody here knows any good alternatives that use ZigBee/ZWave?

From my personal experience, there's a decent chip for ZWave: https://z-uno.z-wave.me/ - it's expensive and very limited because the chip is based on an 8-bit 8051-based CPU, but can work for _years_ from a single battery. The SDK is closed source because of ZWave licensing conditions, but developers are fairy responsive. I'm in awe of all the hacks that were required to get it working: https://habr.com/ru/post/313898/ There's also an upcoming 32-bit ARM-based successor that won't have these limitations.

I've made a couple of projects based on this chip ( https://github.com/Cyberax/ZunoSomfy ) and it works quite well.

My experience with ZigBee was MUCH worse. I was not able to find any good documentation or tools for it. The web is full of "how to connect two Arduinos using XBee" tutorials, but not on how to make a simple light switch.

A look at the ESP8266 for IoT

Posted Jun 15, 2020 18:28 UTC (Mon) by tao (subscriber, #17563) [Link]

Oh yes, +1, me too.

I'd love to have some cheap/reasonably priced chips with ZWave support. I've got light switches covered and various other things covered, but I'd love to integrate my Dyson fan with the rest of my home automation.

A look at the ESP8266 for IoT

Posted Jun 16, 2020 22:49 UTC (Tue) by gerdesj (subscriber, #5446) [Link] (3 responses)

wifi is a medium only (layer 1, straying slightly into L2 territory), Zwave and Zigbee are rather more: Layer 1 to 4 is defined. With wifi, you do what you like for L2-4. Each has its merits or not.

wifi is nearly ubiquitous and generally well understood. Zwave and Zigbee are both pretty specialist. Zigbee uses mostly the same frequency band as wifi ~2.4GHz but not quite, so care is needed but from experience it is all good. Zwave is pretty odd and there are some odd ideas in its design from a modern perspective but it does work eventually and very well.

If you can work within the constraints that each provides then they are all great components for a solution to a problem. I don't see wifi devices as a problem, merely another tool. I nearly always put Zwave, Zigbee and wifi on all my controllers so I have the maximum flexibility depending on what is required.

This is my office: https://www.google.co.uk/maps/@50.9461237,-2.6379529,3a,7...

That building is 19m on the long side and construction is sticks and bricks off of the 1940s. There is one Home Assistant box, which is now an old laptop, with a new battery. There are 40 odd windows and 20 doors, all of which have a Zwave monitor, except for the ones that have a wifi one as well, based on a ESP8266.

Power monitoring is done by noting that our genny/power board thing has a set of terminals on it that will indicate what is going on. Wire up a ESP80266 to them via two GPIOs and (nearly) job done. ESPHome finishes things off - https://esphome.io/index.html

A look at the ESP8266 for IoT

Posted Jun 17, 2020 0:50 UTC (Wed) by Cyberax (✭ supporter ✭, #52523) [Link] (2 responses)

ZWave and ZigBee has a great advantage - they are designed to degrade gracefully. It's possible to connect switches directly to actuators (using associations) that will stay working even without the hub and in some cases even without mains power. ZWave also supports secondary hubs for redundancy.

There's also the problem with WiFi unreliability (at 2.4GHz), power use and attack surface. ZigBee/ZWave are not accessible from most devices, so a random malware on a Windows laptop can't connect to the smart lock and get all the codes from it.

A look at the ESP8266 for IoT

Posted Jun 18, 2020 8:56 UTC (Thu) by intelfx (subscriber, #130118) [Link] (1 responses)

> ZWave and ZigBee has a great advantage - they are designed to degrade gracefully. It's possible to connect switches directly to actuators (using associations) that will stay working even without the hub and in some cases even without mains power. ZWave also supports secondary hubs for redundancy.

Shouldn't it be possible to do some fancy stuff with 802.11s?

A look at the ESP8266 for IoT

Posted Jun 18, 2020 9:08 UTC (Thu) by Cyberax (✭ supporter ✭, #52523) [Link]

> Shouldn't it be possible to do some fancy stuff with 802.11s?
No, not really. ZigBee/ZWave are supremely optimized for sensor networks and complex topologies. For that matter, newer IoT mesh products are re-using the transport layer from ZigBee.

And of course, WiFi also doesn't define protocols between devices.

A look at the ESP8266 for IoT

Posted Jun 15, 2020 18:21 UTC (Mon) by pj (subscriber, #4506) [Link]

If you're familiar with platform.io, you might like the ESPurna project, which uses platform.io to let you customize ESP8266 firmware.

A look at the ESP8266 for IoT

Posted Jun 15, 2020 18:45 UTC (Mon) by logang (subscriber, #127618) [Link] (5 responses)

ESP8266 is pretty obsolete at this point. The ESP-32 series of products is the replacement and only slightly more expensive.

There is an "open" SDK developed by espressif for the ESP8266 on github[1] and also for the ESP-32[2]. The latter is much more actively developed but there were vague plans to merge the two. Both include standard open source libraries like FreeRTOS, lwip, mbedtls, etc. The main benefit of the ESP-32 hardware is that you can take advantage of the significantly improved esp-idf project but it also fixes a bunch of corner case hardware bugs and features a much faster CPU with a dual core option.

I put "open" in quotes because the vast majority of the source code is available except for a few core binary modules dealing with the Wifi hardware. Additionally, espressif keeps a pretty tight leash on things: they have a CLA; and PRs from outside contributers can take many months to get merged (even if it's a trivial bug fix). I've seen bugs fixed by people that couldn't sign the CLA so they just languish forever.

Code style also isn't great and they seem to be more focused on features than stability. They have nearly a thousand open issues and pull requests on github and this number seems to be growing. I've hit a number of tricky concurrency bugs that are impossible to track down and need to be worked around.

In summary, it's not perfect but it's still far better than any of what the competition offers.

[1] https://github.com/espressif/ESP8266_RTOS_SDK
[2] https://github.com/espressif/esp-idf

A look at the ESP8266 for IoT

Posted Jun 15, 2020 19:08 UTC (Mon) by arnd (subscriber, #8866) [Link] (2 responses)

Moreover, there is even a Linux port to the ESP-32 in progress: https://github.com/jcmvbkbc/linux-xtensa/tree/xtensa-5.6-...

Its usefulness is somewhat limited by the the extremely low amount of RAM (4 or 8 MB of external PSRAM on the most expensive boards; on-chip RAM is not enough) and the lack of an MMU, but it's still pretty cool that this is possible at all.

A look at the ESP8266 for IoT

Posted Jun 15, 2020 19:33 UTC (Mon) by coogle (guest, #138507) [Link]

That is an interesting project, I will check it out!

A look at the ESP8266 for IoT

Posted Jun 17, 2020 5:03 UTC (Wed) by jcmvbkbc (subscriber, #86115) [Link]

It's kind of stuck, because it's hard to write a useful application in a form of FLAT binary given the memory restrictions. So I'm looking at adding FDPIC support to the compiler.

A look at the ESP8266 for IoT

Posted Jun 15, 2020 19:46 UTC (Mon) by coogle (guest, #138507) [Link]

I would disagree that the ESP8266 is obsolete, myself. I agree the ESP32 is a more powerful chip of course, but many applications simply don't need it. The ESP8266 chipset has more diversity in terms of development boards / memory / price than the ESP32 does and still has a very strong community around it. As a maker myself, I can afford to keep a pile of various different ESP8266 modules in a drawer handy more affordably than ESP32s -- and it's nice to be able to grab an ESP8266-01 instead of a full NodeMcu board when I just want a very simple small device that pumps one-wire data over to MQTT. For example, the ESP8266-05 board is smaller than the ESP32 module without the development board around it.

To your other comments, it is unfortunate in all circumstances that Espressif hasn't really embraced open source for their line of chips. It helps things like the RTOS version of the SDK is built on mostly open source projects, but I'd like to see the network stack and related binary blobs released as well.

A look at the ESP8266 for IoT

Posted Aug 14, 2020 12:21 UTC (Fri) by rebeliousconformist (guest, #140794) [Link]

on the surface the esp-idf looks professional, but it lacks a lot of tools, especially for debugging. Unexplained heap crashes are quotidian. The tools from the manufacturer are buggy. Even though it is cheap . the cost of development for units <1000 is higher than an ARM based linux embeddded system.

A look at the ESP8266 for IoT

Posted Jun 16, 2020 7:45 UTC (Tue) by guus (subscriber, #41608) [Link]

Another nice feature of the ESP8266 and ESP32 chips is that they run from 2.3V up to 3.6V. This means that you can easily power the chip, without needing any voltage regulation besides perhaps a capacitor, from 2 AA(A) batteries or a 3V lithium battery like a CR2. Since it has some very lower power modes, this can be enough to keep the chip running for a year while polling a sensor every so often.

If power it not a constraint, then especially the ESP32 is a very powerful chip. See for example the Odroid Go device for an example of what you can make with it: https://www.hardkernel.com/shop/odroid-go/

A look at the ESP8266 for IoT

Posted Jun 16, 2020 11:27 UTC (Tue) by kbee (subscriber, #4468) [Link]

RIOT OS has good support for ESP8266 and ESP32. See the 8266 doc page [1]. I wrote an app to use the sensor/actuator API to push temperature readings via LwM2M [2,3].

[1] http://doc.riot-os.org/group__boards__esp8266.html
[2] http://doc.riot-os.org/group__sys__saul__reg.html
[3] https://github.com/kb2ma/riot-lwm2m-client/tree/master/cl...

A look at the ESP8266 for IoT

Posted Jun 18, 2020 6:29 UTC (Thu) by rakoenig (subscriber, #29855) [Link]

https://sensor.community/en/ shows a nice application to measure environmental data and submit it to the internet, so that maps showing the air pollution can be build. Have installed one device, solar powered with a backup battery in my garden, works perfectly fine.

A look at the ESP8266 for IoT

Posted Jun 25, 2020 11:17 UTC (Thu) by kragil (guest, #34373) [Link]

There was a talk about doing this with Nim at NimConf 2020 https://conf.nim-lang.org
Video here: https://www.youtube.com/watch?v=eCCrkZI0rVU&list=PLxL...


Copyright © 2020, Eklektix, Inc.
This article may be redistributed under the terms of the Creative Commons CC BY-SA 4.0 license
Comments and public postings are copyrighted by their creators.
Linux is a registered trademark of Linus Torvalds