Embedded IoT Serverless Compute Helper (Google Cloud Functions Node.js/Javascript Module)

GitHub Source Code: https://github.com/RedPillAnalytics/gcloud-iot-helper

“What the? How? Why? Why isn’t this workingggg…?!?” -Me

Embedded electronics is a personal hobby of mine; the ability to write low level code, compile it, run it and see physical changes in the real world (spinning motors, flashing LEDs, sound, encoder input, etc.) is an incredible feeling. With the emergence of IoT popularity and consumer desire, there has been an explosion of affordable and well documented embedded development boards that support internet connectivity (in the form of ethernet or wireless internet). This explosion of development boards opened the door to let real world things change using the plethora of data available on the internet as stimuli, which frankly… is REALLY cool. The trouble is I find myself constantly mumbling under my breathe, making exclamations, and questioning why my HTTP requests which were hastily, poorly, and manually written by ME never work quite right. They were not written poorly because I lack any particular knowledge of HTTP, nor because I am unskilled in C, but primarily because they are hobby projects and there’s no particular reason to clean them up. As long as they work long enough for me to be happy with my project, I’m content; and don’t much care if the code is not sustainable.

A lot of projects revolve around pulling data from somewhere on the internet (ingesting/polling data if you will), and then doing something with it: lighting up LEDs etc. This is particularly cumbersome because a lot of interesting data endpoints/APIs on the internet (and certainly most of the well maintained ones) have strict security requirements such as TLS and it’s difficult (from a compute/time/energy perspective) for low level embedded controllers to communicate with these powerful servers and manage these complex cryptographic handshakes. As a result many of the projects you see for embedded low power IoT devices (especially in hobby arena) result to scraping the web for these data metrics desired, but web scraping is prohibited by most websites in their ToS, and even in the websites where it’s not prohibited, it is not a good long term solution as webpages can (and often do) get updated and scraping logic can break and require constant maintenance.

In playing around with Google Cloud I found myself drawn to Cloud Functions; they are easy to deploy, basically maintenance free, reliable, powerful, and have lots of options/flexibility. We used them extensively in one of our company’s products, as I grew familiar with them I found myself wondering what else could be done with them beyond what we were using them for. So I found myself at Google Search and found quite a few interesting and rather impressive things others had built with what is essentially server-less compute power. One person had even built a server-less API. But I wanted to build something different, and almost immediately my mind went to IoT (again a hobby of mine). After a few minutes of brainstorming I had the idea to use Google Cloud Functions as a server-less but ever ready helper and middle man between the low powered micro-controllers and many of the high complexity website APIs. Resulting in a data flow as seen below:

Request/Poll: Micro-controller -> Cloud Function -> Website Endpoint/API
Response: Micro-controller <- Cloud Function <- Website Endpoint/API

The codebase for the final result can be found at https://github.com/RedPillAnalytics/gcloud-iot-helper so feel free to download and use it yourself. The only other thing you will need is a Google Cloud account which has a free trial available and comes with free credits. So you should be able to follow along with this entire project for the low low price of $0.00. Essentially it consists of two parts:

  1. The Node.js backed Google Cloud Function code which listens/responds to the micro-controller and makes authenticated calls to a chosen API (in this case, YouTube)
  2. The embedded hardware and code (which calls the cloud function)

We will start with the backend/helper first. You will need to create a new Google Cloud project, setup an API key, deploy a cloud function using code in the above GitHub repo/below, and enable the YouTube Data API.

Code was derived from YouTube Data API: https://developers.google.com/youtube/v3/docs/

const rp = require('request-promise')
const channel = 'UChoXabQMSsYA8JRbZ_St65Q'
const key = '****************'

/**
 * Responds to any HTTP request that can provide a "message" field in the body.
 *
 * @param {!Object} req Cloud Function request context.
 * @param {!Object} res Cloud Function response context.
 */
exports.main = (req, res) => {
    rp({uri: `https://www.googleapis.com/youtube/v3/channels?part=statistics&id=${channel}&fields=items/statistics/subscriberCount&key=${key}`, json: true}).then((body) => {
        res.status(200).json(body.items[0].statistics)
    }).catch((error) => {
        console.error(error)
        res.status(500)
    })
}

That completes the helper portion of the code/setup. Of course in this instance I’m pulling the number of subscribers for my YouTube channel, but this helper could do any of a number of complex tasks, prior to returning data to the embedded device. Remember the idea is to offload the API setup and processing to the cloud function!

Below is the code for the embedded device, you’ll notice how simple the HTTP GET code is because it only has a single call with no parameters or API keys to make and it gets back the subscriber information via the Google Cloud Function. This can be deployed on an Arduino with the included software/hardware, and can easily be modified for use with WiFi boards, an ESP8266 board, or other internet enabled boards with web client libraries. The code gets the data and simply prints it to the console, but you could easily display the subscriber count on LEDs, etc. The code can also be made to update on a fixed schedule, every hour, etc.

#include <SPI.h>
#include <EthernetV2_0.h>
byte mac[] = {  0xDE, 0xAD, 0xBE, 0xEF, 0xFE, 0xED };
char server[] = "http://us-central1-emilyplusplus-iot-helper.cloudfunctions.net"; // Google

EthernetClient client;
#define W5200_CS  10
#define SDCARD_CS 4
void setup() {
  Serial.begin(9600);
  pinMode(SDCARD_CS,OUTPUT);
  digitalWrite(SDCARD_CS,HIGH);//Deselect the SD card
   while (!Serial) {
    ;
  }

  if (Ethernet.begin(mac) == 0) {
    Serial.println("Failed to configure Ethernet using DHCP");
    for(;;)
      ;
  }

  delay(1000);
  Serial.println("connecting...");

  if (client.connect(server, 80)) {
    Serial.println("connected");
    client.println("GET /yt-subscribers HTTP/1.1");
    client.println("Host: us-central1-emilyplusplus-iot-helper.cloudfunctions.net");
    client.println("Connection: close");
    client.println();
  } 
  else {
    Serial.println("connection failed");
  }
}

void loop()
{
  if (client.available()) {
    char c = client.read();
    Serial.print(c);
  }

  if (!client.connected()) {
    Serial.println();
    Serial.println("disconnecting.");
    client.stop();

    for(;;)
      ;
  }
}

And that concludes the tutorial 🙂 I hope it is of help, everything is fair game, open source, etc. So enjoy and use as you wish!


 Originally published at Emily++.

Leave a Reply

Your email address will not be published. Required fields are marked *