I know there are some very good IOT services such as Blynk and ThingsSpeak, but as a full stack developer I like being in full control. So this post is not about those 3rd party components. Well, Amazon is also a 3rd party tool but I want to use it just because I don't want to setup a server. Anyway let's start.
You have started writing ESP8266 applications. Good. This means sooner or later you will want to access online services, be it your own web service or 3rd party Web APIs. ESP8266 could be used as a standalone web server but it may not be capable of running the tasks you create because of its speed and memory.
You wanted to read the entire contents of a Web page and fetch only some bits of it. For example; you want to print Box Office from IMDB on an OLED screen. In order to do that you need to find the HTML code of the Box Office list and parse it for your needs. This is a fairly laborious process for ESP8266. Theoretically it can be possible, but this type of work requires heavy processing and it should be handled on a back-end server. In this case ESP8266 is the client as you can imagine. It's more appropriate for ESP8266 to work on pre-processed data such as JSON.
In most cases we find an API to do the job. Most of them are either completely free or free within certain limits which is more than what a hobby developer needs. For example, using OMDB Rest API we can get information about all the films without parsing IMDB data. But what if we don't have such API and your task is not really a standard task. It means you need to write your own API.
To write your own API you need you need a server of your own. To run that server you at least need 5-10 dollars a month. But if we use AWS Lambda we will not need this. And may be we don't even need to pay for it as AWS Lambda is free for first 1 million requests not only for free-tier users but for all members.
There's one last detail. Most API services usually work HTTPS. This means that we need to establish the connection using the library "WifiClientSecure". Many examples on the Internet use WifiClient library. You can not use these examples via HTTPS. We need to send our requests to port 443. In that case we need to do 3 main components:
- AWS Lambda
- AWS API Gateway
- Sending HTTPS requests with ESP8266
For my example I wanted to read the contents of surf-forecast.com and generate a JSON that contains wind and wave data for the selected surf points. But same logic could be done for anything where you can use GET requests. Using this JSON you can easily use ESP8266 to print on OLED screen.
Let's first create our function on AWS Lamda. I've created a Node JS 6 function with a name of "surfReport". This example of course is tightly dependent on the design of surf-forecast website. If one day they decide to change the design of the page, you'll also need to change your service.
Let's briefly explain what we've done here. Lambda is reading the contents of the web page using GET. In this content finding the parts for wave quality, size and wind direction and creating a JSON response. For example for this example I used "North Piha" of New Zealand. So using a querystring I can parse every single point from the surf-forecast.
http://www.surf-forecast.com/breaks/North-Piha/forecasts/latest/six_day
After writing our AWS Lambda we need to create endpoints to make it available to public. (We can restrict the access using authorizers but that's not the scope of this post).
Using Triggers-API Gateway we will create a new gateway. The important point here is that we should select "Use Lambda Proxy Integration" in "Integration Request". After this "Invoke URL" will be ready. We will be able to use this URL inside our ESP8266 code.
After doing our OLED-ESP8266 connections as below we will start getting results:
Results:
