Hands-on IoT applications with OpenWhisk and Rust
Introduction
Serverless computing is a hot topic in technology right now. As a result of this interest, all the main cloud vendors are providing their own serverless solutions. In this article, we’ll explore Apache OpenWhisk (which we’ll refer to simply as OpenWhisk), an open-source, distributed serverless platform that allows us to execute functions in response to events.
OpenWhisk supports a lot of languages out of the box and can be extended to use other languages. In our case, we are going to use Rust as our main primary language. Rust is an all-in-one language; you can program using imperative or functional programming, it can be used at a low level or a high level, and it has a nice, kind online community :)
In this article, we are going to develop a simple IoT solution based on OpenWhisk where we will try to use all its features. Before getting our hands dirty and starting to program, though, let’s get to know our new friend OpenWhisk a little better.
What is OpenWhisk?
As we mentioned before, OpenWhisk is an event-driven serverless platform. Put simply, it is a server that executes code when it receives an event. These events could be an HTTP GET/POST request, a hook dispatched by a tool, or even a notification from another service.
The key concept of OpenWhisk is that it is based on Docker containers, so you don’t have to worry about the architectural details. It works out of the box and can be deployed wherever you want using your favorite Docker-compatible deployment tool, so you only have to focus on implementing the magic that it is important to you. In fact, IBM is using OpenWhisk as its serverless solution for their cloud service under the name Cloud Functions.
Internally the architecture works as an async and loosely coupled system; as developers, we only need to develop functions that will be uploaded as actions. The actions are totally independent, which is great: because of this absence of coupling, the system is free to spawn or kill new processes based on the demand.
When we have implemented the actions, we then have to implement the triggers, which are basically endpoints that will be called by the event sources, databases, applications, hooks, etc. Finally, we have to define a set of rules in order to bind the triggers with the actions. That’s all we need to create magical applications with little effort. OpenWhisk will take care of everything else for us because it has an interesting internal architecture orchestrating the whole system.
Internally OpenWhisk has an Nginx server as an entry point (1) to the system. This webserver works mainly as a proxy for API and SSL. Then all the requests are injected into the controller (2).
This controller verifies the authentication and implements the OpenWhisk API. The controller works hand-in-hand with couchDB (3), where the code, credentials, metadata, namespaces, and other actions definitions are saved.
The controller also works with Consul (5), a key/value database that works as a service discovery system and keeps a reference to the invokers (the units that perform actions).
Then we have Kafka (6), which works as a glue layer between the controller and the invokers. In this case, Kafka works as a buffer for messages sent from the controller to the invokers.
And finally, we have the invokers (7). They are the guys in charge of spawning and executing the code inside Docker containers. The invokers copy the code from CouchDB and inject it into the Docker container. The number of invokers will depend on the server load.
Installation and first steps
It’s time to put our hands on OpenWhisk. In fact, OpenWhisk works out of the box without much effort:
just go to the OpenWhisk site and follow the guide if you want, or simply execute the following commands on your console:
$ git clone https://github.com/apache/incubator-openwhisk-devtools.git
$ cd incubator-openwhisk-devtools/docker-compose
$ make quick-startAfter a long wait downloading, installing, and configuring a lot of stuff in Docker, you already have your local OpenWhisk working! In order to stop the instance, just execute `make stop` and for the next run execute `make run`.
There is a really good Command Line Interface (CLI) to interact with OpenWhisk instances. To get it, go to this site and download the version appropriate to your architecture. Then just unzip it and add it to your path, or move it to a `bin` folder in your machine.
At this point, we are ready to implement our Hello OpenWhisk. First, create a file wherever you want called `hi_world.js` and paste the following code into it:
function main (params) {
var name = params.name || 'World'
return { response: 'Hello from OpenWhisk, ' + name + '!' }
}Now we need to create an action from our function. To do this we need our fantastic CLI with the following command:
$ wsk action create hiBro hi_world.js -iThe `-i` parameter is the insecure command execution in order to avoid problems with certificates. Now we have created the action and we only need to call it with:
$ wsk invoke -r hiBro -p name Brooo
{
"response" : "Hello from OpenWhisk, Brooo!"
}Yay!! It’s working, and it was really easy, wasn’t it?
The next step is to call the action from Postman. The development environment has one spacename domain called ‘guest’ and in order to obtain the user and name to be used to authenticate the request we have to call the following command:
$ wsk property get
client cert //this is weird and I think that for this I have to add -i param.
Client key
whisk auth xxxxxxxxxxxx:xxxxxxxxxxxxxx
whisk API host 192.168.1.44
whisk API version v1
whisk namespace guest
whisk CLI version 2018-09-05T11:05:38+00:00
whisk API build Unknown
whisk API build number UnknownInside the whisk auth property we have our username and password separated by a colon (`:`). This user and password have to be used in Postman in order to authenticate our requests. So now it’s time to go to our Postman window and call our action from there with the following URL:
https://192.168.1.44/api/v1/namespaces/guest/actions/hiBro?result=true&blocking=true
We also have to set the authentication method as ‘Basic Auth’ and write down the username and password retrieved from properties. In order to pass parameters to the action, we have to set the body, with the content type set as ‘application/json’, and pass a JSON with the following structure:
{
"name":"Bro"
}Why Rust?
Why not…? With this decision, we might start a huge debate about programming languages. To be honest, I chose Rust because I started to learn it under the #100DaysOfCode movement on Twitter and wanted to apply it in a project.
Rust was created by the Mozilla Foundation in 2010 and has won the “most loved programming languages” in the Stackoverflow Developer Survey for three consecutive years. Well… until now just marketing content 😜.
The main features of Rust are:
- Statically typed language.
- No more null pointer exceptions.
- The compiler will save you or will freak you. It catches a lot of possible errors as a consequence of the type system.
- You can program using Object Oriented Programming.
- You can program using Functional Programming.
- You can create very high-level code.
- You can work at a low level.
- It’s a compiled language, so in theory, the performance will be better.
- Library management is AWESOME!
- The memory management is curious. For me, it’s a mix of C and Objective C, because Rust doesn’t have Garbage Collector, but it has a funny method of managing the memory.
- The community is the friendliest, most helpful and polite around.
If all of these reasons do not convince you to give Rust a try, here is a free book with a ton more reasons: Why Rust.
Our first OpenWhisk function with Rust
Rust is not directly supported by OpenWhisk, so in order to use it we have to create a compiled version of our OpenWhisk functions and tell OpenWhisk to run that compiled code inside an invoker. To achieve this we have to build our code over the proper architecture.
First, we need to learn how to pass parameters from Rust into our OpenWhisk function. In this case, it’s an easy task, because it is the typical input parameters parsing in Rust with the following structure:
if let Some(arg1) = env::args().nth(1) {
let params = Json::from_str(&arg1).unwrap();
if let Some(params_obj) = params.as_object() {
if let Some(params_name) = params_obj.get("measurement") {
measurement = params_name.as_string().unwrap().to_string();
}
}
};In some cases, you’ll need environment variables but… OpenWhisk doesn’t support environment variables. The most similar thing is the Default Parameters. Default parameters are parameters that are defined at the moment when we create the action and they are set for all function execution unless they were overridden from input parameters. Well… Default Parameters could help a lot in order to have some kind of environment variables, but… you know, in some cases you are forced to use environment variables because library X only works through them. For those cases here is the trick: Pass your environment variable values as Default Parameters to your OpenWhisk function, and once you are inside your function:
use this:use std::env;
env::set_var("name_var","var_value");And that’s it. You have set an environment variable from Rust. Obviously, you have to do this before the environment variable is needed. In our demo, we used this trick in order to pass the AWS access key and secret to be used by Rusoto (the AWS library used).
Another two tips that freaked me out during development. If you are using OpenSSL you need to init all the environment variables inside the container. To do that you have to include this in your code before the place where are you going to use OpenSSL:
openssl_probe::init_ssl_cert_env_vars();Moreover, if you want to return something from your function, it should be a JSON. If it’s not, you’ll get an error in runtime telling you that the response of your action is not a proper dictionary.
Once you have your function ready to be used in OpenWhisk, it’s time to generate a build in the appropriate architecture. We can do this in two different ways, locally or using a prepared Docker container to build the function for us. In our case, it seems that our OpenSSL library is not properly configured, so instead of spending a couple of hours with OpenSSL we went for the Docker version. Here is a good guide from James Thomas to doing this locally on your computer and also using the Docker version.
We used this Docker machine to generate the build. In order to generate the build, go to your function folder and run:
alias rust-musl-builder='docker run --rm -it -v "$(pwd)":/home/rust/src ekidd/rust-musl-builder'
rust-musl-builder cargo build --releaseAfter this we have our function compiled and ready to be executed as a native action in OpenWhisk. We only need to add the build with the name `exec` into a zip file and create an action with the ` — native` flag:
wsk action create heyRust your_zipped_file.zip --nativeAnd that’s it!! We have our function made with Rust running on OpenWhisk.
Proposed demo architecture
We are going to develop an IoT solution and we will try to use the main OpenWhisk features on it. The idea is to try to create a Real Time Dashboard for an existing IoT system working with MQTT. We are going to use an external bridge between MQTT and OpenWhisk that will call a trigger endpoint in OpenWhisk. The trigger will fire the following actions:
- Save action that adds the received data on a AWS DynamoDB.
- Comparison function in order to dispatch an alarm in case the value is over a certain threshold.
- Connect to the previous action with an Action Chain where, if the alarm is fired, we will notify the user using a Telegram bot.
- Resend the received data through a RealTime channel in order to notify our RealTime Dashboard.
Additionally, we have some actions in order to configure the Maximum value to dispatch the alarm, and another action to get the latest values to initialize the graphs in our dashboard. A complete picture of the proposed architecture is shown below:
Demo project organization
Speaking of project organization, Rust is very flexible and you can organize your project however you want. For our demo, we decided to create one binary for each OpenWhisk function, but each executable binary has a common library that is in charge of all the logic and data modeling for the demo. The next image shows a general view of the demo project folder structure:
RealTime dashboard
The last part of this demo project is a React front end to show all the information in real time. When the front starts, it loads information saved in DynamoDB to fill all the charts with the historic data. After that, the front is subscribed to the RealTime channel in order to listen for new incoming measurements. Every time OpenWhisk publishes a message, the status of the front is updated to reflect the new received measurement. For this demo we wanted to simulate a beer factory where we have a two tank process with heaters and all the necessary elements to transfer the content between tanks.
As measurement variables, we are going to measure the temperature in both tanks, and the status of the valves, pumps, and heaters involved in the process. Moreover we have sensors for measure the two tanks’ liquid levels.
Without entering a lot of details about the beer fabrication process, in our IoT system, the first tank (mashing tank) is in charge of mixing water with malt for an hour at a fixed temperature (around 65 ºC). Then the resultant wort is transferred to the second tank, where it is boiled for sanitizing for another hour. Finally, the wort is chilled and transferred to start the fermentation. Below is a screenshot of the dashboard. We have some widgets showing the system elements’ status in real time and also some charts showing historical data about temperature measurements.
In this article we have commented only on things that are relevant to the project. For further reference please go to the demo repository and there you can explore all of the code.
Conclusions
OpenWhisk provides a lot of flexibility and simplicity, and is a very good option for implementing your own serverless environment in your private cloud or in your own data center. It has a lot of interesting features (some of them not covered in this demo) that give you a lot of options for developing your serverless solution.
Rust has been demonstrated to be a very adaptative language. We were able to develop the demo using Rust without much effort. One of the key things about Rust is its flexibility: we’ve organized the code the way we want, and the language hasn’t imposed any restrictions on it. Finally, the combined React and RealTime framework works very smoothly, and it is a very simple way to add RealTime support to your applications.
In conclusion, we think that this technology stack is very powerful for IoT applications and is a very good stack for data processing procedures in IoT environments where the different nodes are just simple capturing, leaving all the data processing to the cloud.
