Programming virtual sensors (advanced) – Consibio Customer Portal

Consibio Cloud receives millions of datapoints every day from thousands of connected devices. These datapoints originates from sensors or other peripherals connected to Consibio's IoT devices.

In more advanced applications it is often desired to perform some kind of analysis on the collected data to derive other parameters, detect changes over time or compare different datasets.

Virtual sensors is a tool to do just that in a completely automated setup in a very simple scripting language. Every time a sensor value is sampled, it can automatically trigger a virtual sensor which will analyze the data according to the specific setup. This enables two things:

Minimizes repetitive work
You don't have to spend the time downloading data as CSV, importing it into your analysis software (Excel, JMP, R studio, Python or other), pretreat it and then do the analysis - and repeat this process every time the analysis should be done on new data.
Automated actions
You can automate other actions based on the calculated values coming from the virtual sensors. Virtual sensor outputs can eg. trigger automatic SMS alarms, be used as input for automation rules and much more.

Basic setup

A virtual sensor configuration has four overall setting fields:

Trigger
Every time a new measurement is received for the trigger element, the virtual sensor calculations will be performed.
Expression
The expression is the actual program or script, that will be executed when the virtual sensor is triggered.
Destination
The destination is the element, where the output of the virtual sensor calculation should be saved, each time it is triggered.
Execute on
The place, where the calculation should happen. This can either happen in the cloud or locally on the device, if it has access to all the needed values. If in doubt, leave it to the default value (execution in the cloud). See Handling delays due to data non-continuous data transmission for more info.

A new virtual sensor virtual sensor can be configured by opening the desired project at https://consibio.cloud, going to "Virtual sensors" in the top menu and clicking "Add new virtual sensor":

This will open a drawer where the new virtual sensor can be configured:

The destination and trigger elements can be selected by simply clicking the selectors. It is also possible to create new elements when clicking these.

As described above, the virtual sensor calculations will be performed when a new measurement is saved to the trigger element, whereas the output of the virtual sensor calculation is saved in the destination element.

The Expression Builder

The logic of the virtual sensor calculation is defined by the program written in the expression builder.
The expression builder contains 2 fields: The primary expression field and the result field:

The primary expression field is used for all intermediate calculations and can be used to store values in variables.

The result field is a one-liner and the final output of the calculation, that should generate a single number.

Using element values

In both fields, you can insert references to elements, which will be replaced with the actual values of those elements and the time of the calculation. Elements are referenced by their ID and enclosed in double brackets as such:

{{element_id_goes_here}}

If an element is opened under the "Elements" tab in the top menu, the ID is listed at the top. This id can be typed or pasted in, but as element id's are not easy to remember, they can be inserted in the expression field, by placing the cursor in the field and then clicking on the "Insert element" button.

This will open an element selector, where you can find an element by it's name and insert the ID by simply clicking it:

Example

Say you have a device, which reports temperature data in Celsius and you want to convert it to Fahrenheit, and that the measurements are saved in an element named "Cool temp test".

Then, you could do this, by:

Placing the cursor in the primary expression field and click on "Insert element":
Find the element you want to use ("Cool temp test" in this case) and click it.

This will insert the element id in brackets for you as such:
If you write "celsius = " in front of this id, you can now refer to element by the variable name "celsius" in the following lines (which is often easier to read):
A Celsius temperature is converted to Fahrenheit by multiplying by factor of 1.8 and then adding 32. We can do this in the next line and save the result to another variable named "fahrenheit":
The final step is to define, that our result should simply be the value now contained in the "fahrenheit" variable. We can simply type this variable in the result field as such:

And that's it! Now, whenever a new measurement is received for the trigger element, the calculation above will be performed (using the latest known value for each element referenced in the script) and the result will be saved to the destination element.

It is often desirable to use one of the elements referenced in the expression as the trigger element (in this case the element named "Cool temp test" with the ID "3inLpy3tdC4ijm0ecrqF"). If we were to save the Fahrenheit temperature to an element called "Cool temp Farh.", the full setup would look like this:

Programming syntax

The virtual sensors are programmed in a custom scripting language tailored for mathematical expressions.

Basic mathematical operators

The syntax support all basic mathematical operations listed below:

Symbol	Operation	Example
`+`	Addition	`2 + 2`
`-`	Subtraction	`4 - 2`
`/`	Division	`2 / 2`
`*`	Multiplication	`2 * 2`
`()`	Grouping	`2 / (2 + 2)`
`^`	Exponentiation	`2 ^ 2`
`=`	Assignment	`my_variable = 2`
`%`	Modulus	`4 % 3`

Pre-defined constants

Symbol	Operation	Value
`pi`	Pi	`3.1415926535898`
`e`	Euler's number	`2.718281828459`

Basic functions

Symbol	Operation	Example
`exp(x)`	Natural exponential function	`exp(0) # = 2.718..`
`log(x)`	Natural logarithm of x	`log(e) # = 1`
`log(x,y)`	Logarithm of x with base y	`log(100, 10) # = 2`
`abs(x)`	Absolute value of x	`abs(-2) # = 2`
`sqrt(x)`	Square root of x	`sqrt(4) # = 2`

Trigonometric functions and related

Symbol	Operation	Example
`sin(x)`	Sine	`sin(pi/2) # = 1`
`cos(x)`	Cosine	`cos(pi) # = 1`
`tan(x)`	Tangent	`tan(pi) # = 0`
`sinh(x)`	Hyperbolic sine	`sinh(1) # = 1.175`
`cosh(x)`	Hyperbolic cosine	`cosh(1) # = 1.543`
`tanh(x)`	Hyperbolic tangent	`tanh(1) # = 0.762`
`asin(x)`	Inverse sine	`asin(1) # = 1.571`
`acos(x)`	Inverse cosine	`acos(1) # = 0.0`
`atan(x)`	Inverse tangent	`atan(1) # = 0.785`

Boolean logic

Simple boolean logic can be executed using ternary operators. For example, if a calculated value must not be smaller than 0.0, it can be evaluated as such:

maybe_negative_value = {{element_id}}
limited_value = (maybe_negative_value < 0) ? 0.0 : maybe_negative_value

Comments

You can insert comments with the sharp sign character. You can start a line with a comment, or you can insert a comment inline after an expression:

# This is a comment

a = 2 + 2 # Here, we add 2 and 2 and assign the result (4) to the variable 'a'

Advanced templates

Until now, we have only discussed the basic element template of the form:

{{my_element_id}}

where the placeholder will be replaced with the most recent value for that element at the time of execution.

However, if you need to perform some kind of numerical calculus, time series analysis or digital signal processing, you need to be able to access older samples as well. Using advanced templates allows you to do just that.

The template above is actually a shorthand notation for this complete template syntax:

{{my_element_id[n].val}}

The generic structure for this template syntax is:

{{<element_id>[<index>].<datapoint_property>}}

The 3 fields in the template syntax are described below:

element-id
The id of the element that should be used.
index
The index is used to select either the newest sample for the element (index n, the default), the second newest (index n-1), third newest (index n-2) etc. Only the latest 8 measurements can be accessed this way (ie. from index n to index n-7).
datapoint_property
Each datapoint has two properties: .val (the default, if nothing is specified) and .time
The .valproperty holds the value of the measurement, whereas the .timeproperty holds the time of the measurement expressed as a unix timestamp in seconds.

Therefore, if you wanted to measure the change in a variables value, it could be done as below. Here we utilize, that the default value of the datapoint_property is .val, so we don't have to write it out:

{{my_element_id[n]}} - {{my_element_id[n-1]}}

If we wanted to measure the rate of change of a variable, we could divide the change in the value with the difference in time between the two measurements:

delta_value = {{my_element_id[n].val}} - {{my_element_id[n-1].val}}
delta_time = {{my_element_id[n].time}} - {{my_element_id[n-1].time}}

rate_of_change = delta_value / delta_time

All timestamps are stored as unix timestamps in seconds, so thedelta_time variable above is in seconds.

Best practices

Use semantically named variables for element ids

When you insert elements ids into an expression, you could use it directly with something like:

result = ({{first_element_id}} + {{second_element_id}}) / 2

However, as element id's are usually randomly generated, the above can be very hard to read.
Therefore it is good practice to store the values of element in variables that are semantically named based on which kind of parameter the element represents:

inside_temperature = {{first_element_id}}
outside_temperature = {{second_element_id}}
average_temperature = (inside_temperature + outside_temperature) / 2

result = average_temperature

Advanced topics

Numerical calculus

Based on the advanced template syntax described above you can perform numerical integration and differentiation of measurements using virtual sensors.

Numerical differentiation

If you eg. want to calculate the rate of change of a temperature measurement, you can get that by calculating the first order derivative as such. Here we assume that the temperature measurement is saved in an element with the idmy_temperature_id:

delta_y = {{my_temperature_id[n].val}} - {{my_temperature_id[n-1].val}}
delta_t = {{my_temperature_id[n].time}} - {{my_temperature_id[n-1].time}}

derivative = delta_y / delta_t

Numerical integration

Integration is a state dependent operation, where we need to add the calculated result to the previous output of the same virtual sensor. We can reference the previous output of the virtual sensor by using the id of the element, that is used as the destination for the virtual sensor.

If we for example want to integrate a hydraulic flow rate to calculate an accumulated volume, we could:

Save the flow measurement (from a sensor) in an element with id flow_element_id
Create a virtual sensor which uses this element as the trigger, and another element with an id = volume_element_id as the destination element.

In that case, we could perform the numerical integration using the trapezoidal rule as such:

# Save element values in different variables
y0 = {{flow_element_id[n-1]}}
y1 = {{flow_element_id[n]}}
t0 = {{flow_element_id[n-1].time}}
t1 = {{flow_element_id[n].time}}

# Calculate the step size from the sampling rate
h = t1 - t0

# Get last value of the destination element used for this virtual sensor
Y0 = {{volume_element_id}}

# Calculate accumulated integral using the trapezoidal rule
integrated = Y0 + (h * y0) + (0.5 * h * (y1 - y0))

Handling delays due to data non-continuous data transmission

The calculations performed with a virtual sensor usually depends on sensor measurements originating from an IoT device. If this device is battery operated it will usually not measure and transmit continuously, but will instead perform several measurements before it again connects to Consibio Cloud and transmits the values (based on the configured measurement and transmission intervals, respectively).

Depending on which type of calculation we perform in the virtual sensor and how the result is used, we might need to account for this in our virtual sensor logic.

When a device is accumulating measurements before transmitting it over a (maybe flaky) wireless connection, we cannot expect that the Cloud receives the datapoints in order and all at once. The devices will attempt to do this at a best effort approach, but it prioritizes short transmission times over correct tranmission in order to preserve battery.

For state-dependent operations like accumulations or integrations, this might change the calculated output as datapoints might be skipped.

The best way to avoid this situation is to use elements in the virtual sensor, that all originates from the same device and then select the device as the execution target. If you for example need a flow and temperature measurement for an enthalpy balance calculation, you should strive to connect these sensors to the same device. In the cases where all elements are sampled on the same device, you can select in the user interface that the virtual sensor should be executed on the given device:

Then, the device will perform all calculations locally every time it samples the sensors, and transmit the result to the cloud the next time it connects. In this way, you can be sure that everything happens in order.

This also has the added benefit, that the virtual sensor output can be used as an input for local automation rule without any transmission delays.

Similarly, if an alarm is configured for the element used as destination for a virtual sensor, you can force the device online before time, if a calculation meets specific criteria.

Related to