Service Reference

APIs

Django API

This is the reference implementation for an BEMCom API service.

Configuration

Ports

This service serves plain HTTP only.

Note: If the API service should be exposed on the public it is absolutely important to place a reverse proxy like nginx in front of the service to secure the connection with HTTPS.

Port	Usage/Remarks
8080	REST interface and admin user interface on plain HTTP

Environment Variables

Enironment Variable	Example Value	Usage/Remarks
MQTT_BROKER_HOST	broker.domain.de	The DNS name or IP address of the MQTT broker. localhost will not work, use the full DNS name of the host machine instead.
MQTT_BROKER_HOST	1883	The port of the MQTT broker. Defaults to 1883.
ACTIVATE_CONTROL_EXTENSION	TRUE	If set to TRUE (the string) will make the additional endpoints (on REST API) and additional configuration options (in Admin UI) available that are required to interact with controllers. In particular this enables the *setpoint/ and *schedule REST endpoints that allow interaction with Schedule and Setpoint messages. Furthermore, the Controllers, Controlled datapoints, Datapoint setpoints and Datapoint schedule pages will be activated in the administration UI. Note that some of these endpoints/pages depend also on ACTIVATE_HISTORY_EXTENSION. Defaults to FALSE to keep the interface clean for all those who don’t need the controller functionality.
ACTIVATE_HISTORY_EXTENSION	TRUE	If set to TRUE (the string) the API service will record all Value, Setpoint and Schedule Message it receives from the broker in the database. Note that this can be a lot of data, especially for low resource devices like a RaspberryPI. Setting this flag will expose *value, *setpoint, and *schedule REST endpoints to interact with the recorded values. Furthermore, the Datapoint values, Datapoint setpoints and Datapoint schedule pages will be activated in the administration UI. Note that some of these endpoints/pages also depend on ACTIVATE_CONTROL_EXTENSION. Defaults to FALSE to keep the resource usage small and the interface clean by default.
LOGLEVEL	INFO	Defines the log level. Should be one of the following strings: DEBUG, INFO, WARNING, ERROR, CRITICAL. See the Django docs on logging for details. Defaults to INFO.
DJANGO_DEBUG	FALSE	If set to TRUE (the string) will activate the debug mode of django, which should only be used while developing not during production operation. Defaults to False
DJANGO_ADMINS	‘[“John”, “john@example.com”]’	Must be a valid JSON string. Is set to ADMINS setting of Django. Defaults to an empty list.
DJANGO_SECRET_KEY	oTg2aWkM…	Can be used to specify the SECRET_KEY setting of Django. Defaults to a random sequence of 64 chars generated on container startup. Note that changing the secret key will invalidate all cookies and thus force all user to login again.
DJANGO_ALLOWED_HOSTS	[“bemcom.domain.com”]	A list of fully qualified hostnames the API service should be hosted on. Must be encoded as JSON string. See the ALLOWED_HOSTS setting of Django for details. Defaults to '["localhost"]', which means that API can only be accessed from the local machine.
DJANGO_SUPERUSER_USERNAME	admin	If username, password and email is provided will attempt to create a superuser account with these credentials.
DJANGO_SUPERUSER_PASSWORD	pass	See above.
DJANGO_SUPERUSER_EMAIL	admin@example.com	See above.
DJANGOAPIDB_HOST	timescaledb.domain.de	The DNS name or IP address of the TimescaleDB used to persist data. Note that localhost will not work, use the full DNS name of the host machine instead. If left empty defaults to use a SQLite with data stored in file source/db.sqlite3 . See Database Setup below for details.
DJANGOAPIDB_PORT	5433	The port of the TimescaleDB. Defaults to 5432.
DJANGOAPIDB_USER	johndoe	The username used for authentication at TimescaleDB. Defaults to bemcom.
DJANGOAPIDB_PASSWORD	VerySecret123	The password used for authentication at TimescaleDB. Defaults to bemcom.
DJANGOAPIDB_DBNAME	bemcom	The name of the of the database inside TimescaleDB to store the data in. Defaults to bemcom
N_MTD_WRITE_THREADS	1	The number of parallel threads the api_main/mqtt_integration.py MqttToDb class uses to push incomming MQTT messages into the Database. This must be an integer. Defaults to 1 as SQLite DBs don’t support parallel read or write operations. For TimescaleDBs Values like 32 or above give a significant increase in write throughput.
N_WORKER_PROCESSES	16	The number of parallel worker processes that are used by the production server (UVicorn) to run the application. A sane number may be roughly 2-4 times the number of cores. Defaults to 1.
ROOT_PATH	bemcom/	Use this if BEMCom is served on a subpath behind a reverse proxy.
HTTPS_ONLY	TRUE	If set to TRUE (the string) the API will serve cookies over https only.

Volumes

None for most scenarios. Eventually a volume may be used to persist the SQLite database file. See below.

Database Setup

The Django API service is intended to use a TimescaleDB to persist data, which allows good performance even if larger number of datapoint value/setpoint/schedule messages are stored.

TimescaleDB

No special configuration of TimescaleDB is necessary to use it with the Django API service. See the docker-compose.yml file of the service for an example how to start a Timescale Container. See also the TimescaleDB documentation for further details.

SQLite

SQLite database are not recommended for production use. No setup is required for just testing the container.

Please note: Some features of the Django API do not work while using SQLite. In particular this is holds for the interval parameter of the GET /datapoint/{dp-id}/value/ of the REST API endpoint, as well as any other features that uses TimescaleDBs time_bucket to aggregate data to desired intervals.

If the SQLite file should be persisted beyond the container life it is necessary to carry out the following steps:

• # Create an empty file so docker has something to mount
  touch /source/db.sqlite3
  

• Add the following line to the docker-compose file:

          volumes:
              - ./source/db.sqlite3:/source/db.sqlite3
  

Initialize the Database

If you have added a clean new database you need to initialize the database.

• If you develop locally you need to apply the migrations to create the required tables and layout. To do so execute:

  source/api/manage.py migrate
  

  Note: This is done automatically if you start up the API container.

• To create the Admin user (that you need to login into the AdminUI) use:

  • For local development:

    source/api/manage.py createsuperuser
    

  • While running in the container:

    docker exec -it django-api /source/api/manage.py createsuperuser
    

Backup & Restore

Starting from version 0.2.7 this container integrates a script to backup datapoint metadata and datapoint messages stored in the metadata database. The process uses the REST API and can thus be carried out from remote and is independent of the actual database used.

The command reference of the corresponding script can be displayed with (on bash):

docker run bemcom/django-api:<tag_of_target_django-api> python /source/api/ems_utils/simple_db_backup.py --help

An example that backs up the data of 2021-08-01 to the current working directory is the following command (on bash):

docker run --rm -v ${PWD}:/data -u "$(id -u):$(id -g)" --name django-db-backup bemcom/django-api:<tag_of_target_django-api> python /source/api/ems_utils/simple_db_backup.py -b -t <URL_of_target_django-api> -s 2021-08-01 -e 2021-08-01 -d /data/ -u <username_at_target_django-api> -p <username_at_target_django-api>

Development Checklist

Follow the following steps while contributing to the connector:

• Create a .env file with suitable configuration for your local setup. It is usually a good idea to set the DEBUG=TRUE and LOGLEVEL=DEBUG for developing.

• If you have no Database set up yet, follow the steps of Database Setup above.

• For local development (the code is executed in a python environment on the machine):

  • Install dependencies with:

    pip install -r ./source/requirements.txt
    

  • Define a an export temporary directory for Prometheus exporter (see here for details).

    export PROMETHEUS_MULTIPROC_DIR="$(mktemp -d)"
    

  • Start the MqttToDb process if needed with:

    source/api/manage.py mqtttodb
    

  • Start the development server if needed with:

    source/api/manage.py runserver
    

  • Implement your changes as well all tests to cover your changes.

  • To run the tests use (check the pytest docs for more information):

    pytest source/api/
    

• For development in the container (use MODE=DEVL in .env file for debug log output and to enable nice features of Django):

  • Start the container with:

    docker-compose up
    

  • Implement your changes as well all tests to cover your changes.

  • To run the tests use the following line in a second terminal (check the pytest docs for more information):

    docker exec django-api pytest /source/api/
    

• After everything is ready check that service also works in production mode by executing:

  DJANGO_DEBUG=FALSE docker-compose up --build
  

• Update the tag (version) in source/api/api_main/settings.py and execute the shell script to build an updated image.

  # This will fail if not all tests are passed.
  bash build_docker_image.sh
  

• Document your changes and new tag by appending the list below.

• git add, commit and push.

Changelog

Tag	Changes
0.1.0	Initial functional version
0.2.0	Massive performance improvement for handling MQTT messages. Extend REST Interface to allow importing Datapoint metadata from JSON. Improve display of Datapoints in AdminUI.
0.4.0	Datapoint Value and Available Datapoint messages and a can now contain any JSON data type as value.
0.5.0	Transition to TimescaleDB. Datapoint value messages can now be resampled to intervals (operation happens in DB) with REST parameter. Restore function implemented. Note: this is a breaking update. All data must be backed up manually before upgrading to this version and restored manually afterwards.
0.6.0	Django-API service now exposes Prometheus metrics to support monitoring of BEMCom applications.
0.7.0	Django-API service can now be run with multiple worker processes. Added dedicated process to write MQTT messages to DB. Removed support for direct HTTPS endpoint (not supported by Gunicorn).
0.8.0	Added ACTIVATE_CONTROL_EXTENSION and ACTIVATE_HISTORY_EXTENSION flags.

Connectors

aquametro AMBUS Net Connector

This connector allows linking metering devices connected to aquametro’s AMBUS Net device.

Supported Gateways

Manufacturer	Model	Tested?/Remarks?
aquametro	AMBUS Net	Tested. The device should in theory be able to communicate with any SOAP client. However the SOAP configuration of our test device was slightly broken, which required some tweaking of the requests. This is incorporated in the connector.

Supported Devices

The connector should be able to process sensor datapoints of any meter that is supported by the AMBUS Net device.

Manufacturer	Model	Tested?/Remarks?
aquametro	CALEC ST computation unit	Tested.
aquametro	AMFLO SONIC UFA-113 ultrasonic volume flow sensor	Tested in combination with the CALEC ST device.

Configuration

Ports

Port	Usage/Remarks
1880	Node-RED development user interface.

Environment Variables

Enironment Variable	Example Value	Usage/Remarks
AMBUSNETWS_SOAP_URL	http://127.0.0.1/AmbusNetWS/Service1.asmx	The Soap URL of the Ambus Net device.
METER_IDS	[1, 2]	The ids of the meters connected to the Ambus Net device that should be read out.
POLL_SECONDS	30	Configures how often the meters should be polled.
MAX_POLLS_PER_PERIOD	15	Configure rate limit, that is the delay between polling one and the next meter to prevent something DDOS like. This is the maximum number of requests per POLL_TIME that are sent to Ambus Net, which yields to POLL_SECONDS / MAX_POLLS_PER_PERIOD delay between two requests to Ambus Net. MAX_POLLS_PER_PERIOD should be significantly larger then number of entries in METER_IDS.

Volumes

None.

Development Checklist

Follow the following steps while contributing to the connector:

• Create a .env file with suitable configuration for your local setup.

• Optional: Update the image of the node-red-connector-template by editing source/Dockerfile

• Start the development instance with docker-compose up -d

• Edit the flows, ensure everything works as expected.

• Export the changed flows and update/create the files in ./source/flows/. The filenames should be the flows ids.

• Update the image tag in ./build_docker_image.sh and execute the shell script to build an updated image.

• Run the new image and check once more everything works as expected.

• Document your changes and new tag by appending the list below.

• git add, commit and push.

Changelog

Tag	Changes
0.1.0	Initial version.
0.1.1	Minor bug fix.
0.2.0	Update to connector template 0.3.0

Fronius Modbus Connector

This is a Modbus connector specifically for Fronius Symo GEN24 inverters. Manufacturer’s information can be found here.

The reason for this special connector is the requirement to write two registers in order to change the power limit of the inverter (see below).

TODO

This connector contains a lot of code duplicated from modbus-tcp-connector. It would be better to build the image based on the modbus-tcp-connector image and import these code pieces.

Supported Devices/Gateways

This connector is made for Fronius Symo GEN24 devices.

Manufacturer	Model	Tested/Remarks
Fronius	Symo Gen24	Tested.

Configuration

To set the power limit, the following registers have to be written:

Name	Type	Unit	Register
WMaxLimPct	uint16	% of max power (10 kW)	40242
WMaxLim_Ena	enum16		40246

When WmaxLimPct is to be written, the given value is multiplied by the scaling factor of 100 and WmaxLim_Ena is set to 1 automatically so you only have to set the desired percentage of the power. If you’d like to control those Modbus registers directly, it is recommended to use the generic Modbus TCP Connector instead.

Environment Variables

Enironment Variable	Example Value	Usage/Remarks
CONNECTOR_NAME	modbus-tcp-connector-device-name	The name of the connector. Must be unique and is used to compute the MQTT topics. Use all lowercase chars and only dashes for separation to prevent clashes with Dockers internal name resolution system.
MQTT_BROKER_HOST	broker.domain.de	The DNS name or IP address of the MQTT broker. localhost will not work, use the full DNS name of the host machine instead.
MQTT_BROKER_HOST	1883	The port of the MQTT broker.
SEND_RAW_MESSAGE_TO_DB	TRUE	If set to TRUE (that is a string of capital letters) will publish all received raw messages on topic ${CONNECTOR_NAME}/raw_message_to_db
DEBUG	TRUE	If == “TRUE” (i.e. the string) will set the loglevel of the connector the logging.DEBUG. Else is logging.INFO.
POLL_SECONDS	30	The period of polling the Modbus device/gateway for sensor datapoints.
MODBUS_MASTER_IP	192.168.0.1	The ip adress or DNS name of the Modbus master device which we want to connect to.
MODBUS_MASTER_PORT	502	The port on which the master device awaits Modbus communication.
MODBUS_CONFIG	see below.	see below.
MODBUS_MAX_RETRIES	3	Maximum number of retrying a failed read/write operation before the connector shuts down with an error. Default value is 3.
MODBUS_RETRY_WAIT_SECONDS	15	Wait time in seconds after a failed read/write operation before trying again. Defaults to 15 seconds.
MODBUS_POLL_BREAK	0.1	Wait time in seconds between two consecutive requests. Some devices react with errors if getting polled too often. Defaults to 0.0
MODBUS_DISCONNECT_BETWEEN_POLLS	TRUE	If == “TRUE” (i.e. the string) will disconnect from Modbus master device between polls. This is useful for devices that can only handle a single connection or that react with errors if POLL_SECONDS is larger then a few seconds. Defaults to FALSE.
MODBUS_W_MAX_LIM_PCT	40242	Register of “WmaxLimPct”. Change it if your inverter uses a different one than 40242. Otherwise it can beleft blank
MODBUS_W_MAX_LIM_ENA	40246	Register of “WmaxLim_Ena”. Change it if your inverter uses a different one than 40246. Otherwise it can beleft blank
The MODBUS_CONFIG allows to specify which registers should be read out by the connector as well as how to parse the values.

A JSON string is expected that is is structured like this:

{
    "read_coils": [],
    "read_discrete_inputs": [],
    "read_holding_registers": [
        {
            "address": 40071,
            "count": 46,
            "unit": 1,
            "datatypes":">fffffffffffffffffffffff"
        },
        {
            "address": 40242,
            "count": 1,
            "unit":1,
            "datatypes":">H"
        }

    ],
    "read_input_registers": [],
    "write_coil": [],
    "write_register": [
    {
        "address": 40242,
        "unit": 1,
        "datatypes": ">H",
        "scaling_factor": 100

    },
    {
        "address": 40246,
        "unit": 1,
        "datatypes": ">H",
        "scaling_factor": 1

    }
    ]
}

The keys on the first level (e.g. read_holding_registers) specify the Modbus functions that should be used to interact with the device. Modbus functions that are not used at all (e.g. read_holding_registers here) can be omitted.

It is possible to define several ranges per function, here only one range is defined for read_input_registers. Each range item is an object containing the keys address, count and unit. These keywords are forwarded to the corresponding pymodbus method used for communication using the appropriate Modbus function. Here address is the start address, i.e. the first Modbus address that is requested. count specifies how many registers/coils are requested. unit corresponds to the Modbus unit number, which allows to communicate with several devices through a Modbus master gateway. Finally, the datatypes string is only applicable to registers and specifies how the bytes should be parsed, it must be a valid struct format string.

In the example above 20 input registers starting at address 19000 are read out by the connector from a Modbus device that has the id 1. From the device manual it is known that the registers of interest hold 10 32bit float values in big-endian encoding, the datatypes format string is set accordingly. Furthermore the value on register number 19002 is scaled (i.e. multiplied with) a factor of 0.1 before it is published on the broker.

The read_coils example above requests 8 bits starting (and including) address 10.

Volumes

None.

Development Checklist

Follow the following steps while contributing to the connector:

• Update the .env file with suitable configuration for your local setup.

• Specify all required dependencies in source/requirements.txt.

• Optional: Install a local version of dependencies for local development:

  • # in service_templates/connectors/Python/
    pip install ./source/
    

  • # in the folder of your connector
    pip install -r ./source/requirements.txt
    

• Place your code under source/connector but keep the source/connector/main.py as entrypoint. Place your tests in source/connector/tests.

• Ensure all relevant tests exist and all of those are passed before preceding.

• Update the image name and tag in ./build_docker_image.sh and execute the shell script to build an updated image.

  # This will fail if not all tests are passed.
  bash build_docker_image.sh
  

• Document your changes and new tag by appending the list below.

• git add, commit and push.

Changelog

Tag	Changes
0.1.0	First productive version.

Fronius Solar API Connector

This is a connector that allows reading datapoint values from the Powerflow REST API of Fronius PV inverters.

Supported Devices

Some remarks about the devices or just nothing.

Manufacturer	Model	Tested?/Remarks?
Fronius	Symo GEN24	Tested.

Configuration

Ports

None.

Environment Variables

Enironment Variable	Example Value	Usage/Remarks
CONNECTOR_NAME	brand-new-connector	The name of the connector. Must be unique and is used to compute the MQTT topics. Use all lowercase chars and only dashes for separation to prevent clashes with Dockers internal name resolution system.
MQTT_BROKER_HOST	broker.domain.de	The DNS name or IP address of the MQTT broker. localhost will not work, use the full DNS name of the host machine instead.
MQTT_BROKER_HOST	1883	The port of the MQTT broker.
SEND_RAW_MESSAGE_TO_DB	TRUE	If set to TRUE (that is a string of capital letters) will publish all received raw messages on topic ${CONNECTOR_NAME}/raw_message_to_db
DEBUG	TRUE	If == “TRUE” (i.e. the string) will set the loglevel of the connector the logging.DEBUG. Else is logging.INFO.
POLL_SECONDS	60	The period of polling the inverter.
INVERTER_POWERFLOW_URL	http://192.168.0.44:80/solar_api/v1/GetPowerFlowRealtimeData.fcgi	The URL of the Powerflow API, including IP address (or DNS name), port and HTTP path.

Volumes

None.

Development Checklist

Follow the following steps while contributing to the connector:

• Update the .env file with suitable configuration for your local setup.

• Specify all required dependencies in source/requirements.txt.

• Optional: Install a local version of dependencies for local development:

  • # in service_templates/connectors/Python/
    pip install -e ./source/
    

  • # in the folder of your connector
    pip install -r ./source/requirements.txt
    

• Place your code under source/connector but keep the source/connector/main.py as entrypoint. Place your tests in source/connector/tests.

• Ensure all relevant tests exist and all of those are passed before preceding.

• Update the version number (aka. the tag) in source/connector/main.py before building the docker image.

• Execute ./build_docker_image.sh to build an updated image.

  # This will fail if not all tests are passed.
  bash build_docker_image.sh
  

• Document your changes and new tag by appending the list below.

• git add, commit and push.

Changelog

Tag	Changes
0.1.0	First productive version.
0.2.0	Clean up environment variables and improve docs.
0.4.0	Update to python connector template 0.4.0 (Datapoint Value and Available Datapoint messages and a can now contain any JSON data type as value.)

Homematic Connector

This is a connector to integrate eQ-3 Homematic devices.

Supported Gateways

Manufacturer	Model	Tested?/Remarks?
eQ-3	Homematic CCU3 (HmIP-CCU3)	Tested.
eQ-3	Homematic CCU2 (HM-Cen-O-TW-x-x-2)	Not tested, should nevertheless work.

Supported Devices

The connector should be able to process sensor datapoints of all existing Homematic devices. Currently only actuator datapoints of parameter type SET_TEMPERATURE are recognized automatically and exposed as available datapoints. The following devices have been tested

Manufacturer	Model	Tested?/Remarks?
eQ-3	Wall Thermostat (HM-TC-IT-WM-W-EU)	Tested, also writing the SET_TEMPERATURE datapoint. Pushing values other then allowed range (5-30.5°C) to SET_TEMPERATURE will result in an off setpoint.
eQ-3	Window Handle Sensor (HM-Sec-RHS)	Tested.
eQ-3	Heating Actuator (HM-CC-RT-DN)	Tested.

Configuration

Ports

Port	Usage/Remarks
1880	Node-RED development user interface.
${CALLBACK_BINRPC_PORT}	The external port the CCU will try to connect to for the BINRPC protocol. Should be identical for the host as internal for the container.
${CALLBACK_XMLRPC_PORT}	The external port the CCU will try to connect to for the XMLRPC protocol. Should be identical for the host as internal for the container.

Environment Variables

Enironment Variable	Example Value	Usage/Remarks
CONNECTOR_NAME	brand-new-connector	The name of the connector. Must be unique and is used to compute the MQTT topics. Use all lowercase chars and only dashes for separation to prevent clashes with Dockers internal name resolution system.
MQTT_BROKER_HOST	broker.domain.de	The DNS name or IP address of the MQTT broker. localhost will not work, use the full DNS name of the host machine instead.
MQTT_BROKER_HOST	1883	The port of the MQTT broker.
SEND_RAW_MESSAGE_TO_DB	TRUE	If set to TRUE (that is a string of capital letters) will publish all received raw messages on topic ${CONNECTOR_NAME}/raw_message_to_db
CCU_DNS_NAME	ccu.domain.de	The DNS name or IP address of the CCU to connect to.
CALLBACK_DNS_NAME	hostname.domain.de	The DNS name or IP of the machine the connector is run on. Is used by the CCU to connect and push updates.
CALLBACK_BINRPC_PORT	2069	See ports. Must be identical to the port exposed on the host to allow the Node-RED flow to send the correct port value to the CCU for callback.
CALLBACK_XMLRPC_PORT	2070	See ports. Must be identical to the port exposed on the host to allow the Node-RED flow to send the correct port value to the CCU for callback.

Volumes

None.

Development Checklist

Follow the following steps while contributing to the connector:

• Create a .env file with suitable configuration for your local setup.

• Optional: Update the image of the node-red-connector-template by editing source/Dockerfile

• Start the development instance with docker-compose up -d

• Edit the flows, ensure everything works as expected.

• Export the changed flows and update/create the files in ./source/flows/. The filenames should be the flows ids.

• Update the image tag in ./build_docker_image.sh and execute the shell script to build an updated image.

• Run the new image and check once more everything works as expected.

• Document your changes and new tag by appending the list below.

• git add, commit and push.

Changelog

Tag	Changes
0.1.0	Initial version
0.1.1	Fix bug in send command flow

KEBA P30 Connector

This is a connector to integrate the KEBA P30 charge station.

The Connector uses the UDP interface specified here.

This connector allows to integrate multiple P30 charge stations with one connector, which is in contrast to the general BEMCom design principle to use one connector per network address. However, as this connector must bind port 7090 to receive responses there can be only one connector at a time.

Supported Gateways

No gateway, the charge station communicated directly via Ethernet.

Supported Devices

The following devices have been tested

Manufacturer	Model	Tested?/Remarks?
KEBA	P30 charge station.	Tested.

Configuration

Ports

Port	Usage/Remarks
7090	UDP Port used by the charge station to receive and send messages. The charge station insists on sending the messages back on exactly this port.

Environment Variables

Enironment Variable	Example Value	Usage/Remarks
CONNECTOR_NAME	brand-new-connector	The name of the connector. Must be unique and is used to compute the MQTT topics. Use all lowercase chars and only dashes for separation to prevent clashes with Dockers internal name resolution system.
MQTT_BROKER_HOST	broker.domain.de	The DNS name or IP address of the MQTT broker. localhost will not work, use the full DNS name of the host machine instead.
MQTT_BROKER_HOST	1883	The port of the MQTT broker.
SEND_RAW_MESSAGE_TO_DB	TRUE	If set to TRUE (that is a string of capital letters) will publish all received raw messages on topic ${CONNECTOR_NAME}/raw_message_to_db
DEBUG	TRUE	If == “TRUE” (i.e. the string) will set the loglevel of the connector the logging.DEBUG. Else is logging.INFO.
POLL_SECONDS	30	The period of polling the P30 charge stations.
KEBA_P30_CHARGE_STATIONS	‘{“p30_garage”: “192.168.0.99”}’	A JSON dictionary containing names and ip addresses (or DNS names) of P30 charge stations. This defines which charge stations should be polled for data. Must be set.

Volumes

None.

Development Checklist

Follow the following steps while contributing to the connector:

• Update the .env file with suitable configuration for your local setup.

• Specify all required dependencies in source/requirements.txt.

• Optional: Install a local version of dependencies for local development:

  • # in service_templates/connectors/Python/
    pip install -e ./source/
    

  • # in the folder of your connector
    pip install -r ./source/requirements.txt
    

• Place your code under source/connector but keep the source/connector/main.py as entrypoint. Place your tests in source/connector/tests.

• Ensure all relevant tests exist and all of those are passed before preceding.

• Update the image name and tag in ./build_docker_image.sh and execute the shell script to build an updated image.

  # This will fail if not all tests are passed.
  bash build_docker_image.sh
  

• Document your changes and new tag by appending the list below.

• git add, commit and push.

Changelog

Tag	Changes
0.1.0	Initial version
0.2.0	Switch from Node-RED template to Python template to frequent connection losses due to the charge station ignoring the source port of the UDP message.
0.3.0	Add functionality to write actuator values to charge station.
0.4.0	Update to python connector template 0.4.0 (Datapoint Value and Available Datapoint messages and a can now contain any JSON data type as value.)
0.5.0	Add automatic shutdown of connector if connection to charge station times out 5 times.

KNX Connector

This is a connector to integrate KNX devices.

Supported Gateways

Note that currently only KNX IP gateways are supported that allow a TCP tunnel connection.

Manufacturer	Model	Tested?/Remarks?
JUNG	KNX Spannungsversorgung 320 mA mit IP-Schnittstelle	Tested.

Supported Devices

Any KNX device should work in theory. It might happen that some KNX datapoint types are not supported yet. Check the definition of transcoder_by_dpt_number in transcoder.py.

Configuration

Ports

None.

Environment Variables

Enironment Variable	Example Value	Usage/Remarks
CONNECTOR_NAME	brand-new-connector	The name of the connector. Must be unique and is used to compute the MQTT topics. Use all lowercase chars and only dashes for separation to prevent clashes with Dockers internal name resolution system. Beware: This is not the name of the image of the connector but of the created container.
MQTT_BROKER_HOST	broker.domain.de	The DNS name or IP address of the MQTT broker. localhost will not work, use the full DNS name of the host machine instead.
MQTT_BROKER_HOST	1883	The port of the MQTT broker.
SEND_RAW_MESSAGE_TO_DB	TRUE	If set to TRUE (that is a string of capital letters) will publish all received raw messages on topic ${CONNECTOR_NAME}/raw_message_to_db
DEBUG	TRUE	If == “TRUE” (i.e. the string) will set the loglevel of the connector the logging.DEBUG. Else is logging.INFO.
KNX_GATEWAY_HOST	knx_gw.example.com	The DNS name or IP address of the KNX gateway to connect to.
KNX_GATEWAY_PORT	3672	The corresponding port for the gateway. Defaults to 3671
KNX_DATAPOINTS	see below	see below

KNX_DATAPOINTS is a JSON object mapping from KNX group addresses to KNX datapoint types. It is necessary to tell the connector how to interpret the raw bytes received from the KNX bus.

The KNX_DATAPOINTS object must contain a sensors and actuators entry and could look like follows:

{
    "sensor": {
        "2/3/111": "DPST-3-7",
        "2/3/112": "DPST-5-1",
        "2/3/113": "DPST-1-1"
    },
    "actuator": {
        "2/3/113": "DPST-1-1"
    }
}

You may want to create the mapping from group address to datapoint type from an ETS CSV export with the following lines:

import json
import pandas as pd

knx_datapoints = pd.read_csv(file_name, sep="\t", encoding="latin-1")
json.dumps({r.Address: r.DatapointType for _, r in knx_datapoints.iterrows()}, indent=4)

Volumes

None preferably. You should only add volumes, especially file mounts if it is really necessary. It introduces a lot of trouble with user id and file ownerships stuff.

Development Checklist

Follow the following steps while contributing to the connector:

• Update the .env file with suitable configuration for your local setup.

• Specify all required dependencies in source/requirements.txt.

• Optional: Install a local version of dependencies for local development:

  • # in service_templates/connectors/Python/
    pip install -e ./source/
    

  • # in the folder of your connector
    pip install -r ./source/requirements.txt
    

• Place your code under source/connector but keep the source/connector/main.py as entrypoint. Place your tests in source/connector/tests.

• Ensure all relevant tests exist and all of those are passed before preceding.

• Update the version number (aka. the tag) in source/connector/main.py before building the docker image.

• Execute ./build_docker_image.sh to build an updated image.

  # This will fail if not all tests are passed.
  bash build_docker_image.sh
  

• Document your changes and new tag by appending the list below.

• git add, commit and push.

Changelog

Tag	Changes
0.1.0	First productive version.

Modbus/TCP Connector

This is a connector to communicate with Devices over Modbus/TCP. A short introduction to Modbus can e.g. be found here.

The following Modbus function codes are supported:

Modbus Function Code	Name
1	Read coil
2	Read discrete input
3	Read holding register
4	Read input register
5	Write single coil
6	Write single holding register
16	Write multiple holding registers

Supported Devices/Gateways

This connector should be able to communicate with any Modbus Device or Gateway. The following devices have been explicitly tested:

Manufacturer	Model	Tested?/Remarks?
Janitza	UMG-96RM-E	Tested.

Configuration

Environment Variables

Enironment Variable	Example Value	Usage/Remarks
CONNECTOR_NAME	modbus-tcp-connector-device-name	The name of the connector. Must be unique and is used to compute the MQTT topics. Use all lowercase chars and only dashes for separation to prevent clashes with Dockers internal name resolution system.
MQTT_BROKER_HOST	broker.domain.de	The DNS name or IP address of the MQTT broker. localhost will not work, use the full DNS name of the host machine instead.
MQTT_BROKER_HOST	1883	The port of the MQTT broker.
SEND_RAW_MESSAGE_TO_DB	TRUE	If set to TRUE (that is a string of capital letters) will publish all received raw messages on topic ${CONNECTOR_NAME}/raw_message_to_db
DEBUG	TRUE	If == “TRUE” (i.e. the string) will set the loglevel of the connector the logging.DEBUG. Else is logging.INFO.
POLL_SECONDS	30	The period of polling the Modbus device/gateway for sensor datapoints.
MODBUS_MASTER_IP	192.168.0.1	The ip adress or DNS name of the Modbus master device which we want to connect to.
MODBUS_MASTER_PORT	502	The port on which the master device awaits Modbus communication.
MODBUS_CONFIG	see below.	see below.
MODBUS_MAX_RETRIES	3	Maximum number of retrying a failed read/write operation before the connector shuts down with an error. Default value is 3.
MODBUS_RETRY_WAIT_SECONDS	15	Wait time in seconds after a failed read/write operation before trying again. Defaults to 15 seconds.
MODBUS_POLL_BREAK	0.1	Wait time in seconds between two consecutive requests. Some devices react with errors if getting polled too often. Defaults to 0.0
MODBUS_DISCONNECT_BETWEEN_POLLS	TRUE	If == “TRUE” (i.e. the string) will disconnect from Modbus master device between polls. This is useful for devices that can only handle a single connection or that react with errors if POLL_SECONDS is larger then a few seconds. Defaults to FALSE.
MODBUS_LSRF_WORD_ORDER	TRUE	If == “TRUE” (i.e. the string) will swap the registers of floats before parsing. This allows data exchange with systems that follow the least significant register first convention.

The MODBUS_CONFIG allows to specify which registers should be read out or written to by the connector as well as how to parse the values.

A JSON string is expected that is is structured like this:

{
    "read_coils": [
        {
            "address": 10,
            "count": 8,
            "unit": 1,
        }
    ],
    "read_discrete_inputs": [], # Similar to "read_coils"
    "read_holding_registers": [
        {
            "address": 19000,
            "count": 20,
            "unit": 1,
            "datatypes": ">ffffffffff",
            "scaling_factors": {
                "19002": 0.1
            }
        }
    ],
    "read_input_registers": [], # Similar to "read_coils"
    "write_coil": [
        {
            "address": 10,
            "unit": 1,
            "example_value": "true"
        }
    ],
    "write_register": [
        {
            "address": 19000,
            "unit": 1,
            "datatypes": ">h",
            "example_value": "22"
        }
    ],
    "write_registers": [
        {
            "address": 19001,
            "unit": 1,
            "datatypes": ">f",
            "example_value": "21.0"
        }
    ]
}

The keys on the first level (e.g. read_holding_registers) specify the Modbus functions that should be used to interact with the device. Modbus functions that are not used at all (e.g. read_input_registers here) can be omitted.

It is possible to define several ranges per function, here only one range is defined for read_input_registers. Each range item is an object containing keys. The following table defines which keys must be provided per modbus function.

Key	Applicable to functions	Explanation
address	all	Specifies the start address, i.e. the first Modbus address that is requested.
count	read_*	Specifies how many registers/coils are requested.
unit	all	Corresponds to the Modbus unit number, which allows to communicate with several devices through a Modbus master gateway.
datatypes	All that interact with registers	Specifies how values are parsed from/to bytes , it must be a valid struct format string.
scaling_factors	read_discrete_inputs, read_holding_registers	Allows scaling the values received from the Modbus device before publishing on BEMCom (i.e. multipling the value with the factor).
example_value	write_*	Defines an example value for display in the API service.

The read_coils example above requests 8 bits starting (and including) address 10.

In the example above, the range defined under read_coils requests 8 bits starting (and including) address 10 from a Modbus device that has the id 1.

The range under read_holding_registers configures the connector to read 20 registers starting at address 19000 from a Modbus device that has the id 1. From the device manual it is known that the registers of interest hold 10 32bit float values in big-endian encoding, the datatypes format string is set accordingly. Furthermore the value on register number 19002 is scaled (i.e. multiplied with) a factor of 0.1 before it is published on the broker.

The range under write_coil specifies that a single Bool can be written to coil 10 of Modbus device with id 1. The example value is the boolean value True.

The range under write_register specifies that a single integer can be written to register 19000 of Modbus device with id 1. The example value is 22.

The range under write_registers specifies that a single float can be written to registers 19001 and 19002. This is as the value of datatypes defines a float 32 in big-endian encoding, which will thus span two registers. How many registers are affected is determined automatically based on the data type. Again the value is written a Modbus device with id 1 and the example value is 21.0.

Volumes

None.

Development Checklist

Follow the following steps while contributing to the connector:

• Update the .env file with suitable configuration for your local setup.

• Specify all required dependencies in source/requirements.txt.

• Optional: Install a local version of dependencies for local development:

  • # in service_templates/connectors/Python/
    pip install -e ./source/
    

  • # in the folder of your connector
    pip install -r ./source/requirements.txt
    

• Place your code under source/connector but keep the source/connector/main.py as entrypoint. Place your tests in source/connector/tests.

• Ensure all relevant tests exist and all of those are passed before preceding.

• Update the version number (aka. the tag) in source/connector/main.py before building the docker image.

• Execute ./build_docker_image.sh to build an updated image.

  # This will fail if not all tests are passed.
  bash build_docker_image.sh
  

• Document your changes and new tag by appending the list below.

• git add, commit and push.

Changelog

Tag	Changes
0.1.0	First productive version.
0.2.0	Can read coils and discrete inputs now and allows application of scaling factors.
0.3.0	Allow disconnecting between polls and breaks between requests in one poll run.
0.4.0	Update to python connector template 0.4.0 (Datapoint Value and Available Datapoint messages and a can now contain any JSON data type as value.)
0.5.0	Connector can now write to coils and registers. Unit id is now part of the internal datapoint id to prevent collisions while interacting with the same addresses from different devices.
0.6.0	Update to python connector template 0.5.0 (It is now possible to stop processing messages in run_sensor_flow.)
0.7.0	Update to python connector template 0.8.0 with reduced image size and enabled ARM builds.
0.8.0	Add option for handling the least significant register first convention for floats.

MQTT Connector

This is a connector to integrate data that is published on some MQTT broker (not the BEMCom app internal message broker.)

Sending values to actuator datapoints, i.e. to publish on the broker is currently not implemented.

Supported Devices/Gateways

Any broker that implemented MQTT version 3.1.1 should work.

Configuration

Ports

None.

Environment Variables

Enironment Variable	Example Value	Usage/Remarks
CONNECTOR_NAME	brand-new-connector	The name of the connector. Must be unique and is used to compute the MQTT topics. Use all lowercase chars and only dashes for separation to prevent clashes with Dockers internal name resolution system.
MQTT_BROKER_HOST	broker.domain.de	The DNS name or IP address of the MQTT broker. localhost will not work, use the full DNS name of the host machine instead.
MQTT_BROKER_HOST	1883	The port of the MQTT broker.
SEND_RAW_MESSAGE_TO_DB	TRUE	If set to TRUE (that is a string of capital letters) will publish all received raw messages on topic ${CONNECTOR_NAME}/raw_message_to_db
DEBUG	TRUE	If == “TRUE” (i.e. the string) will set the loglevel of the connector the logging.DEBUG. Else is logging.INFO.
REMOTE_MQTT_BROKER_HOST	broker.domain.de	The DNS name or IP address of the remote MQTT broker.
REMOTE_MQTT_BROKER_PORT	1883	The port of the remote MQTT broker.
REMOTE_MQTT_BROKER_USE_TLS	FALSE	If == “TRUE” (i.e. the string), will use TLS to encrypt the connection.
REMOTE_MQTT_BROKER_CA_FILE	Wrap the multiline ca file with quotes.	A CA certificate (full chain) in pem format. If provided will use this CA certificate to verify that server certificate.
REMOTE_MQTT_BROKER_USERNAME	john-doe	If not empty will try to login at the remote broker with this username.
REMOTE_MQTT_BROKER_PASSWORD	very-secret	If not empty (and username not empty) will try to login at the remote broker with this password.
REMOTE_MQTT_BROKER_TOPIC_MAPPING	see below	A json string defining which topics should be forwarded to which datapoints. See Readme.md for details.
REMOTE_MQTT_BROKER_PARSE_JSON	FALSE	If == “TRUE” (i.e. the string), will try to parse the payload of the message received from the remote broker as JSON. Defaults to FALSE

The content of REMOTE_MQTT_BROKER_TOPIC_MAPPING should be JSON string structured like this:

{
    "sensor_topics": {
        "sensor/topic/1": {},
        "sensor/topic/with/single/+/wildcard": {},
        "sensor/topic/with/mulitlevel/wildcard/#": {}
    },
    "actuator_topics": {
        "actuator/topic/1": {
            "example_value": "22.0"
        }
    }
}

For "sensor_topics" just the topic strings (i.e. the keys) are relevant for now (this may change in future), wildcards are allowed. The example values are taken from the first message arriving per topic. The connector will try to parse any incoming payload as JSON.

For "actuator_topics" the keys also specify the topics on which the value message should be published. Note that wildcards are not allowed for the actuator datapoints, as it would not be clear on which topic to publish in these cases.

Volumes

None.

Development Checklist

Follow the following steps while contributing to the connector:

• Update the .env file with suitable configuration for your local setup.

• Specify all required dependencies in source/requirements.txt.

• Optional: Install a local version of dependencies for local development:

  • # in service_templates/connectors/Python/
    pip install -e ./source/
    

  • # in the folder of your connector
    pip install -r ./source/requirements.txt
    

• Place your code under source/connector but keep the source/connector/main.py as entrypoint. Place your tests in source/connector/tests.

• Ensure all relevant tests exist and all of those are passed before preceding.

• Update the version number (aka. the tag) in source/connector/main.py before building the docker image.

• Execute ./build_docker_image.sh to build an updated image.

  # This will fail if not all tests are passed.
  bash build_docker_image.sh
  

• Document your changes and new tag by appending the list below.

• git add, commit and push.

Changelog

Tag	Changes
0.0.1	Some work in progress development version.
0.1.0	First productive version.
0.4.0	Update to python connector template 0.4.0 (Datapoint Value and Available Datapoint messages and a can now contain any JSON data type as value.)
0.6.0	Update to python connector template 0.6.0

OCPP Connector

This is a connector to communicate with ocpp-capable charging stations using OCPP 1.6. The library used is provided by The Mobility House (https://github.com/mobilityhouse/ocpp). The Open ChargePoint Protocol (OCPP) defines the central system (here BEMCom) as a server, while the ChargePoint acts as a client. This is different to many hardware communication protocols.

Implementation

Besides the classes ActuatorFlow, SensorFlow and Connector there is a class called ChargePoint that inherits from the corresponding class in the ocpp module from The Mobility House. This class contains the protocol-specific messages. OCPP uses websockets. The messages from the charge point are routed to the corresponding method where the response to the charge point is defined. If there are values to be sent to the BEMCom message broker, a callback method with the name “sensor_flow_handler” is executed. Messages passed to it will be made available in the BEMCom framework by using the method “run_sensor_flow” of the Connector class. Once the connector starts, the synchronous DeviceDispatcher starts a sever using asyncio. Once the charging station connects to the server, the ChargePoint is instantiated and the communication begins.

Configuration

OCPP_CONFIG	Usage/Remarks
execute_send_charging_profile	No parameters, defines the actuator datapoint so that (constant) charging profiles can be sent. Interpreted as maximum power values in W.
execute_trigger_message	No parameters, defines the actuator datapoint to trigger the chargepoint to send specific messages like meter values.

Ports

Port	Usage/Remarks
9000	Port of the BEMCom Host (server), not the charging station. Has to be configured in the charging station as well if it is desired to use another one.

Environment Variables

Enironment Variable	Example Value	Usage/Remarks
CONNECTOR_NAME	ocpp-connector	The name of the connector. Must be unique and is used to compute the MQTT topics. Use all lowercase chars and only dashes for separation to prevent clashes with Dockers internal name resolution system.
MQTT_BROKER_HOST	broker.domain.de	The DNS name or IP address of the MQTT broker. localhost will not work, use the full DNS name of the host machine instead.
MQTT_BROKER_HOST	1883	The port of the MQTT broker.
SEND_RAW_MESSAGE_TO_DB	TRUE	If set to TRUE (that is a string of capital letters) will publish all received raw messages on topic ${CONNECTOR_NAME}/raw_message_to_db
DEBUG	TRUE	If == “TRUE” (i.e. the string) will set the loglevel of the connector the logging.DEBUG. Else is logging.INFO.
OCPP_CONFIG	“execute_send_charging_profile”	list of actuator datapoints. Have to begin with “execute_”
OCPP_PORT	9000	Port of the Server (BEMCom host)

Volumes

None preferably. You should only add volumes, especially file mounts if it is really necessary. It introduces a lot of trouble with user id and file ownerships stuff.

Usage

Charging profiles consist of a single power value (in Watt) that remain active until the next value is sent. The messages sent from the chargepoint, like meter values, can be triggered using the datapoint “execute_trigger_message”. There, the following vales are allowed:

Value in API	Message
0	“BootNotification”
1	“DiagnosticsStatusNotification”
2	“FirmwareStatusNotification”
3	“Heartbeat”
4	“MeterValues”
5	“StatusNotification”

For more information according OCPP, you can download the specification here: (https://www.openchargealliance.org/downloads/)

Development Checklist

Follow the following steps while contributing to the connector:

• Update the .env file with suitable configuration for your local setup.

• Specify all required dependencies in source/requirements.txt.

• Optional: Install a local version of dependencies for local development:

  • # in service_templates/connectors/Python/
    pip install -e ./source/
    

  • # in the folder of your connector
    pip install -r ./source/requirements.txt
    

• Place your code under source/connector but keep the source/connector/main.py as entrypoint. Place your tests in source/connector/tests.

• Ensure all relevant tests exist and all of those are passed before preceding.

• Update the image name and tag in ./build_docker_image.sh and execute the shell script to build an updated image.

  # This will fail if not all tests are passed.
  bash build_docker_image.sh
  

• Document your changes and new tag by appending the list below.

• git add, commit and push.

Changelog

Tag	Changes
0.1.0	First successful communication.
0.2.0	First successfully sent charging profile
0.2.1	Possible to trigger and receive meter values
0.2.2	Bugfix logs, increasing robustnes.

Senertec Dachs Connector

This is a connector to integrate the Senertec Dach combined heat and power plant. It uses the REST interface of the Dachs for communication.

Please refer to the official users manuel “Beschreibung der GLT-Schnittstelle - Dachs Ethernet” for detailed information about the available Datapoints.

Supported Devices

Manufacturer	Model	Tested?/Remarks?
Senertec	Dachs 5.5	Tested.

Configuration

Ports

Port	Usage/Remarks
1880	Node-RED development user interface.

Environment Variables

Enironment Variable	Example Value	Usage/Remarks
CONNECTOR_NAME	brand-new-connector	The name of the connector. Must be unique and is used to compute the MQTT topics. Use all lowercase chars and only dashes for separation to prevent clashes with Dockers internal name resolution system.
MQTT_BROKER_HOST	broker.domain.de	The DNS name or IP address of the MQTT broker. localhost will not work, use the full DNS name of the host machine instead.
MQTT_BROKER_HOST	1883	The port of the MQTT broker.
SEND_RAW_MESSAGE_TO_DB	TRUE	If set to TRUE (that is a string of capital letters) will publish all received raw messages on topic ${CONNECTOR_NAME}/raw_message_to_db
POLL_SECONDS	30	Wait time in seconds between polling the CHP for sensor data.
DACHS_IP	dachs.example.com	IP address or DNS name of the Dachs CHP that should be communicated with. We expect the REST API exposed on port 8080 of that address.

Volumes

None.

Development Checklist

Follow the following steps while contributing to the connector:

• Create a .env file with suitable configuration for your local setup.

• Optional: Update the image of the node-red-connector-template by editing source/Dockerfile

• Start the development instance with docker-compose up -d

• Edit the flows, ensure everything works as expected.

• Export the changed flows and update/create the files in ./source/flows/. The filenames should be the flows ids.

• If you have added functionality that needs usernames and passwords, you also need to export the flow_cred.json file from the container, as the credentials are not exported with the flows. You might do this with something like:

  # Execute this in the root directory of the connector.
  CONTAINER_NAME=$(cat docker-compose.yml | grep container_name | cut -d : -f 2 | xargs )
  docker cp $CONTAINER_NAME:/data/flows_cred.json ./source/
  

  Please note: The credentials are stored unencrypted. If the connector needs credentials that are not well known (e.g. provided in the user manual of the device), it is better set them at runtime via environment variable. This way the credentials will never appear in the repository and can be set by each user to the required values.

• Update the image tag in ./build_docker_image.sh and execute the shell script to build an updated image.

• Run the new image and check once more everything works as expected.

• Document your changes and new tag by appending the list below.

• git add, commit and push.

Changelog

Tag	Changes
0.1.0	First productive version. Read only yet.
0.2.1	Added support for requesting Dachs operation via actuator datapoint.
0.3.0	Update to connector template 0.3.0

Socket Connector

This is a connector that allows receiving messages that are published by a device or gateway on a socket. It is not designed to pull information, as the latter usually requires more device/gateway specific interaction for which it very likely more simple to to develop a custom connector then handling every special case here. Currently only TCP is implemented.

Supported Devices/Gateways

This should work with any device or machine that implements standard TCP and uses it to publish information.

Configuration

Ports

None.

Environment Variables

Enironment Variable	Example Value	Usage/Remarks
CONNECTOR_NAME	brand-new-connector	The name of the connector. Must be unique and is used to compute the MQTT topics. Use all lowercase chars and only dashes for separation to prevent clashes with Dockers internal name resolution system.
MQTT_BROKER_HOST	broker.domain.de	The DNS name or IP address of the MQTT broker. localhost will not work, use the full DNS name of the host machine instead.
MQTT_BROKER_HOST	1883	The port of the MQTT broker.
SEND_RAW_MESSAGE_TO_DB	TRUE	If set to TRUE (that is a string of capital letters) will publish all received raw messages on topic ${CONNECTOR_NAME}/raw_message_to_db
DEBUG	TRUE	If == “TRUE” (i.e. the string) will set the loglevel of the connector the logging.DEBUG. Else is logging.INFO.
SERVER_IP	192.168.0.55	The DNS name or IP address of the target device or gateway.
SERVER_PORT	587	The port which the connector should connect to on the target device or gateway.
RECV_BUFSIZE	1024	The buffersize used while receiving information. The data from the device or gateway should not exceed this many bytes as the raw message will else be split up and may end up unreadable. See also the socket docs for more information. Defaults to 4096.
RECV_TIMEOUT	15	Max wait time before a message is expected. Will shutdown connector if no message is received after that time (to allow a restart and reconnection). 0 means we expect a message immediately (non-blocking mode). Negative values wait as long as possible (blocking mode). See here for details.
PARSE_AS	YAML	Defines how the raw data should be parsed as objects. Either YAML or JSON is supported. Defaults to JSON.

Volumes

None.

Development Checklist

Follow the following steps while contributing to the connector:

• Update the .env file with suitable configuration for your local setup.

• Specify all required dependencies in source/requirements.txt.

• Optional: Install a local version of dependencies for local development:

  • # in service_templates/connectors/Python/
    pip install -e ./source/
    

  • # in the folder of your connector
    pip install -r ./source/requirements.txt
    

• Place your code under source/connector but keep the source/connector/main.py as entrypoint. Place your tests in source/connector/tests.

• Ensure all relevant tests exist and all of those are passed before preceding.

• Update the version number (aka. the tag) in source/connector/main.py before building the docker image.

• Execute ./build_docker_image.sh to build an updated image.

  # This will fail if not all tests are passed.
  bash build_docker_image.sh
  

• Document your changes and new tag by appending the list below.

• git add, commit and push.

Changelog

Tag	Changes
0.1.0	First productive version.
0.4.0	Update to python connector template 0.4.0 (Datapoint Value and Available Datapoint messages and a can now contain any JSON data type as value.)
0.5.0	Add timeout to allow automatic restarts if connection is lost.

Controllers

Python Controller

This is a Python based implementation of a Controller service. It listens to all setpoint and schedule messages (see the message format documentation for details) published on the broker, whereby the setpoint reflects the users desired value and the schedule the optimal value computed by an optimization process. Based on these messages the controller computes the value for the actuator datapoint.

The controller ensures the following points:

• That each item in a setpoint message is executed, as long as the value for to_timestamp is not in the past. Executed means thereby that the fields preferred_value, and if existing acceptable_values or min_value and max_value are loaded and used while computing the value for the actuator datapoint.

• That each item in a schedule message is executed, as long as the value for to_timestamp is not in the past. Executed means thereby that the field value, is loaded and used while computing the value for the actuator datapoint.

• values for actuators are only sent if the value changes, to prevent permanently sending the same information to the devices.

• That logic for computing the actuator value at any time are as follows:

  • If neither a setpoint nor a schedule exist the value of the actuator will not be changed at all.

  • If no setpoint exists but a schedule, the value defined in the schedule will be sent to the actuator.

  • If no schedule exists but a setpoint is, the preferred_value of the setpoint message is used as value for the actuator.

  • If setpoint and schedule exist but the datapoint is not of type continuous_numeric or discrete_text/numeric (i.e. the setpoint message contains no flexibility as neither acceptable_values or min_value and max_value are defined) the schedule message is ignored.

  • If setpoint and schedule exist and the datapoint is of type continuous_numeric (i.e. the fields min_value and max_value exist) the the value of the schedule is sent to the actuator, as least as long as the corresponding sensor datapoint is with the range specified by min_value and max_value. If the sensor value leaves the allowed range the preferred_value is sent as value to the actuator.

  • If setpoint and schedule exist and the datapoint is of type discrete (i.e. the field acceptable_values exists) the the value of the schedule is sent to the actuator, as least as long as the corresponding sensor datapoint is with the range specified byacceptable_values. If the sensor value leaves the allowed range the preferred_value is sent as value to the actuator.

Configuration

Ports

None.

Environment Variables

Enironment Variable	Example Value	Usage/Remarks
MQTT_BROKER_HOST	broker.domain.de	The DNS name or IP address of the MQTT broker. localhost will not work, use the full DNS name of the host machine instead.
MQTT_BROKER_HOST	1883	The port of the MQTT broker.
MQTT_TOPIC_CONTROLLED_DATAPOINTS	python_controller/controlled_datapoints	The topic on which the controller listens for configuration data.

Volumes

None.

Changelog

Tag	Changes
0.1.0	Initial version

MQTT Brokers

Mosquitto MQTT Broker

MQTT message broker service based on the Mosquitto docker image, see https://hub.docker.com/_/eclipse-mosquitto for details. This provides currently no additional value to using the official Mosquitto image directly, but is here for consistency. However the latter may change in future.

Changelog

Tag	Changes
0.1.0	Initial version, uses Mosquitto 1.6.10

Raw Message DBs

Mongo Raw Message DB

This is a MongoDB based Raw Message DB for BEMCom.

Configuration

Ports

Port	Usage/Remarks
27017	Port to directly connect to MongoDB. Opening this port is not necessary for storing raw messages by connectors, only for access with external tools. You should definitely configure the passwords before opening this port.

Environment Variables

Enironment Variable	Example Value	Usage/Remarks
MQTT_BROKER_HOST	broker.domain.de	The DNS name or IP address of the MQTT broker. localhost will not work, use the full DNS name of the host machine instead.
MQTT_BROKER_PORT	1883	The port of the MQTT broker.
MONGO_INITDB_ROOT_USERNAME	mongo-admin	If set, will automatically create an admin account on the very first startup. See here (Section Environment Variables) for details.
MONGO_INITDB_ROOT_PASSWORD	very!secret&	See above.
MONGO_USERNAME	mongo-user	Starts the MongoDB with access control if not left blank. If not blank is used as the username of the account that is used by the mqtt-integration.py script to connect to the MongoDB.
MONGO_PASSWORD	very!secret&	The password of the account that is used by the mqtt-integration.py script to connect to the MongoDB.
MONGO_LOGIN_DB	admin	The Login DB of the account that is used by the mqtt-integration.py script to connect to the MongoDB. Should be set to admin unless you have changed it manually.
MQTT_TOPIC_ALL_RAW_MESSAGES	+/raw_message_to_db	The topic on the broker on which the raw messages will be published. Stick to the convention of the example value given here to prevent messy surprises.
MQTT_INTEGRATION_LOG_LEVEL	INFO	The level of log messages that will be logged to stdout of the container for the mqtt-integration.py script. Can be any of DEBUG, INFO, WARNING, ERROR, Critical. The Info is reasonable for normal operation.

Volumes

Path in Container	Usage/Remarks
/data/db	Folder in which MongoDB stores its DB. Is required to persist the Database. The folder on the host machine must exist before the container is run for the first time and the container must be started with the user id of the user who own the folder on the host machine.

Inital Startup Checklist

This creates a non-superuser account for the mqtt integration script to use.

Note: If you have populated MONGO_INITDB_ROOT_USERNAME and MONGO_INITDB_ROOT_USERNAME on the first startup you may need to extend these steps with a login.

• Create an empty directory for the containers persistent files. Set USER_ID and GROUP_ID to the ids of the user who owns the directory. Set MONGODB_VOLUME to that directory.

• Spin up the container with blank value for MONGO_USERNAME.

• Create a user for the Python MQTT Integration script. See https://docs.mongodb.com/manual/tutorial/create-users/ for details. Give the user read and write permissions for the database bemcom_db. E.g. with this:

  docker exec -it  your-bemcom-app-mongo-raw-message-db mongo
  use admin
  db.createUser(
        {
                user: "bemcom",
                pwd: passwordPrompt(),
                roles: [
                        { role: "readWrite", db: "bemcom_db" }
        ]
        }
  )
  

• Stop the container.

• Set the remaining environment variables (MONGO_USERNAME, MONGO_PASSWORD, MONGO_LOGIN_DB) according to the values you defined previously.

• Restart the container, inspect the logs to verify that everything works as expected.

Changelog

Tag	Changes
0.0.1	Initial version, some bug while shutting down container.
0.1.0	Should be fully functional now. Container shutdown stops everything gracefully now.
0.1.1	Improve credentials handling.
0.2.0	Update Mongo to 5.0, and allow automatic admin account creation.

Tools

Demo Device

A service that should behave like a simple Modbus device. This tool is solely intended for demonstration of BEMCom. It has no practical value besides this function.

The demo can be interpreted as as a single room for which the temperature is measured and an actor (say an AC system that can heat and cool) exists to manipulate the room temperature.

The demo device exposes the following registers:

Type	Register	Data Type	Description
Input Register	1	16bit float	The current room temperature in °C.
Holding Register	1	16bit float	The setpoint for the room temperature in °C. Values below 15°C and above 30° are clipped to that range.

Configuration

Ports

Port	Usage/Remarks
502	Container accepts ModbusTCP connections on this port. Use Modbus unit ID 1.

Environment Variables

Enironment Variable	Example Value	Usage/Remarks
LOG_VALUES	FALSE	If set to TRUE (the string) will log every change to the values with level INFO. Default to TRUE

Volumes

None.

Changelog

Tag	Changes
0.1.0	First productive version.

EMP Link

This is a simple tool that automatically pushes new data to the FZI Energy Management Panel.

Configuration

Ports

None.

Environment Variables

Enironment Variable	Example Value	Usage/Remarks
ORIGIN	EmpLinkTest	The value for the origin when pushing datapoints up.
AUTO_RELOAD	TRUE	If TRUE (i.e. the string) will host the service with auto reloading enabled, that is, the service restarts if files have changed. This is nice for development but should not be enabled in production.
MQTT_BROKER_HOST	broker.domain.de	The DNS name or IP address of the MQTT broker. localhost will not work, use the full DNS name of the host machine instead.
MQTT_BROKER_PORT	18830	The port of the MQTT broker. Defaults to 1883.
BEMCOM_API_URL	https://bemcom.example.com	The URL of the BEMCom API. Used for fetching the datapoint metadata.
BEMCOM_API_VERIFY_SSL	FALSE	If == “FALSE” (i.e. the string) will disable certificate checking for the BEMCom API connection. Useful if self signed certificates are used but a potential security risk. See also the requests docs on the topic. Defaults to TRUE.
BEMCOM_API_USERNAME	jon_doe	The username (HTTP basic auth) to use for connecting to BEMCom API.
BEMCOM_API_PASSWORD	verySecret123	The corresponding password.
EMP_API_URL	https://emp.example.com	The URL of the EMP API. All new data is pushed there.
EMP_API_VERIFY_SSL	TRUE	Like BEMCOM_API_VERIFY_SSL but for the EMP API.
EMP_API_USERNAME	jon_doe	The username (HTTP basic auth) to use for connecting to EMP API.
EMP_API_PASSWORD	verySecret123	The corresponding password.

Volumes

None.

Development Quick Start

On a Linux system you can start developing on the code quickly by following these steps.

• In a terminal execute:

  docker-compose down -v && USER_ID=$(id -u) GROUP_ID=$(id -g) docker-compose up --build
  

• For automatic execution of tests run the following in a second terminal:

  docker exec -it emp-devl auto-pytest /source/emp/
  

• For an interactive python terminal execute:

  docker exec -it emp-devl /opt/conda/bin/python /source/emp/manage.py shell
  

Grafana Tool

This service provides a Grafana instance with custom plug-ins that allow:

• bemcom-django-api: Data retrieval from the REST interface of the BEMCom Django-API service.

• prediction-service-api: Request and display data provided via the prediction service REST API.

• nwpdata-service-api: Request and display of data provided via the nwpdata service REST API.

Please note that the prediction service and nwpdata service REST API have not been published to the public yet. You can simple ignore these plugins thus.

Configuration

Ports

Port	Usage/Remarks
3000	Default Grafana Ui port

Environment Variables

Enironment Variable	Example Value	Usage/Remarks
GF_SECURITY_ADMIN_USER	admin	The default admin user. Defaults to bemcom
GF_SECURITY_ADMIN_PASSWORD	very!secret&	The default admin password. Defaults to bemcom

Volumes

Path in Container	Usage/Remarks
/var/lib/grafana/grafana.db	Grafana SQLite database file. Store on local file system to persist grafan settings.
/var/lib/grafana-plugins/bemcom-django-api	Allows mounting in the custom plugin. Use for development only.
/var/lib/grafana-plugins/nwpdata-service-api	Allows mounting in the custom plugin. Use for development only.
/var/lib/grafana-plugins/prediction-service-api	Allows mounting in the custom plugin. Use for development only.

Hint: Add custom volumes to persist changes in Grafana.

Usage Instructions

Initial Setup

• Ensure that the ${GRAFANA_DB_FILE} exists and has read/write permissions for the user running the container.

Data source configuration

The datasources are configured by providing the following settings:

• required:

  • Name of the datasource inside grafana

  • url to the APIs root. For example http://django-api.example.com:8000/

• optional:

  • use basic authentication

  • basicAuth user

  • basicAuth password

  • skip TLS verification - check this to ignore self signed certificates. important this option renders https insecure. To enable verification and security a custom CA Authority for verification of certificates is needed.

Optional settings are initally only of importance to the BEMCom API.

Query configuration

BEMCOM API plugin

A query can either display meta data on the API or timeseries data.

Meta data is toggled by a switch. The received table-like data gives information on all available datapoints.

Timeseries data can be of the above described data types.

• Simply choose the datapoint by its short name and the datatype. The dropdown also features an autoselection when typing.
  

• Optionally you can define a frequency of the queried entries and an offset of the frequency to define the time range over which the average is taken. Warning: This option is not implemented in BEMCom, yet (Juli 2021).
  

• A custom name and scaling factor can optionally be defined. Click “apply” or press enter to commit changes here.

Nwpdata and stochastic prediction service API plugin

Use the textfield to define the parameters needed for the request as JSON string. The actual format depends on the service queried and can be found on the Swagger UI at the root URL of the respective service. The following settings work well with the FZI internal DWD MOSMIX Data Service for example expects parameters as followed:

{
  "station": "Q712",
  "start_timestamp": $from,
  "end_timestamp": $to,
  "updates": "latest",
  "mosmix_parameters": ["TTT"]
}

Development

The plugin is written in typescript and based on the grafana plugin creator and tutorial.

You need to have node.js and yarn installed for the following commands. If you have conda installed you can install these with:

conda create -n node -c conda-forge nodejs==12.* yarn

To see your changes in a local grafana instance, change the docker-compose.yml file to development mode (see comments in the file) and start grafana as docker container.

Then, to actually work on the plugin:

• Install all needed modules for development from within the bemcom-django-api folder with
  yarn install

• Make your changes to the source files under ./source/grafana_plugins/bemcom-django-api/src

• Hot build the plugin to see changes in the browser on reload with yarn dev --watch

• Run the tests with yarn test

• Build the plugin for production with yarn build

Once finished with developing do:

• Update the image tag in ./build_docker_image.sh and execute the shell script to build an updated image.

• Document your changes and new tag by appending the list below.

• git add, commit and push.

Further instructions about working with Grafana:

• Build a data source plugin tutorial

• Grafana documentation

• Grafana Tutorials - Grafana Tutorials are step-by-step guides that help you make the most of Grafana

• Grafana UI Library - UI components to help you build interfaces using Grafana Design System

Changelog

Tag	Changes
0.1.0	Initial version
0.1.1	Simpler setpoint selection
0.1.2	Basic authentication enabled. Skipping TLS verification enabled for self signed certificates.
0.1.3	Additional fields for custom name, scaling factor and datapoint description
0.1.4	Restructured query editor. Optional field to define an offset. Integrate grafana’s query option ‘interval’
0.2.0	Added additional plugins for querying nwpdata and stochastic prediction services
0.3.0	Update to Grafana 8.3.* due to a zero day exploit in Grafana.
0.4.0	Update to Grafana 9.0.

pgAdmin4 Tool

This is a wrapper around the pgAdmin4 Docker image. The image here adds automatic HTTPs certificate generation.

ToDo

This image does not support running as arbitrary user

Configuration

Ports

Port	Usage/Remarks
443	HTTPs port for pgAdmin user interface.

Environment Variables

The most essential settings are these:

Enironment Variable	Example Value	Usage/Remarks
SSL_CERT_PEM	—–BEGIN CERTIFICATE—– MIIFCTCCAvG…	The certificate to use for HTTPS. Will generate a self signed certificate if SSL_CERT_PEM or SSL_KEY_PEM are empty. The self signed certificate will make HTTPS work, but browsers will issue a warning.
SSL_KEY_PEM	—–BEGIN PRIVATE KEY—– MIIJQgIBADANBg…	Similar to SSL_CERT_PEM but should hold the private key of the certificate.
PGADMIN_DEFAULT_EMAIL	user@example.com	The username for logging in.
PGADMIN_DEFAULT_PASSWORD	VerySecret!	The password for logging in.
PGADMIN_ENABLE_TLS	TRUE	Should always be set as we have certificates and there seems to be no point on serving plain HTTP.

See also the official docs for additional configuration possibilities.

Volumes

Path in Container	Usage/Remarks
/var/lib/pgadmin	Folder in which pgAdmin stores its files and configuration. Mount these to host file system if settings should be persisted.

Changelog

Tag	Changes
0.1.0	Initial version. Uses pgAdmin4:5.0 docker image.