Steps to reproduce cases in your own personal environment

Step 1 # Setting up Docker

Install docker desktop. Depending upon your Operating System (OS), steps might be different. The Install Docker Engine website provides the installation steps in different OS.

You may not have to install Docker if you do not wish to run the code in a containerized environment. This might happen if you want to run separate components of model adapters or other components of the IT infrastructure separately. To be on the safe side, I would still suggest to install Docker (Docker Desktop is for Windows).

Step 2 # Setting up Orchestrator

  • General set-up

You may or may not have to install the Orchestrator (Apache Airflow, in our case) in your own OS environment. To not have to install Airflow, you must install Docker Desktop. Apache Airflow provide a ‘Docker Compose’ file which can be downloaded from their official website and can be run as a command docker compose up in a terminal.

The detailed instruction, including the Docker Compose file, for airflow installation and run in a Docker environment can be found in the Running Airflow in Docker website.

General installation instructions can be following in the Installation of Airflow website.

It is not advisable to install airflow directly on Windows OS. It is handled properly by having Linux Subsystem, such as WSL, installed in such a system.

  • Project-specific set-up

In this project, it is instructed to use Airflow in a Docker environment. See instruction steps in Installing Apache Airflow. A general suggestion is to create a virtual environment in the new directory after cloning from GitHub. This avoids conflicts between global and local python packages, along with other advantages one can find on the internet. See instructions for creating virtual environment in differnt environments.

If you get UID warning on your OS and want to avoid this, you can created an environment or .env file (if not created) and write AIRFLOW_UID=50000. Non-linux systems might give errors when running the command AIRFLOW_UID=$(id -u) docker compose up -d. If that’s the case, you can simply run docker compose up command. -d is for running in detached mode, inputting which is not neccessary. In addition, you also do not need to set up USERNAME and PASSWORD in the .env file as they have been taken care of in the docker compose file under the services airflow-init. Environment file command just overrides the set up in the Docker Compose file, as mentioned in the instructions.

Step 3 # Minio installation and connection

  • General set-up

Installing Minio provides instructions for installing and deploying Minio in your personal environment. This webpage takes you to the Windows OS, but has tabs for other environments such as Kubernetes, Docker, Linux, and macOS.

This generally accesses the port 9000. You can use http://localhost:9000 or http://127.0.0.1:9000 on your web-browser.

  • Project-specific set-up

For this docker need to be installed and running on your system. See instruction steps in Installing Minio. You can adjust the volumes within Minio services of the docker compose file to redirect to a specific file location/directory for your Minio activity, such as creating buckets, editing, and removing them. A folder can be created within the directory Database-MinIO, and location can be included in the volumes.

Step 4 # Model registry installation

This keeps track of all model adapter services that are active on your environment. Follow the instruction steps in Installing Model Registry.

Step 5 # Installation of model adapters

Either you can install all model adapters or case-specific model adapters from GitHub (see Installing Adapters).

Step 6 # Running a case

If step 2 is performed correctly, you could see DAGs on your locahost screen. Creating a use-case acquints you with tasks, operators, etc. used in this project for different cases.

We focused on a sample case of etm_essim_demo. For this, the following model adapters need to be running on your Docker environment:

  1. ESSIM

  2. ETM-Price-Profile

  3. ESDL-Add-Price-Profile

Apart from these, the following repository components need to also run on Docker:

  1. Database-MinIO

  2. Model-Registry

  3. Model-Orchestrator

Running a use-case provides basic information to familiarize with the Orchestrator, Apache-Airflow, interface. For the etm_essim_demo DAG, the Trigger DAG w/ config shows a blank configuration JSON object. This needs to filled with etm_essim_demo.json data (copy-paste the entire content). The current JSON file (present in the model repository) is giving errors. This was rectified and here.

etm_essim_demo.json
 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
{
	"metadata": {
		"user": "mmvib",
		"project": "bedrijventerreinommoord",
		"scenario": "Scenario_1_II3050_Nationale_Sturing",
		"experiment": "Trial_1",
		"run": "MM_workflow_run_1"
	},
	"modules": {
		"model_registry": "http://mmvib-registry:9200/registry/"
	},
	"databases": {
		"Minio": {
			"api_addr": "minio:9000",
			"db_config": {
				"secure": false,
				"access_key": "admin",
				"secret_key": "password"
			}
		},
		"Influx": {
			"api_addr": "influxdb:8086",
			"db_config": {
				"db_name": "energy_profiles",
				"use_ssl": false
			}
		}
	},
	"tasks": {
		"Pull_Data_Minio": {
			"type": "transaction",
			"api_id": "Minio"
		},
		"Load_Data_Influx": {
			"type":"transaction",
			"api_id": "Influx"
		},
		"Generate_ETM_Price_Profile": {
			"type": "computation",
			"api_id": "ETM",
			"model_config": {
				"etm_config": {
					"endpoint": "https://engine.energytransitionmodel.com/api/v3/",
					"path": "scenarios/{}/curves/electricity_price.csv",
					"scenario_ID": "763305"
				},
				"output_file_path": "ETM_price_profile_adapter/elektrictitetisprijs_profiel.csv"
			}
		},
		"Combine_ESDL_ETM": {
			"type": "computation",
			"api_id": "ESDL_ETM",
			"model_config": {
				"input_esdl_file_path": "ESDL_add_price_profile_adapter/Hybrid HeatPump.esdl",
				"input_csv_file_path": "ETM_price_profile_adapter/elektrictitetisprijs_profiel.csv",
				"output_file_path": "ESDL_add_price_profile_adapter/HHP_profile.esdl"
			}
		},
		"Run_ESSIM_Simulation": {
			"type": "computation",
			"api_id": "ESSIM",
			"model_config": {
				"essim_post_body": {
					"user": "essim",
					"scenarioID": "essim_mmvib_adapter_test",
					"simulationDescription": "ESSIM MMvIB adapter test",
					"startDate": "2019-01-01T00:00:00+0100",
					"endDate": "2019-01-01T23:00:00+0100",
					"influxURL": "http://influxdb:8086",
					"grafanaURL": "http://grafana:3000",
					"natsURL": "nats://nats:4222",
					"kpiModule": {
						"modules": [{
							"id": "TotalEnergyProductionID",
							"config": {
								"scope": "Total"
							}
						}]
					}
				},
				"input_esdl_file_path": "bedrijventerreinommoord/Scenario_1_II3050_Nationale_Sturing/Trial_1/MM_workflow_run_1/ESDL_add_price_profile_adapter/HHP_profile.esdl",
				"output_esdl_file_path": "bedrijventerreinommoord/Scenario_1_II3050_Nationale_Sturing/Trial_1/MM_workflow_run_1/ESSIM_adapter/HHP_KPIs.esdl",
				"output_file_path": "bedrijventerreinommoord/Scenario_1_II3050_Nationale_Sturing/Trial_1/MM_workflow_run_1/ESSIM_adapter/KPIs.json"
			}
		}

	}
}

(Optional) Alternative Step # Manual running or checking of adapters (without using Docker)

Once you have cloned all the above-mentioned adapters and other repository components mentioned in Step 6, you can run the main:app of each of them. This python file is found within the folder tno\<application_name>. For example, for the ESSIM adapter, the application name is essim_adapter. To run the file from the base directory, you can use the command python tno\<application_name>\main.py. N.B.: \ or / depends on your OS.

To avoid encountering any issues, first start the model registry. Then, model adapters can be started. Each adapter will register itself to the registry. You can check this on the webbrowser with http://localhost:9900/registry. 9900 port is exposed for manual run (see the .env file of each adapter).

For the manual set up, a local minio instance need to run on your OS or local environment (see ‘General set-up’ in Step 3) from a separate or dedicated terminal.