Handling Smart Meter Data

How Unicage Can Be An Asset In Processing Data from Smart Grids

Smart grids are a type of electrical power grid that allows two-way flows of electricity and information. To have this flow of information, the grid uses Smart Meters, which are devices that record and communicate information regarding the consumption of energy or other resources. Having smart meters in place allows the grid to automatically respond to conditions and events within it, which can lead to optimized energy distribution.

However, for this to be possible, it is crucial to gather and disseminate data with strict time requirements. This creates the need for a system that can process and manage large amounts of data efficiently. The amount of data generated by smart meters depends on the frequency with which readings from the smart meters are collected.

Goal

For this use case, our goal will be process, validate and create a file that contains the sum of the readings from the smart meters.  We shall order our process by first parsing the smart meter’s files. Then, we will make a simple validation of the readings. Finally, we shall sum the values from all readings that passed our validation.

Starting Point

Let us take as an example a smart grid composed by 100.000 smart meters that produces daily readings. Each smart meter produces an XML file containing three different types of readings. An example is shown below:

$ cat SM000000001VG_20210101T002650.xml
<MeterReadings>
<MeterReading>
<Meter>
<Names>
<name>SM000000001VG</name>
<NameType>
<description>This is a meter identification number.</description>
<name>MeterID</name>
</NameType>
</Names>
</Meter>
<Readings>
<timeStamp>2021-01-01T00:26:50Z</timeStamp>
<value>12.6316</value>
<ReadingType ref="0.0.0.4.1.1.12.0.0.0.0.0.0.0.0.3.72.0"/>
</Readings>
<Readings>
<timeStamp>2021-01-01T00:26:50Z</timeStamp>
<value>11.8440</value>
<ReadingType ref="0.0.0.12.1.1.37.0.0.0.0.0.0.0.0.3.38.0"/>
</Readings>
<Readings>
<timeStamp>2021-01-01T00:26:50Z</timeStamp>
<value>6.5043</value>
<ReadingType ref="0.0.0.0.0.0.46.0.0.0.0.0.0.0.0.0.23.0"/>
</Readings>
</MeterReading>
</MeterReadings>

Parsing the Files

 To achieve this, since the files are in xml format, we start by using Unicage’s xmldir command to parse the 100.000 files and then merge this data into a single file. For this, we use a combination of Unicage’s self and delr commands together with awk to prepare the data so that it can be transposed using Unicage’s map command:

find -name "*.xml" -exec cat {} +                                         |
xmldir -c3 /MeterReadings/MeterReading -                                  |
self NF-1 NF                                                              |
awk '$1 == "name" || $1 == "timeStamp" || $1 == "value" || $1 == "ref"'   |
delr 2 "MeterID"                                                          |
awk '{if($1=="name"){meter=$0;print $0}
          else{
             if($1 == "timeStamp"){print meter; print $0}
             else{print $0}
          }
      }'                                                                  |
awk '{if($1 == "name"){count++}; print count, $0}'                        |
map num=1                                                                 |
delf 1                                                                    |
delr 3 "0" > READINGS_PARSED

The output will be a single file named “READINGS_PARSED” which will have the following structure, with the smart meter id in the 1st column, the reading reference in the 2nd, the timestamp in the 3rd and the reading value in the 4th column:

SM000000689VG 0.0.0.4.1.1.12.0.0.0.0.0.0.0.0.3.72.0 2021-01-01T12:40:06Z 14.8361
SM000000689VG 0.0.0.12.1.1.37.0.0.0.0.0.0.0.0.3.38.0 2021-01-01T12:40:06Z 7.4433
SM000000689VG 0.0.0.0.0.0.46.0.0.0.0.0.0.0.0.0.23.0 2021-01-01T12:40:06Z 6.5668
SM000000145VG 0.0.0.4.1.1.12.0.0.0.0.0.0.0.0.3.72.0 2021-01-01T08:54:15Z 19.7668
SM000000145VG 0.0.0.12.1.1.37.0.0.0.0.0.0.0.0.3.38.0 2021-01-01T08:54:15Z 10.1405
SM000000145VG 0.0.0.0.0.0.46.0.0.0.0.0.0.0.0.0.23.0 2021-01-01T08:54:15Z 6.9721
SM000000453VG 0.0.0.4.1.1.12.0.0.0.0.0.0.0.0.3.72.0 2021-01-01T06:50:54Z 9.9979
SM000000453VG 0.0.0.12.1.1.37.0.0.0.0.0.0.0.0.3.38.0 2021-01-01T06:50:54Z 19.0457
SM000000453VG 0.0.0.0.0.0.46.0.0.0.0.0.0.0.0.0.23.0 2021-01-01T06:50:54Z 14.0774
(…)

Validation

The next step will focus on the validation of the readings. For this, we have a Master file called “READING_TYPE_CONVERTER” with the reading types that are considered valid. In addition, we have a “TYPE0X” that is associated with each type of reading. This “TYPE0X” can represent, for example, Electricity, Gas or Water, depending on what the smart meter is measuring:

$ cat READING_TYPE_CONVERTER
0.0.0.0.0.0.46.0.0.0.0.0.0.0.0.0.23.0 TYPE01
0.0.0.12.1.1.37.0.0.0.0.0.0.0.0.3.38.0 TYPE02
0.0.0.4.1.1.12.0.0.0.0.0.0.0.0.3.72.0 TYPE03

With this Master file, we can then use Unicage’s cjoin1 command to identify which readings are valid, whilst adding the corresponding type to the reading. To remove the readings which do not appear in the Master file, we use the +ng option and redirect its output to a file which will contain all the invalid readings that are present in the merged smart meter file:

cat  READINGS_PARSED  |
cjoin1 +ng3 key=2 READING_TYPE_CONVERTER - > ALL_VALID_READINGS 3> ALL_INVALID_READINGS

Sum the Readings

With the readings split between valid and invalid, we can advance to the final step, which is making the cumulative sum of the valid readings by their type. This is easily achieved by selecting the desired fields using Unicage’s self, sort them using msort and, finally, sum all the values by using the sm2 command:

self 3 5 ALL_VALID_READINGS         |
# 1:reading_type_name 2:value
msort key=1                         |
# 1:reading_type_name 2:value
sm2 1 1 2 2  > MEASURED_VALUES_by_TYPE

The final result will be a file containing the three types of valid readings and the respective cumulative value of all readings of that type that were measured by the smart meters:

$ cat MEASURED_VALUES_by_TYPE
TYPE01 10143.4150
TYPE02 9915.1172
TYPE03 10087.8375

Final Thoughts

Using this small example, we were able to show how we can use Unicage to handle data from Smart Meters on a daily basis using simple scripts and commands that do not require any additional software. With the generated output files, one can easily import them to any other processing or vizualization tool without the need to upload and process the raw XML files, which can consume considerable computational resources and time in other programming languages or softwares.

If this topic has interested you and you want to know more about our company, methodology or even request a demo that reflects your needs, please feel free to contact us by filling the form bellow.

Find out more

Request a demo and speak with our team about how you can leverage the power of Unicage in your organization.

Privacy Policy