Frequently Asked Questions (FAQ)

Q1. Which data formats are supported?

- The platform supports both JSON and XML formats. Note: Support of XML formats is still experimental.

Q2. How do you create the sensor (data) schema?

- It is crucial that you know the exact format of the data document (i.e. the data "output") prior to creating the data schema. When you have this document in hand, you can either use the Windows desktop application configuration tool, or you can do this manually by using the appropriate REST endpoint directly. The latter case requires some basic knowledge of the Avro specification to map the data attributes correctly. You can also look at the configuration examples to get a better understanding of how this is done.

Q3. What are the key steps for pushing data to the data platform?

#1. A data provider must be registered to obtain valid access token(s). These tokens must be used when interacting with REST endpoints that requires authentication.

#2. A description of the expected format (partial Avro schema) of the data to send must be defined (please refer to Q2).

#3. Use the sensor endpoint URL for data transfer (push).

#4. Implement push logic, either directly in sensor software, or indirectly by using a data transfer mechanism (if sensors are pushing internally to a cloud system) to transfer data.

NOTE: when schema is successfully configured, the endpoint URL for sensor can be found by using the atom URL for sensor given in the HTTP created response.

Q4. Are nested data structures also supported?

- Yes.

Q5. Is it always required to use converters?

- Not always. Converters must be used if the value of the JSON/XML attribute (measurement value or timestamp) is not numeric. If this value already is numeric, then no converter is required.

Q6. Is it always required to use timestamp binding?

- Not always. Mapping of the timestamp attribute must be used if you want to use your own timestamp and not the default system Unix Epoch time. If your datetime attribute is already given as number of seconds since 1970-01-01 UTC (long datatype), then no "StringEpochTime" converter is required.

Q7. What happens in cases when no timestamp binding is possible?

- The system time (Unix Epoch UTC) will be used.

Q8. What if one of measurement values is not present?

- In these cases, the NaN value must represent the missing value. This value will not be stored in backend system.

Q9. What if the sensor schema changes?

- In cases when one (or more) attribute that represents a sensor component needs to be removed, the sensor schema must be changed (or reconfigured). Please read more here.

Q10. What is the usage of the binding path?

- The binding path is the JSON path (JPath) starting from root of the document to a target JSON attribute node. This binding path needs to be used to tell the system which JSON attribute that should be bound to a component and unit.

- By using this path, it will be possible to extract the attribute (i.e. component) value.

Q11. What is the usage of the target path?

- The target path is (like the binding path) the JSON path (JPath) starting from root of the document to a target JSON attribute node. The target path is used by a converter or field mapper when converting or mapping a JSON attribute value.

Q12. What is an Avro schema?

- Apache Avro is used to define the data schema of a sensor record. A record in this context is the sensor data ("packet"). This schema describes the fields allowed in the record, along with their expected data types.

- Avro schema combined with the sensor data defines the Avro record. This record is transmitted to backend system for processing and storage.

- The usage of Avro schemas allows serialized values to be stored in a very space-efficient binary format (Avro encoded format).

- More information about the Avro API can be found here (external page).

Q13. Is "raw" data also stored?

- Yes, the complete data packet is stored as Avro record in backend system. REST endpoint(s) for retrieving these data has not yet been implemented.

Q14. How is quality control (QA/QC) managed?

- The quality control on sensor data is managed by the Spark applications in backend system. Statistical operations combined with trained machine learning (ML) models are involved in this process.

Note: Not all components are configured for quality control. The exceptions are the humidity, temperature, CO, NO₂, PM_2.5 and PM₁₀ components. More components may be added for QA/QC in future releases. ML models for humidity and temperature have not yet been established.

Q15. What is the usage of timezones?

- The platform uses UTC time for all timestamps. This is to ensure that all data is stored in a consistent manner.

- When local time for measurements is not within the default timezone (and is not already in UTC), which is Europe/Oslo, then timezone must be specified during schema configuration to ensure local time is converted to correct UTC time.