Accessing Just the Required Data From Logs or Archives

Goal

SCADA vendor needs to provide its customers with ad hoc access to selected archived data from standard desktop tools without requiring customers to extract and load the data.

Examples

Applications in which large volumes of data are collected and archived:

  • Real-time controls – Various sensor and actuator measurements are collected over time and archived for reporting and analysis.
  • Financial markets – Performance of various instruments like stock prices and mutual fund prices are stored over time. Also, data collected during test flights could result in many gigabytes of data for each flight.

Issues and Alternatives

The data is typically stored in a compressed proprietary format to allow large amounts of data to be quickly captured and stored. One option to provide access to the data is to extract and load it into formats such as a relational data store so that reporting and analysis tools like Excel and Crystal Reports can access it. However, this is not very efficient or practical, as it requires manual transfer of selected data and massaging it to allow consumption by each user.

Resolution

Provide real-time access to the subset of data requested through a custom DataDirect OpenAccess SDK.

Implementation

OpenAccess allows the data to stay in its native form and be presented as relational tables accessible using SQL. Desktop tools and users then formulate a SQL query to request the sub-set of data they are interested in. This request is processed by OpenAccess using an adapter that is written to plug the data source into the OpenAccess SQL engine. The adapter (or Interface Provider) works closely with the SQL engine to efficiently access just the required data.

Benefits

Quickly provide access to archived and real-time data as if it was stored in a SQL database like SQL Server or Oracle.