Making data into a force for illumination depended on database drivers in the legacy system. The process required a “pull” of data from student and financial management systems (a transactional Informix database) into a Microsoft SQL-based data warehouse. The processes were developed using Microsoft.NET technologies. Though the system had been capable of handling data pulls from the districts nine times per year, the addition of nightly and on-demand data pulls required significantly more processing capability and essentially swamped the system. The updated system worked well with pilot school districts, but the built-in database drivers failed to perform once data was pulled from all districts. Furthermore, the built-in drivers were failing with read errors when executed against the full data set.
Organizational and personnel issues also factored into data management at the ADE. The ADE is not an all-powerful entity. It coordinates activities and programs among the local districts. From an IT perspective, each district is independent, with control over its own system choices. Whatever solution the ADE devised, it had to be contained completely within its own systems. They could not expect the districts to purchase or install any software to make the data pulls work better.
One choice that the IT staff at the ADE faced was whether or not to attempt to correct the data-loading problem by modifying application code along with database scripts and tables. After reviewing the situation and researching possible solutions, the team decided to replace the drivers instead. Understanding that drivers could be part of the solution was something of a revelation for the ADE. “We have not traditionally thought of there being options when it comes to database drivers,” said a member of the programming team. “However, implementation of Progress DataDirect’s ODBC drivers has shown this is an area worth additional research and investigation,particularly when the information system has high data processing requirements. Based on the information system, it’s very possible that substitution of the database driver may eliminate the need for code improvements when re-coding is thought to be the only method to improve processing times.”
DataDirect Platinum ODBC drivers were tested and recommended by the ADE’s information system implementers/developers. They selected DataDirect partly because the product and company had a documented track record of strong performance and the ability to improve the processing time of large data loads. The drivers allow application developers to write, compile,and ship applications without targeting specific database management systems or using embedded SQL. This eliminates the need to recompile the application for each new environment. The ADE could modify the drivers at their end and ensure success without having to make the districts do anything differently. The ADE team was also attracted to Platinum ODBC because it offered a host of new capabilities that suited the ADE mission, including:
DataDirect’s drivers resolved the load errors and delays. The drivers enable successful extract, transform, and load (ETL) processing for the nightly database pulls from the districts. And the whole process worked 20% faster than it had in comparison to the earlier, less frequent cycle of loading. The gain in performance helped the ADE avoid an investment in additional processing capacity. A reduction in errors during ETL also translated into more efficient management of the entire system. People did not have to spend time and resources redoing data pulls and troubleshooting load errors. The ADE also received recognition by the Data Quality Campaign for meeting nine of the critical action items and ten critical elements in its educational information systems.
The decision to increase the data pulls and speed up data availability to the districts is part of a broader drive for greater data transparency and data-driven decision making. The data “flashlight” has been lit, but the plan is to make it shine even brighter. The ADE wants to make more detailed educational data to stakeholders and a growing number of external entities that partner in education. The ADE system manages reporting to the Federal government, for example.
As new programs become available to the ADE, the agency wants to have the ability to produce data that can readily qualify districts for funding and then provide accountability once the programs are in place.
For more information please visit: