Improving Data Accuracy with Enhanced API Integration

Introduction

This post details an enhancement to our application's data aggregation process. We addressed an issue where total counts were inaccurate by switching from a local database to an external API for data retrieval, ensuring more reliable results.

The Problem: Data Discrepancies

Previously, our application relied on a local database table, populated with data from an events API, to determine the total number of items (e.g., pull requests). However, this approach had limitations, primarily due to the events API's data retention policy (maximum 90 days). Consequently, the count was often significantly lower than the actual total. This discrepancy impacted the accuracy of reports and metrics within the application.

The Solution: Leveraging an External API

To address this, we implemented a solution that utilizes an external search API to obtain a more accurate total count. This API provides a comprehensive view of the data, unconstrained by the limitations of the local database. The system now attempts to retrieve the total count from the external API, and only falls back to the local database in case of failure or unavailability of the external API.

def get_total_count(api_client, local_db, query):
    try:
        total_count = api_client.search(query)
        return total_count
    except Exception as e:
        print(f"Error using API: {e}")
        total_count = local_db.count(query)
        return total_count

Handling Missing Data

During the implementation, we also addressed scenarios where data for certain entities might be missing. To ensure data integrity and prevent errors, we implemented an "upsert" operation. This means that when new data is encountered, the system first checks if an existing record exists. If it does, the record is updated; if not, a new record is created. This ensures that all entities have a corresponding entry in the data store.

def update_or_create_entity(data_store, entity_id, data):
    data_store.update_or_create(entity_id=entity_id, defaults=data)

Results

By switching to the external API and implementing the upsert operation, we significantly improved the accuracy and reliability of the data within the application. The total counts are now more representative of the actual figures, leading to more informed decision-making.

Next Steps

Future improvements could include implementing caching mechanisms to reduce the load on the external API and optimizing the query structure to further improve performance. Monitoring the API's availability and response times is also crucial for maintaining data accuracy.