Enhancing AI Auditability: From Raw Diffs to Structured Summaries

Improving the way we audit code changes is crucial for maintaining security and stability in our applications. Recently, we transitioned from feeding raw Git diffs directly to our AI analysis tools to using structured summaries. This shift significantly enhances auditability and reduces the risk of exposing sensitive information.

The Problem with Raw Diffs

Sending raw diffs to AI models presented several challenges:

• Security Risks: Direct exposure of source code could inadvertently reveal sensitive information.
• Audit Complexity: Analyzing raw diffs required complex parsing and interpretation.
• Performance Overhead: Processing large diffs consumed significant computational resources.

The Solution: Structured Summaries

To address these issues, we implemented a system that provides the AI with structured summaries instead of raw diffs. These summaries include:

• File Counts: The number of files modified in a commit.
• File Types: The types of files changed (e.g., JavaScript, Python, configuration files).
• Lines Changed: The number of lines added, modified, or deleted.

This approach allows the AI to analyze the scope and nature of changes without accessing the actual code.

Linking Audit Reports to Data Sources

To further enhance auditability, we've established a link between audit reports and the data sources they analyze. This is achieved by associating a unique identifier from the audit report with the relevant data source entry.

For example, imagine we have a data_sources table that tracks various data feeds used in our application:

CREATE TABLE data_sources (
    id INTEGER PRIMARY KEY,
    source_name VARCHAR(255),
    source_type VARCHAR(255),
    report_id INTEGER -- Foreign key referencing audit reports
);

CREATE TABLE audit_reports (
    id INTEGER PRIMARY KEY,
    report_date DATETIME,
    report_details TEXT
);

The report_id field in the data_sources table allows us to quickly retrieve the audit report associated with a specific data source. This is displayed via a modal in our admin panel.

Benefits of the New Approach

• Enhanced Security: Prevents exposure of sensitive code to AI models.
• Improved Auditability: Simplifies the process of analyzing code changes and linking them to audit reports.
• Reduced Complexity: Makes the auditing process more efficient and manageable.

Conclusion

By replacing raw diffs with structured summaries and linking audit reports to data sources, we've significantly improved the security and auditability of our application. This approach allows us to leverage the power of AI for code analysis while minimizing the risks associated with exposing sensitive information. Moving forward, we will continue to refine our auditing processes to ensure the highest levels of security and compliance.