The devlog-ist/landing project helps keep our landing pages up-to-date with the latest content. We've been working on improving its synchronization process with external data sources.

Key improvements include a new job for initial synchronization that smartly detects the last activity date, replacing a hardcoded 7-day window. This makes the sync process more efficient and accurate.

Introduction

Our application relies on the AutoSyncGeneratePostJob to synchronize data. Recently, we encountered issues with exceeding the rate limits imposed by Azure, leading to job failures and data inconsistencies. To address this, we implemented rate-limiting middleware specifically for this job.

The Problem

The AutoSyncGeneratePostJob frequently triggered requests to Azure services.