Scheduling Automated Updates
How to keep pricing, benchmarks, news, and model data fresh with automated hourly runs.
Current Automation Status
What Gets Updated
| Data | Source | Method | API key? |
|---|---|---|---|
| Pricing and model discovery | OpenRouter API | LIVE | No |
| Provider cross-checks | Official provider pages and APIs | LIVE | Optional |
| Benchmark coverage | Artificial Analysis, arena-style sources | LIVE | Recommended for fuller coverage |
| Speed and TTFT | Artificial Analysis | LIVE | AA_API_KEY recommended |
| News | Public RSS feeds, official blogs, digest fallback | LIVE | No |
Quick Start - Run Manually
# Run the full data pipeline
npm run scrape
# Check for stale or suspicious data
npm run check:staleness
# Rebuild the site with fresh data
npm run build
Windows Task Scheduler
If you want a local mirror of the hosted automation, set up an hourly task that runs the scraper pipeline automatically on your Windows machine.
Step 1: Create a batch script
Save this as update-data.bat in your project root:
@echo off
cd /d "%USERPROFILE%\ai-resource-hub"
echo [%date% %time%] Starting data update... >> logs\scrape.log
call npm run scrape >> logs\scrape.log 2>&1
call npm run check:staleness >> logs\scrape.log 2>&1
call npm run build >> logs\scrape.log 2>&1
echo [%date% %time%] Done. >> logs\scrape.log
Step 2: Create the scheduled task
Open Command Prompt as Administrator and run:
# Create an hourly task
schtasks /Create /SC HOURLY /MO 1 ^
/TN "AI Resource Hub - Update Data" ^
/TR "%USERPROFILE%\ai-resource-hub\update-data.bat"
Step 3: Verify it works
# Test the task manually
Start-ScheduledTask -TaskName "AI Resource Hub - Update Data"
# Check the log
Get-Content .\logs\scrape.log -Tail 50
macOS / Linux (cron)
Use cron for automated hourly runs on macOS or Linux.
# Open crontab editor
crontab -e
# Add these lines (runs every hour)
12 * * * * cd /home/user/ai-resource-hub && npm run scrape >> logs/scrape.log 2>&1
20 * * * * cd /home/user/ai-resource-hub && npm run build >> logs/build.log 2>&1
GitHub Actions
GitHub Actions is the primary scheduled refresh path for the public site, with a full hourly data run and manual reruns available on demand.
.github/workflows/scrape.yml API Keys
The pipeline can run with partial coverage without keys, but fuller validation needs repository secrets.
AA_API_KEY Recommended for speed, TTFT, and some benchmark coverage.
TOGETHER_API_KEY Useful for cross-checking open-source model pricing.
GOOGLE_API_KEY Useful for validating Gemini-related metadata.
GROQ_API_KEY Useful for validating Groq-hosted model metadata.
Add these as repository secrets in GitHub under Settings, then Secrets and variables, then Actions.
TOS Safety
Prefer official sources
Use official APIs and official pricing pages before scraping any third-party website.
Keep the request rate low
Hourly collection still uses a conservative source list and avoids aggressive polling.
Do not automate blocked or login-only sources
If a source is paywalled, private, or login-only, treat it as a manual research task instead.