Leaderboard migration by SamXop123 · Pull Request #1907 · recodehive/recode-website

SamXop123 · 2026-06-17T16:56:45Z

Whats implemented:

Backend Cron Scraper: The Express backend runs a background scheduled cron job (every 2 hours) that scrapes public, unarchived organization repositories, filters for merged PRs matching the "recode" label, aggregates repository statistics, and counts discussions.
Point & Streak Aggregation: Contributors are scored according to Recode rules (Level 1 = 10, Level 2 = 30, Level 3 = 50 points). Streaks are calculated based on active daily consecutive PR merge dates.
Atomic Cache Storage: Scraper data is written atomically to leaderboard.json and stats.json inside the cache directory (compatible with persistent volume mounts).
Backend API: Serves the cached datasets instantly via /api/leaderboard and /api/stats.
Frontend Context Hook: The frontend React context (CommunityStatsProvider) fetches from these two backend API routes in parallel on page load. Client-side sorting and date filtering (week/month/year/all-time) are preserved in memory for instant, responsive UI transitions.

Files Changed:

Backend (leaderboard repository)

server.js: Setup API routes, CORS configs, and initialized startup checks.
utils/cacheHelper.js [NEW]: Implements atomic file writes (writeJsonAtomic) to prevent corruption, file exists checks, and fallback JSON models.
functions/generateLeaderboard.js: Replaced legacy scraper. Implements GraphQL repository traversal, label scoring weight filters, active streak checks, and discussions count queries.
functions/generateCALeaderboard.js: Integrates atomic writes and generation locks.
jobs/updateOSLeaderboard.js: Schedules OS leaderboard cron jobs.
jobs/updateCALeaderboard.js: Standardizes cron patterns.

Frontend (this repository)

docusaurus.config.ts: Declares backendApiUrl customField and removes deprecated gitToken client exposing.
src/lib/statsProvider.tsx: Removes client-side scraping pipelines. Fetches and parses backend API endpoints and binds to context states.
src/services/githubService.ts: Removes dead code path helper methods.

API Contract

A. `GET /api/leaderboard`

Returns the contributors array and legacy compatibility aliases.

{
  "success": true,
  "updatedAt": 1781548394726,
  "lastUpdated": 1781548394726,
  "generated": true,
  "contributors": [
    {
      "username": "contributor1",
      "avatar": "https://github.com/contributor1.png",
      "profile": "https://github.com/contributor1",
      "points": 60,
      "prs": 2,
      "prDetails": [
        {
          "title": "PR 2 - Docs",
          "url": "https://github.com/recodehive/repo1/pull/2",
          "mergedAt": "2026-06-15T12:00:00Z",
          "repoName": "repo1",
          "number": 2,
          "points": 50
        }
      ],
      "streak": 2
    }
  ]
}

B. `GET /api/stats`

{
  "success": true,
  "updatedAt": 1781548394731,
  "lastUpdated": 1781548394731,
  "totalStars": 984,
  "totalForks": 1107,
  "totalContributors": 467,
  "totalRepositories": 10,
  "publicRepositories": 9,
  "discussionsCount": 42
}

Migration Strategy

Backend Deploy (Railway): Deploy backend and map CACHE_DIR to a persistent volume (e.g. /data) to preserve cached listings across restarts. Set PORT (default 5000) and the private GIT_TOKEN.
Frontend Config: Expose BACKEND_API_URL pointing to the deployed backend server domain.
Authentication: Clerk login gate remains unchanged. Discussions panel continues fetching via restricted token until a discussions backend proxy endpoint is introduced in a future PR.

Testing Performed

Backend Unit Tests: Verified repository discovery, unarchived filters, point weights, and active streak resets using mock datasets.
E2E Headless Browser Testing: Used Puppeteer to verify:
- Podium: Rendered 1st, 2nd, and 3rd rank performer cards correctly.
- Table: Populated ranking rows, usernames, avatars, and badges.
- Stats Cards: Displayed correct community values.
- PR Modal: Clicking a row count badge successfully launched the PR List Modal with correct titles, URLs, and points.

Fixes #1362

Type of Change

New feature (e.g., new page, component, or functionality)
Bug fix (non-breaking change that fixes an issue)
UI/UX improvement (design, layout, or styling updates)
Performance optimization (e.g., code splitting, caching)
Documentation update (README, contribution guidelines, etc.)
Other (please specify): backend migration

Checklist

My code follows the style guidelines of this project.
I have tested my changes across major browsers and devices
My changes do not generate new console warnings or errors .
I ran npm run build and attached screenshot(s) in this PR.
This is already assigned Issue to me, not an unassigned issue.

vercel · 2026-06-17T16:56:58Z

@SamXop123 is attempting to deploy a commit to the recode Team on Vercel.

A member of the Team first needs to authorize it.

github-actions · 2026-06-17T16:57:05Z

Thank you for submitting your pull request! 🙌 We'll review it as soon as possible. The estimated time for response is 5–8 hrs.

In the meantime, please provide all necessary screenshots and make sure you run - npm build run , command and provide a screenshot, a video recording, or an image of the update you made below, which helps speed up the review and assignment. If you have questions, reach out to LinkedIn. Your contributions are highly appreciated!😊

Note: I maintain the repo issue every day twice at 8:00 AM IST and 9:00 PM IST. If your PR goes stale for more than one day, you can tag and comment on this same issue by tagging @sanjay-kv.

We are here to help you on this journey of open source. Consistent 20 contributions are eligible for sponsorship 💰

🎁 check our list of amazing people we sponsored so far: GitHub Sponsorship. ✨

📚Your perks for contribution to this community 👇🏻

Get free Consultation use code recode50 to get free: Mentorship for free.
Get the Ebook for free use code recode at checkout: Data Science cheatsheet for Beginners.
Check out this weekly Newsletter: Sanjay's Newsletter.

If there are any specific instructions or feedback regarding your PR, we'll provide them here. Thanks again for your contribution! 😊

github-actions · 2026-06-17T16:57:12Z

✅ Synchronized metadata from Issue #1362:

Labels: enhancement, area:backend-sql, level 2, recode
Milestone: recode:launch 3.0

SamXop123 · 2026-06-17T16:58:01Z

The `leaderboard` repo pr: http://github.com/recodehive/leaderboard/pull/1

SamXop123 · 2026-06-17T17:01:31Z

@iitzIrFan this was a multiple repo issue as suggested by @Adez017 so i had to create two pr for backend migration

iitzIrFan · 2026-06-17T17:04:24Z

Okay makes scenes now, @Adez017 will review soon!

Adez017 · 2026-06-17T17:12:47Z

Okay makes scenes now, @Adez017 will review soon!

will review it in the morning @iitzIrFan , need to take a look closer onwards

SamXop123 · 2026-06-17T17:14:58Z

it works perfectly fine on local testing. just needs a few production tests

a little help for reviewers for production tsting:

deploy the backend branch to railway.
add git token and cache directory (CACHE_DIR=/data)
mount a persistent volm at /data
and start the service
then we can verify the /api/leaderboard & /api/stats endpoints
on frontend we set BACKEND_API_URL=<railway-backend-url>
deploy that branch as preview
then we can visit the leaderboard page to actually verify.

Copilot

Pull request overview

This PR migrates the website’s leaderboard/community stats data flow away from client-side GitHub scraping toward a dedicated backend API, aligning with the goal of reducing GitHub rate-limit pressure and improving page-load responsiveness.

Changes:

Added a backendApiUrl Docusaurus customFields value for configuring the backend base URL.
Updated CommunityStatsProvider to fetch /api/leaderboard and /api/stats from the backend in parallel instead of scraping GitHub on the client.
Removed now-obsolete organization stats scraping methods from GitHubService.

Reviewed changes

Copilot reviewed 3 out of 3 changed files in this pull request and generated 4 comments.

File	Description
docusaurus.config.ts	Adds `backendApiUrl` to browser-exposed config; (still) exposes `gitToken` via `customFields`.
src/lib/statsProvider.tsx	Replaces client-side scraping pipeline with backend fetch + mapping into existing context state.
src/services/githubService.ts	Removes org stats scraping/aggregation methods that are no longer used by the frontend.

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

  const token = customFields?.gitToken || "";
+  const backendApiUrl = (customFields?.backendApiUrl as string) || "http://localhost:5000";


sanjay-kv · 2026-06-18T02:13:44Z

          data: {
            contributors: contributorsArray,
-            rawStats: { totalPRs: totalMergedPRs },
+            rawStats: { totalPRs: statsData.totalContributors },
          },


@Adez017 I need your review on merged PR
https://github.com/recodehive/leaderboard/pull/1/changes

This is the backend code.

Adez017 · 2026-06-18T06:48:29Z

+    backendApiUrl: process.env.BACKEND_API_URL || "http://localhost:5000",
    gitToken: process.env.DOCUSAURUS_GIT_TOKEN,
    clerkPublishableKey: process.env.VITE_CLERK_PUBLISHABLE_KEY || "",


I think now we don't need to mount the git token , let me know if it is required or we can move forward and remove this one @SamXop123

vercel · 2026-06-18T01:36:10Z

The latest updates on your projects. Learn more about Vercel for GitHub.

Project	Deployment	Actions	Updated (UTC)
recode-website	Ready	Preview, Comment	Jun 18, 2026 2:16am

Co-authored-by: Copilot Autofix powered by AI <175728472+Copilot@users.noreply.github.com>

github-actions · 2026-06-18T02:14:23Z

✅ Synchronized metadata from Issue #1362:

Labels: enhancement, area:backend-sql, level 2, recode
Milestone: recode:launch 3.0

sanjay-kv · 2026-06-18T15:53:48Z

SamXop123 added 3 commits June 17, 2026 16:39

Update docusaurus.config.ts

1e88777

Update statsProvider.tsx

dc2b118

Update githubService.ts

bff6d8f

SamXop123 requested review from Adez017, iitzIrFan and sanjay-kv as code owners June 17, 2026 16:56

github-actions Bot added in-review The current changes are in review and would need approval and testing before merging level 1 10 points recode this is label for leaderboard labels Jun 17, 2026

github-actions Bot added this to the recode:launch 3.0 milestone Jun 17, 2026

github-actions Bot added area:backend-sql for issues related to the experimintal support of MsSql as backend enhancement New feature or request level 2 30 points labels Jun 17, 2026

iitzIrFan assigned SamXop123 Jun 17, 2026

iitzIrFan added this to @recode-web Jun 17, 2026

iitzIrFan moved this to In Progress in @recode-web Jun 17, 2026

SamXop123 mentioned this pull request Jun 17, 2026

Migrate Leaderboard & Stats Scraping to Backend API Cache recodehive/leaderboard#1

Merged

sanjay-kv requested a review from Copilot June 18, 2026 01:23

Copilot started reviewing on behalf of sanjay-kv June 18, 2026 01:23 View session

Copilot AI reviewed Jun 18, 2026

View reviewed changes

sanjay-kv approved these changes Jun 18, 2026

View reviewed changes

vercel Bot deployed to Preview June 18, 2026 01:37 View deployment

Potential fix for pull request finding

a2f5b2d

Co-authored-by: Copilot Autofix powered by AI <175728472+Copilot@users.noreply.github.com>

vercel Bot deployed to Preview June 18, 2026 02:16 View deployment

		const token = customFields?.gitToken \|\| "";
		const backendApiUrl = (customFields?.backendApiUrl as string) \|\| "http://localhost:5000";

Conversation

SamXop123 commented Jun 17, 2026

Whats implemented:

Files Changed:

Backend (leaderboard repository)

Frontend (this repository)

API Contract

A. GET /api/leaderboard

B. GET /api/stats

Migration Strategy

Testing Performed

Type of Change

Checklist

Uh oh!

vercel Bot commented Jun 17, 2026

Uh oh!

github-actions Bot commented Jun 17, 2026

Uh oh!

github-actions Bot commented Jun 17, 2026

Uh oh!

SamXop123 commented Jun 17, 2026

The leaderboard repo pr: http://github.com/recodehive/leaderboard/pull/1

Uh oh!

SamXop123 commented Jun 17, 2026

Uh oh!

iitzIrFan commented Jun 17, 2026

Uh oh!

Adez017 commented Jun 17, 2026

Uh oh!

SamXop123 commented Jun 17, 2026

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Reviewed changes

Uh oh!

sanjay-kv Jun 18, 2026

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Adez017 Jun 18, 2026

Choose a reason for hiding this comment

Uh oh!

vercel Bot commented Jun 18, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

github-actions Bot commented Jun 18, 2026

Uh oh!

sanjay-kv commented Jun 18, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

5 participants

A. `GET /api/leaderboard`

B. `GET /api/stats`

The `leaderboard` repo pr: http://github.com/recodehive/leaderboard/pull/1

vercel Bot commented Jun 18, 2026 •

edited

Loading