About the Data

Data Sources

This dashboard collects data from multiple public sources. No authentication or special access is required — everything shown here is derived from publicly available information.

What	Source	How
Mailing list activity	Pony Mail	POST to /api/stats.json — message counts per month, per list
Git commits & PRs	GitHub API	Commits and pull request activity per repo, aggregated monthly
SVN commits	ASF Subversion	svn log --xml against remote repos (no checkout needed)
Project roster & PMC data	projects.apache.org	Daily-generated JSON from LDAP, Whimsy, and committee-info.txt
Releases	projects.apache.org	Release history JSON (scraped from dist archives)
Repository inventory	GitHub API	Full org listing — classifies 3000+ repos into projects
New committer dates	Whimsy LDAP	Account creation timestamps from public_ldap_people.json, cross-referenced with project group membership

How It Works

• A collector runs weekly, fetching data from the sources above.
• Data is cached aggressively — past months are immutable and never re-fetched.
• Only the current month's data is refreshed on each run.
• After collection, an activity classification pass categorizes projects by trend (Sharp Decline / Declining / Dormant) from the collected data using fixed thresholds.
• The output is static HTML + JSON — no live server or database.
• Trend lines use linear regression over a 12-month rolling window.
• The current (incomplete) month is extrapolated to a full-month estimate for trend accuracy.
• Each project has a dedicated page showing all its metrics in one place — mailing lists (linked to lists.apache.org archives), git repos, releases, community growth, and activity trends.

What's Measured

Mailing Lists

Monthly message volume per list. All lists with traffic in the past 12 months are shown. Covers dev@, user@, issues@, commits@, and other project-specific lists.

Git/VCS Activity

Per-repository commit counts and unique committers (monthly). For GitHub repos, also tracks pull requests opened, merged, and closed. SVN projects use remote svn log — no checkout required.

Community Growth

Tracks both new committers and new PMC members:

• New committers — detected via LDAP createTimestamp from Whimsy, cross-referenced with people.json group membership to determine project affiliation.
• New PMC members — from roster dates in the committees data on projects.apache.org.

The Community Growth tab on the overview page shows both in a two-color chart (blue = committers, orange = PMC members). Per-project pages list individual names and dates.

Releases

Release publications in the past 12 months, as recorded by projects.apache.org.

Activity Trends

A deterministic classification of quarter-over-quarter trends. Compares the two most recent full quarters (excluding the current partial month) for both mailing list discussion activity and git commit volume.

• 🔴 Sharp Decline — both mailing list and git activity down ≥ 30% quarter-over-quarter.
• 🟡 Declining — one metric down ≥ 30%, with meaningful prior activity (≥ 30 messages or commits in the prior quarter). If the other metric is growing ≥ 30%, the project is not flagged.
• ⚫ Dormant — fewer than 5 commits and 5 mailing list messages in the recent quarter.

Only human discussion lists (dev, user, general, discuss) are counted for mailing list trends — automated lists (notifications, commits, issues) are excluded. The thresholds are fixed constants, not heuristic or ML-based.

Limitations

• Committer detection relies on LDAP createTimestamp — this reflects when the ASF account was created, which is typically the committer grant date but may differ slightly for people who had accounts before being invited to a project.
• Release counts may include multiple artifacts per release for some projects.
• Bot activity is not yet filtered from commit/PR counts.
• Projects that are SVN-only but not yet detected may show zero git activity.
• GitHub API rate limits (5000/hour) mean large initial runs may take 1–2 hours.
• Some projects use Slack, Discord, or GitHub Discussions as primary discussion channels — mailing list declines may not reflect actual community decline for those projects.
• Quarter-over-quarter comparison can be noisy for projects with bursty release cycles or seasonal patterns. A single quarter's decline does not necessarily indicate a long-term trend.

JSON Data API

All data is published as static JSON files under data/ and can be consumed directly for your own analysis. No API key required.

Per-project files

URL pattern	Contents
data/{project}.json	Mailing list stats — active lists, message counts, monthly breakdown
data/{project}_git.json	Git activity — commits, committers, PRs per repo, monthly breakdown

For example: data/accumulo.json, data/accumulo_git.json

Foundation-wide files

URL	Contents
data/_foundation.json	Roster changes — new committers and PMC members with dates and project affiliation
data/_cache/mailing_summary.json	All projects' mailing list totals in one file (used by the overview page)
data/_cache/committees.json	Committee metadata — roster, chair, established date, reporting cycle
data/_cache/releases.json	Release history for all projects
data/_cache/activity_trends.json	Activity trend classifications — declining, at-risk, and dormant projects with trend data
data/_project_map.json	GitHub repo → project mapping (3000+ repos classified into projects)

Source Code

The collector and dashboard are open source:

• github.com/apache/comdev-metrics

Feedback and contributions welcome. See README.md for setup instructions.