Method: Rubric 1.0
Every tool we test goes through the same six gates in an isolated test VM. Each gate gets a whole-number score from 0 to 10, computed from the rules on this page and nothing else. Every score is shown with "Rubric 1.0", the tool version and the test date.
The lab
Every test runs in an isolated test VM, never on a desktop. Each category has a test card that fixes the reference VM, the steady load and the use cases before a tool is tested. For push notification servers that is Debian 13 with 4 vCPU and 8 GiB of RAM, Docker with Compose. Each episode uses its own Docker network, every container gets CPU and memory limits, and the VM is rolled back to a clean snapshot after each episode. Raw logs and measurements are kept.
Before any score: every component the tests use needs an OSI-approved licence. Without one, the tool gets the status Not rated instead of a verdict.
The six gates
1. Install
We use the project's official container install: its Compose file, else its docker run line as Compose,
else the first documented method. "Official" means published under the project's own registry namespace or named as
the supported image in its install docs. It runs behind a Caddy reverse proxy with HTTPS, set up the way the project's
proxy docs say.
The score starts at 10. Steps the page gives for the method we test (filling in placeholders, running its commands) are not changes. A step the page shows only for another method counts as a change, and we say where the page shows it. Every change the page does not tell you about costs points:
- -1 cosmetic: the error message names the fix.
- -2 functional: it fails to start, needs root or an ownership change, or the proxy breaks a feature.
- -4 silent data loss: it looks fine, but data is gone after a restart.
The score never goes below 0.
2. Resources
Idle use, a steady load from the category test card, and a Break Point test that raises the load until the tool cannot keep up. The lowest matching row wins:
| Score | Rule |
|---|---|
| 10 | Within the documented requirements, no errors at the steady load, Break Point at 10 times the steady load or more |
| 8 | Within the documented requirements, Break Point at 3 to 10 times the steady load |
| 6 | Over the documented requirement by less than 50 %, or Break Point under 3 times |
| 4 | Over the documented requirement by more than that |
| 2 | Errors at the steady load |
| 0 | It does not run |
When the project documents no requirement, the category's reference VM counts as the requirement.
3. Real use
Three use cases from the category test card, chosen before we read any pricing page. When the app has users, one of them involves a second user: a second non-admin user must not see the first user's data. A use case that needs a paid tier fails.
| Score | Rule |
|---|---|
| 10 | All three pass every check |
| 8 | All pass, one needs a workaround (each further workaround: -2) |
| 6 | One fails |
| 4 | Two fail |
| 2 | The core use case fails |
| 0 | Nothing works |
4. Resilience
Five checks, 2 points each:
- Kill mid-write: the container is killed in the middle of a stream of writes.
- Crash with no sync: the whole VM crashes with no time to save to disk (an in-guest
sysrq b). - Upgrade: data from the release that was current 12 months before the tested one, upgraded along the documented path.
- Backup and restore: the documented procedure. An undocumented one counts as failed.
- Data exit: a documented export, or data kept in an open format that can be read without the app.
The kill and the crash run 5 times each and pass only if no acknowledged write is lost in any run. The lab counts only while a control program, which writes to SQLite with its strictest durability setting, loses nothing in 5 crashes out of 5. Before we call a loss a finding, we look for its documented cause.
5. Health
Five metrics from the project's last 12 months (following CHAOSS), each rated good, ok or bad. The score is 10 x (good + half of ok) / 5, rounded. GitHub stars are shown, never scored. Health is the only gate that may be n/a, for a project younger than 12 months; the overall is then the mean of five gates, shown as "5/6".
6. Security
Five checks, 2 points each:
- Secure by default: after the documented install, no unauthenticated read or write of user data or admin functions, and no shared default credential.
- Least privilege: the documented container runs as non-root, without privileged mode, the Docker socket, host network or added capabilities.
- Egress: every outbound destination in a 24-hour capture is documented, and usage telemetry is opt-in.
- Account protection: MFA or OIDC in the free self-hosted edition at the tested version. Apps without accounts pass when their docs support forward auth through a reverse proxy; tools with no network-facing interface pass.
- Vulnerability handling: a private reporting channel, and no advisory of medium severity or higher left unpatched for more than 60 days in the last 24 months (GitHub advisories and OSV.dev), for the tested package and its release line.
From scores to a verdict
The overall score is the mean of the gates that apply, with one decimal. It gives one of four verdict words, and the number is always shown next to the word:
| Overall | Verdict |
|---|---|
| 8.0 or more | Use it |
| 6.0 to 7.9 | Use with care |
| 4.0 to 5.9 | Think twice |
| under 4.0 | Skip it |
Caps. A lost acknowledged write in Resilience, a failed core use case, or a failed "secure by default" check holds the verdict at "Use with care" at most. In a data-custody category (backups, passwords, photos and files, notes) a lost acknowledged write holds it at "Think twice". The reason is said and shown, with the mean before the cap.
Floor. Like Euro NCAP's weakest-area rule: a gate under 3 holds the verdict at "Use with care", a gate at 0 or 1 at "Think twice".
Statuses instead of a word. "Not rated" (a component without an OSI licence) and "Unmaintained" (the repository is archived, or there was no release in 12 months; only in re-tests). The measured gates are still shown.
Chips, at most two, each backed by a finding: "Early" (no stable 1.x release, or the first release is under 12 months old), "Solo" (one author made half of the commits or more), "Open core" (features held back for a paid edition), and the reason for a cap or the floor.
Best for, one tag from the results: "Production" when Resilience and Security are 8 or more and the documented restore passed; "Small team" when the second-user use case passed and Security is 6 or more; otherwise "Homelab".
Alternatives in a single-tool episode go through the same load test and are measured, never scored. In a shoot-out, the Pick is the tool with the highest overall among those without a cap or floor, when it leads the next by 0.3 or more; otherwise we say it is too close to call.
Reported, never scored
Facts that matter but do not change a score: the licence family, a CLA, who holds the rights, domain and trademark; the accounts that merged or released in the last 12 months; the project's lineage; an arm64 image; SSO; supply-chain signals (a verified signature, an SBOM, a dated OpenSSF Scorecard, an image scan with the scanner and database date); the funding category; and GitHub stars.
How every episode is checked
- Every factual sentence has a source: a snapshot of the project's own document (README, docs page, licence, release notes, issue) with its address and commit, or a line in our own logs and measurements. A claim without a source is removed, not reworded.
- The writer and the verifiers never share notes. One verifier checks each claim against its source, another recomputes every number from the raw data.
- Every command shown or spoken was run in the lab exactly as shown.
- Opinions appear only when a test result backs them, and are phrased as "in our test".
- The episode page lists the sources. Mistakes are corrected on the page and in the video description, with the date (see corrections).
Category test cards
A test card fixes, before testing, what a category's steady load and use cases are. Cards are versioned on their own, and results from different card versions are never mixed in one comparison.
Push notifications 1.0: steady load of 10 topics with 2 subscribers each and 5 messages a second for 10 minutes, after a 15-minute idle window; a Break Point ramp from 20 to 3,000 requests a second over 180 seconds. Use cases: a backup job reports in (the core use case); a private server with a second user; a nightly report with an attachment and a scheduled reminder. Not a data-custody category: these tools hold messages for delivery, not as your only copy.
Versions and changes
Rubric versions read X.Y.Z. Z changes wording only. Y changes scoring, and old scores are recomputed from the stored measurements. X needs new measurements: older scores are labelled with their version and never mixed in one leaderboard.
- Rubric 1.0, 2026-10-13: first published version, six gates.
- 2026-10-14, clarified before the first episode: a step the install page shows only for another install method counts as a change; check (e) counts advisories for the tested package and its release line, not for a legacy module or major version the project no longer ships.