Open edX Vendor Evaluation — Scoring Framework
How to Use This Framework
This 44-criterion framework covers the full range of factors relevant to selecting an Open edX hosting and implementation vendor. Each criterion has a default weight (0 = out of scope, 1.0 = critical). Adjust weights based on requirements gathered in 01-requirements-gathering.md.
Scoring scale: 1 (Poor) → 2 (Below Average) → 3 (Acceptable) → 4 (Good) → 5 (Excellent)
Weighted score per criterion: score × weight
Final weighted total: Sum of all weighted scores. Use this for cross-vendor comparison.
Hard Filters (Apply Before Scoring)
Some criteria should be treated as non-negotiable for US higher education institutions. Vendors failing any hard filter should be eliminated before entering the scoring matrix.
| Hard Filter | Notes |
|---|---|
| US data residency | Must be confirmed in writing, not just listed as "supported" |
| FERPA compliance | SOC 2 Type II report and/or documented FERPA controls required |
| WCAG 2.1 AA | Applies to all LMS-facing interfaces |
| Minimum uptime SLA | Typically ≥ 99.9% for production academic environments |
Category 1: Hosting & Infrastructure
| # | Criterion | Default Weight | What to Evaluate |
|---|---|---|---|
| 1 | Hosting model | 0.5 | SaaS, dedicated cloud, BYOC availability; which models support institutional ownership requirements |
| 2 | Cloud provider flexibility | 0.5 | AWS, Azure, GCP support; ability to deploy to client-owned cloud accounts |
| 3 | Uptime SLA | 0.4 | Guaranteed uptime percentage; P1/P2/P3 incident response times |
| 4 | Scalability | 0.8 | Kubernetes-based auto-scaling; ability to handle peak loads (exams, enrollment) without manual intervention |
| 5 | Data residency | 1.0 | US-based hosting; confirmation that all data remains within US borders; documentation available |
| 6 | Backup & disaster recovery | 0.5 | Backup frequency, retention period, RTO/RPO targets; offsite replication |
| 7 | Security & compliance | 0.8 | AES-256 encryption at rest; TLS 1.2+ in transit; SOC 2; FERPA; penetration testing cadence |
Weight adjustment guidance:
- If data residency is non-negotiable: treat criterion 5 as a hard filter, not a scored item
- If the institution has existing cloud contracts: increase criterion 2 weight to 0.8–1.0
Category 2: Platform Expertise
| # | Criterion | Default Weight | What to Evaluate |
|---|---|---|---|
| 8 | Years with Open edX | 0.4 | Length of active Open edX practice; not just founding year |
| 9 | Partnership tier | 0.7 | Official Open edX Marketplace tier (Partner > Verified Provider > Marketplace Provider) |
| 10 | Core contributions | 0.6 | Active commits to the Open edX codebase; Core Contributor status; leadership in working groups |
| 11 | Higher ed clients | 0.7 | Verifiable track record with colleges and universities at comparable scale |
| 12 | Migration experience | 0.0* | Prior LMS-to-Open edX migrations; data integrity handling; rollback capability |
| 13 | Release management | 0.7 | Upgrade cadence; how quickly clients move to named releases; who manages the process |
*Set weight to 0 when migration is out of scope.
Weight adjustment guidance:
- If migration is in scope: increase criterion 12 to 0.8
- If open-source community standing matters: increase criterion 10 to 1.0
Category 3: Analytics & Reporting
| # | Criterion | Default Weight | What to Evaluate |
|---|---|---|---|
| 14 | Aspects support | 0.6 | Ability to deploy, configure, and support the native Open edX Aspects analytics suite (ClickHouse + Superset) |
| 15 | Custom dashboards | 0.0* | Custom reports and dashboards beyond Aspects out-of-the-box; Superset customization experience |
| 16 | Data export/integration | 1.0 | Export to data warehouses (Snowflake, BigQuery); integration with institutional BI tools; future-readiness |
| 17 | Accreditation reporting | 0.3 | Ability to generate compliance-specific reports for regional or programmatic accreditation |
| 18 | Predictive analytics/AI | 0.0* | AI-driven learner analytics; at-risk detection; engagement scoring |
*Set to 0 for Phase 1 / minimal analytics deployments.
Weight adjustment guidance:
- If Aspects is a requirement: increase criterion 14 to 1.0
- If external BI integration is required: criterion 16 is already weighted at 1.0
Category 4: Customization & Development
| # | Criterion | Default Weight | What to Evaluate |
|---|---|---|---|
| 19 | Theming & branding | 0.9 | Full visual customization to institutional identity; Paragon/MFE theming capability |
| 20 | Custom XBlocks | 0.2 | Experience building custom XBlocks; SCORM/xAPI support; specialized assessment components |
| 21 | Mobile experience | 0.2 | Official Open edX mobile app support; custom mobile features; responsive web quality |
| 22 | LTI integration | 0.0* | LTI 1.3/Advantage for third-party tool integration; provider and consumer roles |
| 23 | Authentication (SSO/SAML) | 0.2 | SAML 2.0, OAuth2, institutional identity provider integrations (Shibboleth, Azure AD, Okta) |
| 24 | E-commerce | 0.2 | Payment gateway integrations; course monetization; ecommerce plugin support |
| 25 | Accessibility (ADA/WCAG) | 1.0 | WCAG 2.1 AA compliance across all LMS interfaces and custom components; Section 508 |
*Set LTI weight to 0.8–1.0 if external tool integrations are required.
Weight adjustment guidance:
- If branding is Phase 1 deliverable: criterion 19 weight at 0.9 is appropriate as default
- If mobile app is required: increase criterion 21 to 1.0
- If no e-commerce is planned: set criterion 24 to 0
Category 5: Support & Service
| # | Criterion | Default Weight | What to Evaluate |
|---|---|---|---|
| 26 | Support hours & channels | 0.8 | 24/7 vs. business hours; ticket, chat, phone, Slack; escalation paths |
| 27 | Dedicated account management | 0.6 | Named account manager or CSM; regular check-ins; onboarding support |
| 28 | Response time SLAs | 0.6 | P1/P2/P3 response and resolution commitments; documented in contract |
| 29 | Upgrade support | 0.5 | Named release upgrades included or available as add-on; testing and rollback procedures |
| 30 | Training & onboarding | 0.5 | Onboarding sessions; Studio training; documentation quality; knowledge base availability |
| 31 | Migration support | 0.0* | Hands-on migration assistance; data mapping; LMS content conversion |
*Set to 0 when migration is out of scope.
Weight adjustment guidance:
- If institution has no internal technical staff: increase criterion 26 to 1.0 and criterion 30 to 0.9
- If a dedicated CSM is required: increase criterion 27 to 1.0
Category 6: Pricing & Terms
| # | Criterion | Default Weight | What to Evaluate |
|---|---|---|---|
| 32 | Pricing model | 0.7 | MAU-based, compute-based, flat-fee, or usage-based; predictability and transparency |
| 33 | Setup / build costs | 0.6 | One-time costs to stand up the instance; instance installation fees; onboarding costs |
| 34 | Ongoing hosting costs | 0.8 | Annual infrastructure costs; what is and is not included at base rate |
| 35 | Maintenance costs | 0.8 | Upgrade management, monitoring, platform upkeep included or additional |
| 36 | Customization costs | 0.6 | Published hourly rates or project-based quotes for development work |
| 37 | Contract terms | 0.6 | Commitment length; auto-renewal clauses; exit/termination conditions; data portability |
| 38 | Cost scalability | 0.9 | How costs grow as MAU and usage expand; overage billing policies |
Weight adjustment guidance:
- If budget is the primary constraint: increase criteria 32–38 all to 1.0
- If a multi-year contract is preferred: increase criterion 37 to 1.0 and ask specifically about multi-year discounts
Category 7: Strategic Fit
| # | Criterion | Default Weight | What to Evaluate |
|---|---|---|---|
| 39 | Vendor stability | 0.8 | Company age, team size, funding/ownership, client retention signals |
| 40 | Open-source commitment | 0.7 | Core contributor status; community governance participation; code contribution volume |
| 41 | Innovation / roadmap | 0.5 | Active investment in new Open edX features; AI capabilities; roadmap transparency |
| 42 | Timezone alignment | 0.8 | Business hours overlap with institution's primary time zone; language of support |
| 43 | Reference clients | 0.6 | Verifiable references at similar scale; willingness to provide client contacts |
| 44 | Implementation partner fit | 0.9 | Compatibility with the selected implementation partner's workflow, tools, and delivery process |
Weight adjustment guidance:
- If the institution is risk-averse: increase criterion 39 (vendor stability) to 1.0
- If the institution is in a specific time zone with limited overlap: increase criterion 42 to 1.0
Scoring Template
Copy this table for each vendor being evaluated. Fill in score (1–5) per criterion, then compute weighted scores.
| Category | Criterion | Weight | Score (1–5) | Weighted Score |
|---|---|---|---|---|
| Hosting & Infrastructure | Hosting model | 0.5 | ||
| Cloud provider flexibility | 0.5 | |||
| Uptime SLA | 0.4 | |||
| Scalability | 0.8 | |||
| Data residency | 1.0 | |||
| Backup & disaster recovery | 0.5 | |||
| Security & compliance | 0.8 | |||
| Platform Expertise | Years with Open edX | 0.4 | ||
| Partnership tier | 0.7 | |||
| Core contributions | 0.6 | |||
| Higher ed clients | 0.7 | |||
| Migration experience | 0.0 | |||
| Release management | 0.7 | |||
| Analytics & Reporting | Aspects support | 0.6 | ||
| Custom dashboards | 0.0 | |||
| Data export/integration | 1.0 | |||
| Accreditation reporting | 0.3 | |||
| Predictive analytics/AI | 0.0 | |||
| Customization & Dev | Theming & branding | 0.9 | ||
| Custom XBlocks | 0.2 | |||
| Mobile experience | 0.2 | |||
| LTI integration | 0.0 | |||
| Authentication (SSO/SAML) | 0.2 | |||
| E-commerce | 0.2 | |||
| Accessibility (ADA/WCAG) | 1.0 | |||
| Support & Service | Support hours & channels | 0.8 | ||
| Dedicated account management | 0.6 | |||
| Response time SLAs | 0.6 | |||
| Upgrade support | 0.5 | |||
| Training & onboarding | 0.5 | |||
| Migration support | 0.0 | |||
| Pricing & Terms | Pricing model | 0.7 | ||
| Setup / build costs | 0.6 | |||
| Ongoing hosting costs | 0.8 | |||
| Maintenance costs | 0.8 | |||
| Customization costs | 0.6 | |||
| Contract terms | 0.6 | |||
| Cost scalability | 0.9 | |||
| Strategic Fit | Vendor stability | 0.8 | ||
| Open-source commitment | 0.7 | |||
| Innovation / roadmap | 0.5 | |||
| Timezone alignment | 0.8 | |||
| Reference clients | 0.6 | |||
| Implementation partner fit | 0.9 | |||
| TOTAL | =SUM() |
Completed Example
A completed evaluation applying this framework to six vendors is documented in: open-edx-provider-research/24-us-higher-ed-procurement-framework.md
That document includes March 2026 vendor rankings, pricing benchmarks, and US compliance assessment.