Walk into any education technology conference and every booth promises the same thing: transformative AI that personalizes learning, reduces teacher workload, and improves outcomes. Dig into the details and the picture gets murkier. Some vendors have done rigorous research. Others have a demo that looks impressive and nothing behind it. Schools that sign contracts without proper due diligence often end up with tools that do not work as promised, do not integrate with their systems, or create more problems than they solve.
The evaluation problem in EdTech
Most schools evaluate AI vendors the same way they evaluate traditional software — reviewing features, checking references, and negotiating price. AI requires a different approach. The technology behaves differently in production than in controlled demos. Models can degrade. Prompts can break. Integration complexity is higher. And the vendors themselves vary wildly in technical maturity, even when their marketing looks similar.
The best AI demo is not the best AI product. The best AI product is the one that still works six months after deployment.
Technical due diligence
Before signing a contract, dig into the technical foundation. Ask vendors to explain not just what their AI does, but how it works under the hood. The right questions reveal whether a vendor understands their own technology or is just wrapping someone else's API.
- What model or models does the product use, and who provides them? If the vendor does not own the model, what happens if the model provider changes pricing or discontinues the service?
- How does the vendor handle model drift — the gradual degradation of model performance over time? What monitoring do they have in place?
- Can the vendor demonstrate the system working with real data similar to your school's data, not just curated demo data?
- What is the latency profile — how quickly does the AI respond during peak usage times?
- How does the vendor handle errors or failures? Does the system fail gracefully or does it simply stop working?
Data privacy and security
Student data is not optional to protect. FERPA, COPPA, and state-specific regulations create a compliance floor, but the deeper question is whether the vendor treats data as an asset to be monetized or as a responsibility to be protected. The difference shows in their architecture, their contracts, and their answers to tough questions.
- Does the vendor use student data to train models that serve other customers? This is a deal-breaker for most districts.
- Where is data processed? If the vendor uses cloud services, where are the data centers and what are the data residency implications?
- Can the district export all student data in a standard format at any time, including during the contract and after it ends?
- What is the vendor's incident response plan for data breaches? Ask for specifics, not generic assurances.
- Does the vendor have third-party security certifications — SOC 2, ISO 27001, or equivalent?
Integration and deployment
An AI tool that requires teachers to log into a separate system, manually sync data, or work around integration failures will not be used. The integration requirements are often where vendors are most optimistic and schools are most disappointed.
- What authentication systems does the vendor support? Can students and teachers use their existing district credentials?
- How does the vendor handle roster sync — the process of keeping student and teacher lists up to date?
- What LMS, SIS, or other systems does the vendor integrate with, and what is the depth of integration?
- Who owns the integration work — the district, the vendor, or a third party?
- What happens to the integration if the district changes LMS or SIS vendors?
Evidence and outcomes
Every vendor claims their product works. Few can demonstrate it with credible evidence. Ask for specific, verifiable outcomes from schools similar to yours — not case studies written by marketing, but data you can validate.
- Can the vendor connect you with districts similar to yours that have used the product for at least a full school year?
- What specific outcomes did those districts measure? Ask for the metrics, not just the narrative.
- Has the vendor published any third-party research or independent evaluations of their product?
- What does the vendor's internal testing look like? Do they test with diverse learners, or just the average case?
- What happens if the product does not deliver the promised results? Are there contract terms that protect the district?
Support and sustainability
AI products evolve faster than traditional software, which means both the opportunities and the risks are greater. A vendor that cannot sustain development creates risk; a vendor that changes too fast without warning creates a different kind of risk. Find the balance.
- How often does the vendor update the AI model, and how are updates communicated to customers?
- What is the vendor's financial stability? Are they venture-backed burning cash, or profitable and sustainable?
- What support tiers are available, and what is the typical response time for issues?
- What happens if the vendor is acquired or goes out of business? Is there a data export and transition plan?
- Does the vendor have a product roadmap they can share, and how do they incorporate customer feedback?
What Nivorius has seen
In our work with schools evaluating AI vendors, the patterns are consistent. Districts that do thorough technical due diligence avoid the most expensive mistakes. Districts that talk to other schools using the product for a full year catch problems that demos do not reveal. And districts that negotiate contract protections around data portability and performance guarantees protect themselves when things do not go as planned.
The framework above is not about being skeptical for the sake of skepticism. It is about being rigorous because the stakes are high — student data, student outcomes, and taxpayer money. The best vendors welcome these questions because they have good answers. The vendors that deflect or dismiss are telling you something.
Part of the Nivorius research and consulting team, focused on practical applications of AI in education and enterprise contexts.
