1 · Human research
You research a federal dataset's ecosystem — no AI allowed — and rate its risk against the rubric. About 50 minutes.
Start Phase 1What we're testing
Two hypotheses:
- Assessment. Guided by a rubric constructed by data policy experts, an AI agent can assess the risk to data comparable to a human researcher.
- Prompts. When people steer an AI agent toward better assessments, how they steer it tells us how to build a better automated monitoring tool.
This exercise comes out of the Public Data Project's data monitoring work and sits alongside Binoc, a tool that generates changelogs for datasets that don't have them. Binoc reports what has changed; this exercise probes whether AI can anticipate what might change.
Intended Audience
You do not need any experience using AI to complete this exercise. The intended audiences are people who use and think about public data, such as data rescue groups, librarians, policy researchers, scholars, and journalists.
Important note before you begin
Do not consult America's Data Index until the end of the exercise. It's the closest thing to an answer key, and looking early contaminates the comparison. You'll be pointed to it at the finish.
The phases
2 · Human-prompted AI
You run the assessment through an AI agent of your choice, coaching its method — never its answers. About 50 minutes.
Preview Phase 2