Data Annotation Platform Development for AI/ML Platforms

What this project is

Annotation is judgment work dressed as data entry. A labeller deciding whether a tumour boundary includes a faint edge, or whether a comment is harassment, is making a call the model will copy. The platform's job is to make those calls consistent: clear guidelines, examples of edge cases, and a review path for anything uncertain.

Most annotation projects fail on guidelines, not tooling. If two labellers read the same instruction differently, the dataset teaches the model both answers, and accuracy stalls no matter how much data is added. The platform must therefore treat guidelines as a living document, versioned alongside the labels.

Core features

• Task-specific editors: bounding boxes and polygons for images, spans for text, transcripts for audio, and frame ranges for video.
• Guideline panels shown inside the task, with examples and edge cases, so labellers never work from memory.
• Double labelling and consensus rules, routing disagreements to a reviewer instead of averaging them away.
• Agreement scoring per labeller and per task type, flagging drift early.
• Dataset versioning and exports in common formats, with a record of which labels shipped in each version.
• Workforce management: task assignment, pay tracking, and throughput views for managers.

How it is built

1. One data type first. Pick the type your first customer needs, build its editor and export, and prove quality on a real dataset.
2. Quality workflow second. Review queues, consensus, and agreement scoring, because volume without quality checks creates rework.
3. Workforce tooling third. Assignment, onboarding tests, and performance views for larger labelling teams.
4. More data types fourth, reusing the task engine, review flow, and export system.
5. Automation last. Pre-labelling with existing models and active learning, which only helps once human quality is measured.

Technology considerations

Annotation editors run in the browser and must stay responsive on large images and long videos, so media handling matters more than framework choice. Task state belongs in a queue system that can reassign work when a labeller leaves mid-batch. Exports should be plain, documented formats rather than a proprietary dump, since customers feed them into training pipelines they already own. Build timelines vary with the number of data types; confirm with a quote.

• Browser-based editors with lazy loading for large media files.
• A task queue with assignment, timeout, and reassignment rules.
• A relational store for labels, versions, and agreement scores.
• Export jobs that produce documented, standard formats.

Challenges to plan for

• Guideline drift. As projects grow, informal rulings pile up in chat; the platform needs versioned guidelines and a way to relabel affected tasks.
• Labeller fatigue. Repetitive tasks lower care, so batch sizes, breaks, and quality sampling matter as much as pay rates.
• Edge-case pileup. A small share of tasks consume most reviewer time, so the queue must surface hard tasks separately from routine ones.
• Customer disputes. A buyer who disagrees with labels needs evidence: who labelled what, under which guideline version, and what the reviewer decided.

A worked example

A medical imaging startup needs ten thousand chest scans labelled for a detection model, with each scan reviewed by two radiologists and disagreements settled by a senior reviewer. The platform loads the scans, presents them with a polygon tool and the current guideline version beside every task, and routes each scan to two labellers independently. Where the two labellers disagree on a boundary, the task goes to the senior review queue with both versions visible, and the final label records who decided and under which guideline version.

After two weeks, agreement scoring shows one labeller drifting on faint edge cases, so their recent tasks are sampled for re-review before the dataset ships. The final export versions the dataset, records the guideline version used, and ships in a standard format the training pipeline already reads. When the startup later updates its guidelines for a new scanner type, affected tasks are identified and relabelled rather than silently mixed with old labels. That audit trail is what lets the startup defend its dataset to a clinical partner, and it is the part a spreadsheet-based process could never have produced.

Who it is for, and its limits

This fits ML teams and data vendors that label at volume, and companies in regulated fields where label provenance matters, such as medical imaging or autonomous driving data.

It does not fit a one-off labelling job of a few hundred items, where a spreadsheet and a small team are cheaper. It also does not fix a modelling problem: if the task itself is ill-defined, no platform makes the labels right.

Frequently asked questions