Dataset Insufficiencies in ADAS: How Early Can You Catch Them?

• 03 Sep, 2026

Imagine an ADAS perception system designed to operate on rural roads, where tractors are part of the expected operational environment. If the dataset specification does not include a tractor class, the problem exists before a single model is trained.

The resulting dataset may contain too few relevant examples or no dedicated tractor annotations at all. A perception system trained and evaluated on that data can therefore reach development with a weakness that originated much earlier, in the dataset specification itself.

BDD100K provides a concrete example. Its object categories include cars, trucks, buses, riders, bikes and motorcycles, but no dedicated tractor category. This does not make BDD100K inherently insufficient. Dataset sufficiency depends on the intended operational design domain (ODD). If tractors are relevant to that ODD, however, their absence from the dataset specification becomes a completeness issue that needs to be addressed.

*BDD100K is a large-scale driving dataset commonly used to develop and evaluate computer vision systems for autonomous driving and ADAS. It includes annotations for many common road users, such as cars, trucks, buses, bicycles, motorcycles, and riders. However, it does not define a dedicated tractor category. That does not make the dataset inadequate by default: whether BDD100K is sufficient depends on the system’s intended operational design domain (ODD). If the system is expected to operate in environments where tractors are common, their absence becomes a dataset completeness gap that must be addressed.

Object category distribution from BDD100k categories. Plot created by the authors using data from BDD100k. Source

This example, discussed in our whitepaper on early detection of dataset insufficiencies in ADAS, using the BDD100K dataset, illustrates an important point about dataset insufficiencies: they rarely begin at the point where their consequences become visible.  As the whitepaper explains, a single dataset insufficiency can appear as an insufficient dataset specification, degraded AI performance, and, under the right conditions, a contributing factor to safety-related behavior.

The important question, then, is not only whether the dataset is good enough when training begins. It is whether the development process is structured to find problems at the point where they first appear.

What Should Be Defined Before Data Collection Starts? 

Finding dataset insufficiencies later in development generally makes them harder to address. If nighttime operation is part of the ODD, for example, identifying before collection that the selected camera cannot provide suitable low-light data gives the engineering team the opportunity to reconsider the sensor, collection approach or requirements. Discovering the same mismatch after a major collection and annotation effort means revisiting work that has already been completed.

In ADAS development, repeating a collection campaign can mean dealing with substantial volumes of sensor data all over again. One published study on video annotation for autonomous driving estimates that a sensor-equipped vehicle can generate up to 40 TB of raw data in a single collection day, depending on the sensor configuration. At that scale, recollection does not only mean putting a vehicle back on the road. It can also mean repeating data transfer, storage, preprocessing, selection and annotation activities. This makes early identification of requirement and hardware mismatches a practical development concern, not simply a dataset quality issue.

This is why the dataset lifecycle described starts with requirements rather than collection.

ISO/PAS 8800, alongside methodologies such as AMLAS and CoDANN, identifies a number of data-related safety properties that can be translated into dataset requirements. These include:

• Relevance: Does the input data match the intended operational environment?
• Completeness: Are the required objects, scenarios and conditions represented?
• Accuracy: Are labels and other data properties sufficiently correct?
• Balance: Does the distribution adequately represent the conditions in which the system is expected to operate?
• Traceability: Can data be traced to its origin and development history?
• Timeliness: Is the data still representative of the current operational environment?
• Independence: Are training, validation and test datasets sufficiently separated?

These are not abstract quality criteria. They give engineering teams something concrete to validate against throughout the dataset lifecycle.

For a computer vision consultancy working on custom AI development, establishing these requirements is also part of the feasibility assessment. The intended functionality, available data, sensors, hardware constraints and operational environment need to be considered together before moving further into development.

What Does a “Complete” Dataset Actually Look Like?

Completeness sounds straightforward until the ODD becomes complex.

If an ADAS perception system is expected to operate during both day and night, having nighttime data is an obvious requirement. If it also needs to operate in rain, rainy data is required too. But looking at each of these conditions separately does not necessarily tell us whether the dataset represents the real-world conditions the system will encounter.

BDD100K provides a useful example. Looking only at its time-of-day distribution, there is a substantial amount of both daytime and nighttime data. Based on this view alone, there is not enough information to conclude that the dataset has a significant balance problem related to nighttime scenarios.

Time of day metadata distribution from BDD100k categories. Plot created by the authors using data from BDD100k. Source

The picture changes when time of day is analyzed together with weather metadata. As shown in the whitepaper, nighttime scenes are predominantly associated with clear weather, while adverse weather conditions at night are underrepresented.

This reveals a broader issue in dataset validation: individual metadata distributions can look reasonable while important gaps remain hidden in the relationships between them. A dataset may contain nighttime scenes and rainy scenes, for example, but still contain very few rainy nighttime scenes.

That distinction matters because perception systems encounter conditions simultaneously. Weather, illumination, road type, traffic density, occlusion and object categories interact in the operational environment. Balance and completeness therefore need to be evaluated not only through individual category counts, but through the combinations that are relevant to the ODD.

Metadata Is What Makes Those Gaps Visible

Useful metadata can include properties such as:

• time and location
• weather and illumination
• road and junction type
• collection sensor
• relevant scenery conditions
• dataset role, such as training, validation or test

The exact metadata depends on the use case. Its purpose is to make dataset requirements measurable and the data searchable, traceable and suitable for structured validation.

Metadata also helps prevent less obvious problems such as data leakage. If a collection vehicle records continuous footage and the resulting clips are divided randomly between training and test datasets, footage recorded seconds apart can end up in both. The test dataset may then contain frames that are nearly identical to those the model encountered during training.

Timestamp metadata can help enforce temporal separation. Geolocation can provide additional separation when the same route is driven repeatedly. Sensor information becomes useful if a hardware problem is discovered and all data recorded with the affected device needs to be identified.

Moreover, metadata should not be designed only around immediate project needs. Data is frequently reused in later projects, and insufficient metadata can make valuable historical data difficult to select, validate or interpret in a new operational context.

The scale of automotive data makes this relevant. NVIDIA estimates that a single autonomous-driving data collection vehicle can generate upwards of 2 PB of data per year, with the actual volume depending on factors such as the number and type of sensors, operating hours and geographical coverage. At that scale, collecting data without sufficient metadata and traceability can leave teams with large volumes of data that are expensive to store but difficult to search, select and reuse effectively.

When Is a Correct Label Actually Incorrect?

Dataset quality is not only about having the right images. What those images mean also needs to be defined consistently.

Consider a truck pulling a trailer. Should the truck and trailer have one bounding box or two?

There is no universally correct answer independent of the system. What matters is that the annotation represents how the data will be used by the perception layer.

Without a clear requirement, two annotators can make different choices while both reasonably believe they are labeling the image correctly.

The same problem appears in environmental metadata. The whitepaper uses BDD100K images labeled as snowy to illustrate how one category can represent substantially different conditions. Snow outside the driving path is not necessarily equivalent to a road covered in snow, particularly if the label is intended to describe conditions relevant to vehicle perception.

Annotation requirements therefore need to answer three questions:

1. What needs to be labeled?
2. How should it be represented?
3. What makes the annotation acceptable?

Labeling instructions can then turn those requirements into repeatable rules, ideally supported by positive, negative and difficult examples.

DARS, the Data Annotation Requirements Representation and Specification approach, is one method for making annotation requirements more explicit. Its example defines not only the category and bounding-box behavior, but also the scenario context, rationale and measurable acceptance criteria.

Automated labeling can reduce the manual workload, but it does not remove the requirement for review. For safety-relevant data, model-generated annotations can be treated as pre-labels that trained annotators confirm, modify or reject.

What Happens When the World Changes but Your Dataset Does Not?

Dataset requirements do not stop changing once a model reaches satisfactory performance.

A production-ready perception system has to account for an operational environment that evolves. New vehicle categories become common, product requirements expand and field data reveals scenarios that were not sufficiently represented during initial development.

The Early detection of dataset insufficiencies in ADAS white paper uses electric kick scooters as an example.

BDD100K does not contain a dedicated kick scooter category, despite containing images in which people are riding them. Kick scooters also differ from categories such as bicycles and motorcycles in their shape, size and rider pose.

If a perception system now needs to recognize them explicitly, the first step should not be to start collecting scooter images. The new requirement needs to be defined first.

Depending on the existing dataset and the intended functionality, several options may then be considered:

• collect new real-world data
• review and relabel data already available
• generate suitable synthetic data
• use augmentation where its adequacy can be demonstrated

Whichever approach is selected, the new data needs to be evaluated against the wider dataset requirements rather than only against the new object-class target.

This matters because fixing one gap can create another.

Suppose a team needs a certain number of kick scooter examples and reaches that number successfully. If most of the data was collected during dry, sunny weather because that is when people are more likely to ride scooters, the updated dataset may now be less balanced across weather conditions. The target for the new class has been reached, but the overall dataset has changed in a way that may affect performance elsewhere.

Dataset expansion therefore needs to remain iterative, with every significant update followed by structured validation against the requirements that still apply.

One of the reasons why these checks need to be carried out at various stages during development is that it is important to avoid recalling unnecessary data. A new data collection campaign might be required since the ODD has expanded, relevant scenarios are not well represented, the original sensor arrangement was inappropriate, the existing metadata is not able to reliably identify the data needed, or changes to the annotation requirements make it difficult to reuse the data that has already been collected. There are also situations in which the data already exists and the gap can be filled by better selection, relabeling or augmentation. The aim of structured validation is to determine which kind of situation the team is facing before deciding on another collection effort.

Can Automated Checks Tell You Whether the Dataset Makes Sense?

Only up to a point.

Automated checks are particularly useful when an evaluation needs to be objective, consistent and scalable. They can analyze dataset distributions, detect corruption, verify label formats and compare different dataset versions quantitatively.

They cannot reliably determine whether every scenario makes sense in its operational context.

As described in the whitepaper, automated checks might show high label precision, balanced classes, synchronized sensor data and no obvious corruption while human review still uncovers gaps such as:

• children being significantly underrepresented in pedestrian data
• missing regional clothing styles
• adverse-weather examples that technically satisfy a metadata category but are not relevant to the intended perception task
• dangerous scenario combinations that are absent entirely

Human review provides the contextual understanding needed to identify these issues, while automated validation provides the consistency and scale needed to evaluate large datasets. The two are complementary.

The timing of that review is just as important as the method. Reviewing only the final dataset means allowing potential insufficiencies to pass through several stages before they are examined.

As the dataset lifecycle in our whitepaper shows, review can be built into each stage of dataset implementation, from data collection and preprocessing to file selection, labeling and annotation postprocessing. Findings can also feed back into dataset design and requirements, making validation an iterative part of development rather than a final checkpoint.

Diagram of suggested way of working in Dataset implementation phase. Source

Raw collected data can be reviewed before annotation, preprocessing outputs can be checked before training, selected data can be examined for representativeness, annotations can be audited for consistency and postprocessed labels can be verified against what will actually enter the training pipeline. Each review point creates an opportunity to identify an insufficiency closer to where it was introduced.

How Early Can Dataset Safety Analysis Start?

Before the data exists.

Under ISO/PAS 8800, dataset safety analysis is an iterative process for identifying safety-relevant dataset insufficiencies and understanding their causes and consequences. The results can feed back into dataset requirements, design and implementation rather than serving only as evidence at the end of development.

Consider a collection plan designed to accumulate driving kilometers efficiently. Highway driving may be attractive because large amounts of footage can be collected quickly, but if the ODD also contains parking areas, urban intersections and other low-speed environments, that collection strategy can create a representativeness problem before the vehicle has recorded a single frame.

The whitepaper illustrates this with the BDD100K scene distribution. If parking spaces were structures the autonomous vehicle was expected to use, the relatively small representation of parking-lot images could constitute a relevant dataset insufficiency. Analyzing the collection plan against the ODD before collection can expose that type of imbalance while it is still relatively straightforward to correct.

Dataset safety analysis can then be repeated when the ODD changes, the dataset is expanded or a new hazard is identified.

Where Can Dataset Insufficiencies Be Caught?

The answer is not at one particular stage. Dataset requirements, design, hardware selection, collection, preprocessing, selection, labeling and dataset expansion can all introduce insufficiencies, which means they also provide opportunities to identify them before they propagate further.

For an end-to-end development process, that means connecting:

• a clearly defined operational environment
• measurable dataset requirements
• appropriate sensors and collection methods
• metadata designed for validation and traceability
• explicit annotation requirements
• automated checks and expert human review
• dataset versioning
• iterative dataset safety analysis

Not every insufficiency can be predicted in advance, and real-world conditions will continue to reveal new cases as the product evolves. The goal is not to build a dataset once and declare it complete. It is to establish a development process that can detect gaps, understand their impact and address them systematically.

For a computer vision consultancy working as an R&D partner, this is fundamental to custom AI development. A production-ready perception system depends on more than model architecture or benchmark performance. Its data, sensors, requirements, validation approach and deployment constraints have to support the same operational objective.

This becomes particularly important for Edge AI, where the perception layer eventually has to perform on the target hardware and under the real-world conditions defined for the product.

As we show in the whitepaper, every stage where a dataset insufficiency can be introduced is also a stage where it can potentially be detected. Building structured validation and review into those stages does not eliminate the iterative nature of AI development. It makes that iteration traceable, evidence-based and considerably more useful for getting from an initial concept to a production-ready system.

Dataset engineering is only one part of that process. See how we approach custom AI development end-to-end, from understanding the operational problem and assessing feasibility to iterative development, structured validation and deployment.