Testing Cellular-Based Traffic Data
Collection Technologies
Performance Required for Road Management
and Traveler Information
Based on Case Study of Benchmarking Cellint's Trafficsense with Test Drives and Road Sensors
Abstract:
There are many challenges in testing data collection accuracy and latency of floating car technology, especially cellular based. Using floating test vehicle as reference, for example, will provide good results even if the detection system doesn't detect slowdowns at all, when averaged for a long time. Cellint's results, validated by third parties, are outlined in this article. Slowdown detection latency is the most important factor, which should be benchmarked against road sensors ONLY during times of SPEED FLUCTUATIONS. A few minutes latency in detecting slowdowns and 10% difference in travel time can be achieved by these systems.
Introduction:
Cellint began to develop its TrafficSense system on 2001, and by 2004, when our first release was ready, we approached several potential customers to test the system and evaluate its performance. We found it challenging, as regular methods like floating test vehicles couldn't provide a reliable reference. For example, even if TrafficSense reports free flow 24 hours a day (which wasn't true) we still get excellent results on average. Because of such problems we started to analyze the challenge of testing floating vehicle detection technologies in general, and specifically cellular based. This paper addresses a set of questions that were identified during this study, along with the conclusions we reached as a result of these findings.
Setting and Testing Required Performance
Testing performance of cellular based traffic data collection should be done 24/7, but most of the data will not be relevant, since its free-flow traffic and detection cannot be evaluated during this part of the day. Even a “poor” traffic detection system, that can't detect slowdowns, will be correct 22 hours out of the 24 hours a day because it is free flow times. This means that if we measure the average error of this system during these 24 hours, the average error will be less than 10%!
Conclusion 1: Accuracy criteria must be tested ONLY in times of Speed Fluctuations.
Main Criteria - Slowdown Detection Latency
Slowdown detection latency is critical for all types of applications, such as road management, reliable travel time and incident alerts:
-
For road management – this will be the main purpose of detecting incidents in order to clear them quickly. Early alerts can also mitigate traffic and reduce the impact of slow downs or incidents on the amount of traffic congestion.
-
For travel time – timely data is critical, otherwise travel time is unnecessary.
-
For traveler information – long before the effect of travel time becomes significant, the traveler can already be alerted that a possible incident has occurred, and re-routing recommendations can be considered.
Conclusion 2: Slowdown latency is the most important factor to test.
Which Reference System Can We Use?
Measuring performance during speed fluctuations is not trivial. Every FCD (floating car data) system that is typically employed as reference has inherent problems:
Test vehicle:
It is very most likely that a test vehicle can cover a roadway many times and never be in the “right place at the right time” to detect a rapid speed change. This means that we need to conduct thousands of test drives in order to achieve a reliable statistical sample.
Conclusion 3: Traffic conditions must be measured continuously and simultaneously in many locations for effective performance measurements, and not with sporadic test drives.
Number Plate Readers:
Unlike test vehicle, Number Plate Readers do monitor continuously and are theoretically the best way to evaluate performance of a floating car detection system, since it tracks most of the vehicles most of the time. However, due to the high cost associated with deploying such systems, the detection points will be far apart (several km on average) thus having very long delays in detecting speed change.
Example:
When an incident or slowdown occurs, the queue of vehicles starts at the incident location, and grows over time. Let's define 30% slowdown as the benchmark for an incident alert. In case of 40% slowdown, it will take about 5 minutes for the queue to grow by 0.25 mile during day time. At this point, a high resolution detection system can provide an incident alert. If travel time is only measured over a 2 mile section, then it will change only by 8% if the queue covers one sub section of 0.25 miles. In order to receive incident alert on a 2 mile section (i.e. 30% reduction in average speed), we need to wait a minimum of 5 times 5 minutes, or 25 minutes! The details of this calculation can be viewed in Table 1.
It seems that the only practical way to measure slowdown detection latency of floating car data is to compare the average speed at each short road
section to a road sensor at that location. Road sensors, which detect all vehicles all the time at a specific location, can detect speed changes immediately at these specific points, and can act as a reliable reference.
Although the local speed measured by a road sensor is different than a speed over a short section, as measured by the floating car system, this difference is negligible and doesn't significantly influence the detection of slowdowns, as was proven in the tests described in this article.
Conclusion 4: In order to detect speed fluctuations immediately with the reference system, we must use road sensors as a reference.
Why Not Rely Only on a Drive Test?
Performance of a cellular based traffic detection system changes dramatically due to many parameters, such as traffic volume, time of day, type of road, network density, etc. Defining “true stand-alone” criteria for all these scenarios is not realistic, but testing them is required. Since road sensors also provide different performance for different traffic parameters, a benchmark against existing deployment of road sensors, which are considered to be the standard in the industry, can provide a good indication of system performance during these different scenarios.
Conclusion 5 :Comparison with existing deployment of road sensors will test the system during most relevant scenarios.
Travel Time Measurement
There is an inherent problem in measuring travel time accuracy if measured in large intervals (between junctions which are far apart). Road conditions may change significantly on sections that were previously driven by the vehicle during speed fluctuations. This will result in varying measurements during a single journey. The shorter the section is, the more reliable the test will be. In the extreme case one can take the average speed over very small
sections of 0.25 mile – but then it is actually local speed measurements and not travel-time.
In this case it might be worth testing the raw data directly and avoid averaging of problematic data: by tracking the test vehicle with the cellular system several times over a short interval (e.g. 2 miles) and comparing it to the travel time recorded by the test vehicle itself.
Conclusion 6 : Travel time should be measures over short intervals (~2 miles) and the best way to validate the quality of a system is to test the data provided by the cellular system for the specific test vehicle.
Link Speed (average speed over short road section)
As discussed above, the most appropriate way to measure link speed is by comparing it to a road sensor. The resolution for link length that can be provided by the cellular detection system in urban areas is between 0.1 miles to 0.3 miles, depending on network density and traffic patterns.
Required Performance
One of the most important questions is: What performance can we expect from cellular based traffic data collection?
As mentioned above, since we can't take into consideration all possible traffic scenarios, it is almost impossible to set performance criteria for all this range of parameters. Thus the best way to evaluate this system is to use the performance of the reference system as criteria, while the reference system is considered the standard in the industry.
However, when measuring local speed at specific points on the road where a sensor is deployed, the sensor will always have advantage in detecting slowdown for the following reasons:
-
The sensor detects all the vehicles at that specific point on the road, while the cellular based system detects only a sample of the vehicles
-
The sensor has 5 meters resolution at that point, while the cellular based system can reach an accuracy of few tens of meters at the best case scenario
Based on the above analysis, as well as initial results from the test described in this paper, we prepared initial criteria which include the following:
Latency criteria – Up to a few minutes delay on average for detecting slowdowns in comparison with the road sensors system. This is a possible standard and can be achieved by cellular based detection system
Travel Time accuracy criteria – Up to 10% average difference During Speed Fluctuation between the travel time recorded by the test vehicle and the travel time of that vehicle reported by cellular detection systems over 2 miles sections can be a possible standard and can be achieved by cellular based detection system.
Speed Link – Average difference between the local speed measured by the road sensor and the link speed measured by the cellular detection system around the sensor can be less than 5 miles per hour.
Conclusion 7: Latency criteria, link speed and travel time accuracy criteria for cellular based traffic data collection system can be defined to ensure effectiveness, as well as a proper testing method.
Can these criteria be achieved ? - Several pilots conducted by DOT agencies on Cellint's TrafficSense system proved that such abilities can be provided based on cellular data.
The section below describes the initial results of a test conducted by the Kansas Department of Transportation, as well as tests conducted by Ayalon HW, an Israeli DOT agency and the National Road Company in Israel.
Test Case
Cellint’s TrafficSense system extracts anonymous information out of a cellular network to provide accurate and real time incident alerts, travel time and traffic speed.
The system was benchmarked by a DOT agency for several months in real time with inductive loops (which are 0.3 miles apart on average) and with test drives (over 2-3 mile sections between junctions on average).
TrafficSense reported local speed over small road segments ( 250 yards long) and the data was compared to the local speed reported by inductive loops and side view radars.
The criteria for slowdown was reduction of the local speed by at least 10 mile per hour (mph) within less than 10 minutes, while the speed before the slowdown was larger than 50 mph and after the slowdown it was less than 50 mph.
The results for detecting slowdowns:
-
1 minute average delay in detecting slowdowns in comparison with road sensors over Highway 1 in Israel (at sensors' location ( .
-
4 minutes average delay in detecting slowdowns in comparison with road sensors over the SCOUT system in Kansas City, US (at sensors' location).
The results for detecting travel time and local speed :
-
Average travel time difference of less than 10% in comparison with test vehicles.
-
Less than 5 mile per hour average difference when comparing each sensor's speed to TrafficSense's speed over 250 yards road section around the sensor
Details about the project and the results can be viewed in the following Figures, as well as about similar pilot conducted in Kansas City, US.
Conclusion 8: Cellular-based traffic data collection systems can provide reliable slowdown detection, local speed and travel time measurements.
Figure 1 – Slowdown Latency of Short vs. Long Intervals
The above table shows detection of speed change over 2 mile section will take 25 minutes longerrelative to detection of speed change over 0.25 mile section. It demonstrates that high resolution detection (i.e. over small sections) is necessary for accurate, real time speed change detection. The only way to test the latency of detection in such small sections is by comparing the average speed of each such section to the local speed provided by a road sensor at this point.
Figure 2 - TrafficSense System Interface, Ayalon highway, Tel-Aviv, Israel
The local speed over a road section around each sensor was measured and reported in real time in km per hour (see white squares). Each section's data was compared to the data provided by the road sensor (inductive loop) at that location.
Blue - TrafficSense Magenta - Loop Sensors
Figure 3 - TrafficSense Speed Comparisons With Road Sensors
In the following graph one can see a benchmark comparison of travel time measured by TrafficSense (a cellular based detection system from Cellint Traffic Solutions Ltd.), travel time deducted from road sensors (inductive loops) and actual drive test, conducted by the Ayalon Highway Co. (an Israeli DOT agency).
| Time of day: | 15:06 | 15:31 | 16:01 |
Figure 4 - TrafficSense Travel-time Comparisons with Test Drives
This graph demonstrates that during free flow, both systems (i.e. Inductive loops and TrafficSense) are providing accurate data. However, when delays and fluctuation in traffic starts, the travel time deducted from the road sensors is not accurate, while TrafficSense maintains high accuracy.
Figure 5 - Kansas City Project Area
This is another blind test conducted in Kansas City by the Kansas Department of Transportation. The TrafficSense data was compared to inductive loops which are part of the SCOUT system. The test was conducted in a control environment, and Cellint received the reference data of the pilot only after the pilot ended.
|
|
Figure 6 - Kansas City Project Results
Results from I435 in Kansas City, January 2006
 |
 |
Figure 7 - TrafficSense Project for the Israeli National Road Company
Interface Map and Results from Highway 1, Isreal
The local speed over a road section around each sensor was measured and reported in kilometers per hour. Each section's data was compared to the data provided by the road sensor (side view radars) at that location. Travel time is reported on section at right.
For more information please contact us.
