Confidential document reveals key human role in gunshot tech

2 months ago EL PAÍS in English

Law enforcement personnel use an interactive electronic map for the ShotSpotter Dispatch program running within the Fusion Watch department at the Las Vegas Metropolitan Police Headquarters Wednesday, Jan. 13, 2021, in Las Vegas.

In more than 140 cities across the United States, ShotSpotter’s artificial intelligence algorithm and intricate network of microphones evaluate hundreds of thousands of sounds a year to determine if they are gunfire, generating data now being used in criminal cases nationwide.

But a confidential ShotSpotter document obtained by The Associated Press outlines something the company doesn’t always tout about its “precision policing system” — that human employees can quickly overrule and reverse the algorithm’s determinations, and are given broad discretion to decide if a sound is a gunshot, fireworks, thunder or something else.

Such reversals happen 10% of the time by a 2021 company account, which experts say could bring subjectivity into increasingly consequential decisions and conflict with one of the reasons AI is used in law-enforcement tools in the first place -- to lessen the role of all-too-fallible humans.

“I’ve listened to a lot of gunshot recordings — and it is not easy to do,” said Robert Maher, a leading national authority on gunshot detection at Montana State University who reviewed the ShotSpotter document. “Sometimes it is obviously a gunshot. Sometimes it is just a ping, ping, ping. ... and you can convince yourself it is a gunshot.”

Marked “WARNING: CONFIDENTIAL,” the 19-page operations document spells out how employees in ShotSpotter’s review centers should listen to recordings and assess the algorithm’s finding of likely gunfire based upon a series of factors that may require judgment calls, including whether the sound has the cadence of gunfire, whether the audio pattern looks like “a sideways Christmas tree” and if there is “100% certainty of gunfire in reviewer’s mind.”

ShotSpotter said in a statement to the AP that the human role is a positive check on the algorithm and the “plain-language” document reflects the high standards of accuracy its reviewers must meet.

“Our data, based on the review of millions of incidents, proves that human review adds value, accuracy and consistency to a review process that our customers—and many gunshot victims—depend on,” said Tom Chittum, the company’s vice president of analytics and forensic services.

Chittum added that the company’s expert witnesses have testified in 250 court cases in 22 states, and that its “97% aggregate accuracy rate for real-time detections across all customers” has been verified by an analytics firm the company commissioned.

Another part of the document underscores ShotSpotter’s longstanding emphasis on speed and decisiveness, and its commitment to classify sounds in less than a minute and alert local police and 911 dispatchers so they can send officers to the scene.

Titled “Adopting a New York State of Mind,” it refers to New York Police Department’s request of ShotSpotter to avoid posting alerts of sounds as “probable gunfire” — only definitive classifications as gunfire or non-gunfire.

“End result: It trains the reviewer to be decisive and accurate in their classification and attempts to remove a doubtful publication,” the document reads.

Experts say such guidance under tight time pressure could encourage ShotSpotter reviewers to err in favor of categorizing a sound as a gunshot, even if some evidence for it falls short, potentially boosting the numbers of false positives.

“You’re not giving your humans much time,” said Geoffrey Morrison, a voice-recognition scientist based in Britain who specializes in forensics processes. “And when humans are under great pressure, the possibility of mistakes is higher.”

ShotSpotter says it published 291,726 gunfire alerts to clients in 2021. That same year, in comments to AP appended to a previous story, ShotSpotter said more than 90% of the time its human reviewers agreed with the machine classification but the company invested in its team of reviewers “for the 10% of the time where they disagree with the machine.” ShotSpotter did not respond to questions on whether that ratio still holds true.

ShotSpotter’s operations document, which the company argued in court for more than a year was a trade secret, was recently released from a protective order in a Chicago court case in which police and prosecutors used ShotSpotter data as evidence in charging a Chicago grandfather with murder in 2020 for allegedly shooting a man inside his car. Michael Williams spent nearly a year in jail before a judge dismissed the case because of insufficient evidence.

Evidence in Williams’ pretrial hearings showed ShotSpotter’s algorithm initially classified a noise picked up by microphones as a firecracker, making that determination with 98% confidence. But a ShotSpotter reviewer who assessed the sound quickly relabeled it as a gunshot.

The Cook County Public Defender’s Office says the operations document was the only paperwork ShotSpotter sent in response to multiple subpoenas for any guidelines, manuals or other scientific protocols. The publicly traded company has long resisted calls to open its operations to independent scientific scrutiny.

Fremont, California-based ShotSpotter acknowledged to AP it has other “comprehensive training and operational materials” but deems them “confidential and trade secret.”

ShotSpotter installed its first sensors in Redwood City, California, in 1996, and for years relied solely on local 911 dispatchers and police to review each potential gunshot until adding its own human reviewers in 2011.

Paul Greene, a ShotSpotter employee who testifies frequently about the system, explained in a 2013 evidentiary hearing that staff reviewers addressed issues with a system that “has been known from time to time to give false positives” because “it doesn’t have an ear to listen.”

“Classification is the hardest element of the process,” Greene said in the hearing. “Simply because we do not have ... control over the environment in which the shots are fired.”

Greene added that the company likes to hire ex-military and former police officers familiar with firearms, as well as musicians because they “tend to have a more developed ear.” Their training includes listening to hundreds of audio samples of gunfire and even visits to rifle ranges to familiarize themselves with the characteristics of gun blasts.

As cities have weighed the system’s promise against its price tag -- which can reach $95,000 per square mile per year -- company employees have explained in detail how its acoustic sensors on utility poles and light posts pick up loud pops, booms or bangs and then filter the sounds through an algorithm that automatically classifies whether they’re gunfire or something else.

But until now, little has been known about the next step: how ShotSpotter’s human reviewers in Washington, D.C., and the San Francisco Bay area decide what is a gunshot versus any other noise, 24 hours a day.

“Listening to the audio downloads are important,” according to the document written by David Valdez, a former police officer and now-retired supervisor of one of ShotSpotter’s review centers. “Sometimes the audio is compelling for gunfire that they may override all other characteristics.”

One part of the decision-making that has changed since the document was written in 2021 is whether reviewers can consider if the algorithm had a “high confidence” the sound was a gunshot. ShotSpotter said the company stopped showing the algorithm’s confidence rating to reviewers in June 2022 “to prioritize other elements that are more highly correlated to accurate human-trained assessment.”

ShotSpotter CEO Ralph Clark has said that the system’s machine classifications are improved by its “real-world feedback loops from humans.”

However, a recent study found humans tend to overestimate their abilities to identify sounds.

The 2022 study published in the peer-reviewed journal Forensic Science International looked at how well human listeners identified voices compared to voice-recognition tools. It found all the human listeners performed worse than the voice system alone, saying the findings should lead to the elimination of human listeners in court cases whenever possible.

“Would that be the case with ShotSpotter? Would the ShotSpotter system plus the reviewer outperform the system alone?” asked Morrison, who was one of seven researchers who conducted the study.

“I don’t know. But ShotSpotter should do validation to demonstrate that.”

Sign up for our weekly newsletter to get more English-language news coverage from EL PAÍS USA Edition
Continue reading...

Read On "EL PAÍS in English"
More News On "EL PAÍS in English"
28 days ago - Alameda County landlords owed thousands in rent, call for an end to eviction moratorium 28 days ago - Monday Feb. 27 COVID-19 update: 4 deaths in Douglas County 28 days ago - State basketball preview: 6A, 5A tournaments return to Weber State’s Dee Events Center this week 28 days ago - One Wealth Advisors LLC invests in Enovix Co. (NASDAQ:ENVX) 28 days ago - Uncommon length makes Pleasant Valley’s 2-3 a no-scoring zone 28 days ago - Study: Back-to-back hurricanes likely to come more often 28 days ago - What’s Happening Vegas? – March 2023 28 days ago - Osceola County community events calendar for 03/01/2023 28 days ago - North Adams, East Clinton, Unioto still alive 28 days ago - North Korea holds rare meeting on farming amid food shortage 28 days ago - 'Dilbert,' Scott Adams lose distributor over racist remarks 28 days ago - Soap or phone call? Colo. lawmakers want to make prison phone calls free 28 days ago - EXPLAINER: Windstorm was likely a derecho. What is that? 28 days ago - What's Happening in Las Vegas for this Year's March Madness 28 days ago - Outsmarting humans just one step for AI video game players 28 days ago - 'Cocaine Bear' gets high with $23.1M, 'Ant-Man' sinks fast 28 days ago - 'Everything Everywhere All at Once' dominates at SAG Awards 28 days ago - Digital Transformation: The Revolutionary Impact of Technology in Africa 28 days ago - ShotSpotter (NASDAQ:SSTI) Price Target Increased to $44.00 by Analysts at Lake Street Capital 28 days ago - Season 3 of Outer Banks disappoints critics; watch only if you were a die-hard fan of earlier seasons, they suggest 29 days ago - Board Game and Card Game Market Size in 2023 with [ STATISTICS FIGURES] Future Development Status and Forecast up to 2029 29 days ago - Tabletop Gaming Market Size in 2023 NEW TECHNOLOGICAL DEVELOPMENT and Latest Innovation in Analytics Sector till 2029 29 days ago - Celona Offers Most Comprehensive Private 5G Solutions for U.S. and Foreign Markets 29 days ago - ShotSpotter, Inc. (NASDAQ:SSTI) to Post Q1 2023 Earnings of ($0.03) Per Share, Northland Capmk Forecasts 29 days ago - Asian shares track Wall Street decline on hot economic data 29 days ago - Final Nebraska high school swimming and diving season leaders 29 days ago - Girls BB: Saluting Section Champions 29 days ago - Tens of thousands protest Mexico electoral reforms 29 days ago - Third finals appearance the charm for Hortonville's Skebba; Stoffel makes history for Appleton North 29 days ago - Medical Blades Market Business Opportunities, Top Players and Forecast 2030 29 days ago - Central College Dutch Sports Update – 2/26/2023 29 days ago - Buhro takes individual crown as Oak Harbor earns sectional championship 1 month ago - Nebraska conservatives set sights on education takeover – Associated Press 1 month ago - Back-to-back: Minico successfully defends 4A state wrestling championship 1 month ago - Here are Saturday's high school sports results 1 month ago - Farewell, Fontana: NASCAR's last weekend at a racing gem 1 month ago - Kansas Democrats pick Repass as their new chair despite campaign baggage 1 month ago - Tesla’s Global Engineering HQ in Palo Alto — Opening Party Highlights (Pics, Videos, Quotes) 1 month ago - San Ann'as Pizza and Mexican celebrating 45th anniversary 1 month ago - L.A. on the Record: The Senate takes one more look at Garcetti 1 month ago - Dodge County real estate transfers 1 month ago - How UNL instructors are tackling the emergence of ChatGPT and other AI in higher education 1 month ago - Some Democratic-led states seek to bolster voter protections 1 month ago - Casey Vaughan: Only rain should go down a storm drain 1 month ago - Nebraska conservatives set sights on education takeover 1 month ago - West Michigan Conference basketball: Girls and boys roundup from Feb. 24, 2023 – CatchMark Sports 1 month ago - Jeff Yost: Look Upstream 1 month ago - Brokers Set Expectations for ShotSpotter, Inc.'s Q4 2023 Earnings (NASDAQ:SSTI) 1 month ago - Building affordable homes in Fremont 1 month ago - Local chef to open farm-to-table eatery in Fremont 1 month ago - Jeanna Wilcoxen Murder: Where Is Jeremiah Connelly Now? 1 month ago - More than 70 soldiers killed in Burkina Faso, extremists say 1 month ago - Clyde Council to consider citizens raising chickens in town 1 month ago - Wilhelm: More on Jacksons, Willow Hill and efforts to share insight into African American history 1 month ago - STATE HIGH SCHOOL WRESTLING TOURNAMENTS: Crowded at the top ... Trojans third, but well within striking distance in 5A tournament 1 month ago - Bulldog wrestlers have solid day at state 1 month ago - High school boys basketball: 6A/5A second round recap 1 month ago - Head-To-Head Analysis: Amprius Technologies (NYSE:AMPX) & Novanta (NASDAQ:NOVT) 1 month ago - Here are Friday's high school sports results 1 month ago - It’s Official: California Will Be Tesla’s Engineering & AI Headquarters
free geoip