吃瓜大本营

The had a story last week (expanding on an August report from the that I鈥檇 missed) that the Chicago police have created a list of the 鈥�400 most dangerous people in Chicago.鈥� The Trib reported on one fellow, who had no criminal arrests, expressing surprise over having received a visit from the police and being told he was on this list. A 17-year-old girl was also shocked when told she was on the list.

The database, according to the Verge, is based on historical crime information, disturbance calls, and suspicious person reports. The CPD鈥檚 list is heavily based on social network analysis (which is interesting considering the debates now swirling around the uses of metadata and the analysis such data enables). A sociologist whose work inspired the list, Andrew Papachristos, told the author of a Chicago Magazine piece (which on some of the theory behind the list): 鈥淚t鈥檚 not just about your friends and who you鈥檙e hanging out with, it鈥檚 actually the structure of these networks that matter.鈥�

The list was funded through a Justice Department grant known as 鈥淭wo Degrees of Association.鈥� (At least that鈥檚 one less hop than the NSA uses.)

I鈥檓 still consistently surprised how often things we worry about in the abstract actually show up in the real world. For years, privacy advocates have been warning about how databases might be mined by the authorities for information used to label, sort, and prejudge people. True, there are all too many precedents for this sort of thing, including the CAPPS II program proposed early in the Bush Administration, the nation鈥檚 terrorist watch lists, various police gang lists, and the Automated Targeting System. The TSA鈥檚 Pre-Check whitelist is also a cousin of this kind of program. All are based on using various information sources and grinding them through one or another logic engines to spit out a judgment about individuals and their supposed dangerousness or safeness as a human being. But still, this program amazes me in how starkly it replicates the kinds of things we have been warning about in many different contexts.

Just two weeks ago, for example, I was asked by several news outlets what we think about police officers using Google Glass. I told them that Glass is basically a body camera, and that the issues were the same as those outlined in our white paper on police use of that technology. The principal difference between Glass and the body cameras being marketed to police is that Glass can also display information. I said this shouldn鈥檛 be a problem鈥攗nless (I because of the slightly fanciful nature of this point) the police started using them with face recognition to display some kind of rating or warning for individuals who have been somehow determined to be untrustworthy.

鈥淥f course, that鈥檚 not a problem today,鈥� I said, 鈥渋t鈥檚 more of a futuristic concern.鈥�

Ha! Barely a week later, that scenario doesn鈥檛 seem so futuristic any more to me, especially at a time when some want to use face recognition to warn them when someone on a blacklist tries to enter a store or school. (True, Google doesn鈥檛 currently permit FaceRec apps on Glass, but it鈥檚 unclear how long that will last.)

Some further points and questions about Chicago鈥檚 heat list:

• The principal problem with flagging suspicious individuals in this way may be the risk of guilt by association. Although we don鈥檛 know how valid, accurate, and fair the algorithm is, it鈥檚 important to note that even if its measures were valid statistically鈥攖hat one particular individual really does have an increased risk of crime because of certain things about his or her life鈥攊t may still constitute guilt-by-association for a person who actually remains innocent. It is simply not fair for people to be subject to punishments and disadvantages because of the groups they belong to or what other people in similar circumstances tend to do. I keep going back to the of the man whose credit rating was lowered because the other customers of a store where he shopped had poor repayment histories.
• Why should the police restrict their hotlist to 400? Why not 4,000 or 40,000? In fact, why not give every citizen a rating, between 1 and 100 say, of how 鈥渞isky鈥� they might be? Then the police could program their Google Glass to display that score hovering above the head of every person who comes into their field of vision. This is a path it鈥檚 all too easy to see the police sliding down, and one we should not take even the first steps towards.
• Remember too the point that (as I made here) there are a vast number of laws on the books, many complicated and obscure, and anyone who is scrutinized closely enough by the authorities is far more likely to actually be found to have run afoul of some law than a person who isn鈥檛. In that respect inclusion on the list could become a self-fulfilling prophesy.
• Will the Chicago police carry out any kind of analysis to measure how effective this technique is? Will they look at the success of their predictions, search for any discriminatory effects, or attempt to find out whether these rankings become a self-fulfilling prophesy? The police often have little inclination to do any such things鈥攖o adopt rigorous criteria for measuring whether their new toys and gizmos are providing a good return on investment. Purely from an oversight point of view, every aspect of this program would ideally be made public so the world could scrutinize it鈥攃ertainly the algorithm. Privacy concerns, however, suggest that the names of individuals who are (quite possibly totally unfairly) flagged by these algorithms not be made public, nor any personal data that is being fed into the algorithms.
• A Chicago police commander is quoted as saying, 鈥淚f you end up on that list, there鈥檚 a reason you鈥檙e there.鈥� This framing at the heart of this approach: is it valid and accurate? Such circular logic is genuinely frightening when it comes from a police officer talking about matters of guilt and innocence.
• It鈥檚 true that there could be a fine line between laudable efforts to identify and help 鈥渁t-risk youth,鈥� and efforts to tag some people with labels that are used to discriminate and stigmatize. Research on the 鈥渆pidemiology of violence鈥� could be valuable if used as part of a public health approach to crime. But if it鈥檚 part of a criminal justice 鈥減re-crime鈥� approach, then that鈥檚 where the problems arise.

Overall, the key question is this: will being flagged by these systems lead to good things in a person鈥檚 life, like increased support, opportunities, and chances to escape crime鈥攐r bad things, such as surveillance and prejudicial encounters with the police? Unfortunately, there are all too many reasons to worry that this program will veer towards the worst nightmares of those who have been closely watching the growth of the data-based society.