In the U.S., police have started rounding up suspects based on defective image recognition from grainy surveillance video. Image recognition is known to work poorly on black faces, probably because they were mainly trained on images of white people.
When we roll out new technology, we of carefully explain to the users where and how it can be used. But if we can reasonably expect our users to ignore our admonitions, maybe we shouldn’t sell it at all.