We Are the New Gremlins in the AI Engine

During the Covid pandemic, a healthcare helpline worker encountered callers misusing rapid lateral flow tests, highlighting the disconnect between technology design and real-world use. This issue, termed the “inadvertent misuse problem” (or “imp”), extends to general-purpose chatbots like ChatGPT, which many Americans now use for health self-diagnosis. Despite their advanced capabilities, a recent Oxford Internet Institute study found that participants struggled to use these chatbots effectively, achieving only 34.5% success in identifying health conditions, compared to a control group using search engines. While these tools have potential to enhance healthcare access, designers must consider users’ technological skills in their designs. Special attention is needed for vulnerable users seeking medical advice, as biases in existing therapy chatbots can lead to detrimental outcomes. More rigorous user testing in real-world contexts is essential to optimize these technologies, ensuring they serve their intended purpose effectively.

Source link