By this point, we’d narrowed down the affected users to a single email client - Yahoo Mail, which is where we got suspicious. Had Yahoo Mail introduced any features lately that might be causing this…?

As it turns out, yes, yes they had. A quick Google search revealed that a few months ago Yahoo jumped on the AI craze with the launch of ”AI-generated, one-line email summaries”.

At this point, the penny dropped. Just like Apple AI generating fake news summaries, Yahoo AI was hallucinating the fake winner messages, presumably as a result of training their model on our old emails. Worse, they were putting an untrustworthy AI summary in the exact place that users expect to see an email subject, with no mention of it being AI-generated 🤯