https://www.netcraft.com/blog/large-language-models-are-falling-for-phishing-scams
ChatGPT and other AI chatbots are providing incorrect website URLs for major companies at alarming rates, creating new opportunities for cybercriminals to exploit unsuspecting users. According to research by threat intelligence firm Netcraft, GPT-4.1 family models deliver the correct web address only 66 percent of the time when asked for company login pages. The remaining responses include 29 percent that point to dead or suspended sites and 5 percent that direct users to legitimate but incorrect websites.
This accuracy problem presents a significant security risk as criminals can exploit the AI’s mistakes by purchasing unregistered domains that appear in chatbot responses and setting up phishing sites. Rob Duncan, Netcraft’s lead of threat research, explained that attackers simply need to identify what mistakes the AI models are making and then take advantage of those errors. The issue stems from AI systems focusing on word associations rather than evaluating actual URLs or website reputations, making them vulnerable to manipulation by sophisticated phishing operations.
Cybercriminals are already adapting their tactics to target AI-powered search behaviors, moving away from traditional search engine optimization to focus on poisoning chatbot results. Netcraft researchers discovered attackers creating elaborate fake ecosystems, including dozens of GitHub repositories, Q&A documents, tutorials, and fake social media accounts, all designed to trick AI systems into recommending malicious resources. This represents a significant evolution in phishing techniques, as criminals recognize that users increasingly rely on AI chatbots instead of conventional search engines while remaining unaware that these systems can provide incorrect information.