How to Assess AI Social Engineering Risk in 2026 | AI LLM Hacking Course Day 42 of 90

In Day 41, we looked at how I can chain multiple techniques together when testing AI systems. Today, I’m shifting the focus from the AI model to the people interacting with it. In Day 42, I’ll show you how attackers use AI to make phishing, vishing, impersonation and other social engineering attacks more convincing, personalised and scalable.

Then, in Day 43, we’ll move back toward the technical side and look at AI vulnerability research — how I approach finding previously unknown weaknesses in AI systems and how responsible disclosure works when those vulnerabilities are discovered.

How AI Amplifies Each Social Engineering Vector

When I assess a social engineering attack, I usually look at three things: quality, volume and capability. How convincing is the communication? How many people can the attacker reach? And what can the attacker do now that would have been difficult before? AI has changed all three at the same time, although the biggest change depends on the type of attack.

Take phishing first. This is where I see the biggest change in quality. Traditional advice tells us to look for obvious warning signs — grammatical mistakes, generic messages, strange wording or suspicious links. Those signals are becoming much less reliable. AI can produce polished messages, research a target using publicly available information and personalise the communication without requiring an attacker to write every message manually. The important change isn’t simply that AI writes better emails. It’s that highly personalised messages can now be produced at a scale that previously wasn’t practical.

Vishing shows a different kind of change. Here, capability is the key factor. Voice-cloning technology can reproduce aspects of a person’s voice from publicly available recordings such as interviews, podcasts, conference presentations or social media videos. I no longer have to assume that a familiar voice automatically proves someone’s identity. The voice may sound convincing while the person on the other end of the call is someone completely different.

This is why I teach you to separate identity from evidence. A convincing email doesn’t prove who sent it. A familiar voice doesn’t prove who is calling. And a message containing information that only an executive should know doesn’t automatically prove that the executive sent it. AI makes these signals easier to imitate, so our verification process has to rely on independent evidence rather than familiarity alone.

The same pattern appears in spear phishing and business email compromise, but here the biggest advantage is context. I can use AI to organise information about a target — their role, reporting structure, projects, suppliers and publicly visible relationships — and then use that context to make a message feel relevant. The danger isn’t simply a better-written email. It’s a message that appears to arrive at exactly the right moment and fits naturally into a real business process.

With deepfake impersonation, AI changes the trust signals I normally rely on. A video call, photograph or recorded message can appear to come from a real executive while the underlying media has been manipulated. I therefore teach a simple principle: visual familiarity is not identity verification. When a request involves money, credentials, sensitive information or an unusual change in process, I want an independent verification channel — not another piece of media supplied through the same communication.

AI-generated content at scale creates another problem. An attacker can produce large numbers of personalised messages, adapt the wording for different audiences and quickly iterate when a particular approach doesn’t work. That means defenders cannot rely entirely on manually identifying individual messages. I need controls that work at the process level — strong authentication, reporting mechanisms, transaction controls, verification procedures and monitoring for unusual behaviour.

Then there is social engineering through AI assistants. This is particularly interesting because the target may not be a person at all. An attacker could try to manipulate an AI assistant into treating untrusted instructions as legitimate, revealing information, taking an unauthorised action or influencing a human operator. When I assess an AI-enabled workflow, I therefore ask two questions: What can the assistant access? and What actions can it take? The more authority an assistant has, the more important instruction boundaries, permission checks and human approval become.

Finally, I look at multichannel social engineering. This is where several techniques are combined — perhaps an email establishes the story, a phone call reinforces it and a messaging application provides the final request. Each individual interaction may appear believable, but together they create a much stronger illusion of legitimacy. My defensive lesson here is simple: don’t verify each channel independently and assume that makes the overall story trustworthy. If all the channels ultimately trace back to the same unverified request, I still need an independent way to confirm the person’s identity and the requested action.

That is the real shift AI brings to social engineering. It isn’t just about making one phishing email or one fake voice more convincing. AI allows attackers to combine research, personalisation, content generation, impersonation and automation into a coordinated deception. So when I assess an organisation, I don’t ask only, “Can our employees spot a phishing email?” I ask, “Can our processes still verify a legitimate request when every familiar signal can be convincingly imitated?”

AI-Enhanced Spear Phishing Assessment

Let me break down what makes AI-enhanced spear phishing different. In the scenario from the hook, the attacker doesn’t start by writing an email. They start by building a picture of the target. LinkedIn can reveal the organisation structure and reporting relationships. The company website and press releases can reveal current projects and recent events. Executive interviews, earnings calls and conference presentations can provide examples of how a particular person communicates and what subjects they regularly discuss.

The important change is what happens next. AI can bring all of that publicly available information together and help generate a highly personalised message. The email can use the recipient’s real name, refer to a genuine project, include accurate context and imitate the communication style of the supposed sender. The attacker still needs to make decisions and validate the information, but AI dramatically reduces the amount of manual work required to produce convincing content.

So when I assess an organisation, I don’t make “Can our employees spot a phishing email?” the only question. A well-crafted AI-generated message may not contain the obvious spelling mistakes, generic wording or awkward language that traditional awareness training teaches people to look for.

Instead, I ask a more useful question: “What happens when a convincing email passes the human suspicion test?” That’s where process controls become critical. For example, a financial authorisation should have an independent verification step. If someone requests a large transfer by email, I want the recipient to call a trusted number already stored in the company’s directory or approved contact system — not a number included in the request itself.

This is the principle I want you to remember: don’t make your security depend entirely on detecting the deception. Build verification procedures that still work when the deception is convincing. AI can make the fake message better, but it doesn’t have to make a well-designed verification process fail.

Practical Verification Checklist

When I receive an unusual request involving money, credentials, sensitive data or a change to an established process, I don’t try to prove that the message is fake. I follow a verification routine that works even when the message looks completely legitimate.

• Pause before acting. I don’t let urgency decide the verification process. A request that says “immediately” still gets verified.
• Check the actual sender. I inspect the complete email address, domain and reply-to address rather than relying on the display name.
• Verify through a trusted channel. I contact the supposed sender using a phone number, messaging channel or directory entry that I already trust — never contact details supplied in the suspicious message.
• Confirm the request, not just the identity. I ask the person to independently confirm what they want done, the amount involved, the recipient and the reason for the request.
• Use a second person for high-risk actions. For large financial transfers, credential changes or sensitive disclosures, I follow the organisation’s dual-control or approval process rather than relying on one person’s confirmation.
• Check for process changes. If the request asks me to bypass normal approvals, use a new bank account, change payment details or skip an established procedure, I treat that as a verification trigger.
• Don’t trust voice or video alone. A familiar voice or face is useful context, but it isn’t sufficient proof of identity when the requested action carries significant risk.
• Report suspicious attempts. I preserve the original message, relevant headers and other evidence and report the attempt through the organisation’s established security channel.

My thumb rule: the more damaging the requested action, the stronger the verification should be. I don’t need to identify exactly how an attacker generated the email, cloned a voice or created a deepfake. I need a process that prevents an unverified request from becoming a real-world action.

My Five-Minute Response Sequence

When I receive a suspicious AI-enhanced request, I don’t spend five minutes trying to prove whether the message is AI-generated. I use those five minutes to prevent the request from turning into an incident. This is the response sequence I teach:

1. Minute 1 — Stop. I don’t click the link, open an attachment, transfer money, disclose credentials or reply to the request. If the message creates unusual urgency, I treat that urgency as a reason to slow down.
2. Minute 2 — Inspect. I check the sender address, reply-to information, domain, links and the actual request. I look for changes from the normal business process rather than relying only on grammar or spelling.
3. Minute 3 — Verify independently. I contact the supposed sender through a trusted channel that I already have. I don’t use the phone number, link or contact details supplied by the suspicious message.
4. Minute 4 — Apply the control. For a financial transfer, credential change or sensitive-data request, I follow the required approval or dual-control process. If the request cannot be independently verified, I don’t proceed.
5. Minute 5 — Report and preserve. I report the attempt through the organisation’s security process and preserve the original email, headers, messages and relevant call details. If I already interacted with the attacker, I report that immediately rather than trying to hide the mistake.

The key lesson: I don’t need to recognise a deepfake, identify the AI model behind a phishing email or prove that a voice was cloned. My first objective is to break the attacker’s chain of trust. Stop the action, verify independently, apply the required control and report what happened.

AI Vishing — Voice Cloning in Social Engineering

Vishing worries me because it attacks one of the strongest trust signals we have: recognising someone’s voice. When I hear the voice of a colleague, manager or family member, my brain naturally treats that familiarity as evidence of identity. AI voice cloning makes that assumption much less reliable.

This is why I like the idea of a pre-agreed verification phrase for high-risk situations. Think of it as a personal version of an organisational out-of-band verification procedure. If someone contacts me with an unusual or urgent request, I can use a verification method that an attacker cannot simply obtain by cloning a voice. At an organisational level, the equivalent control is straightforward: if a call requests an urgent financial action, I end the call and contact the person again using a trusted number from the corporate directory — not the number displayed on caller ID or provided during the call.

When I assess vishing resilience in an authorised engagement, I don’t need to clone a real executive’s voice. A safer approach is to use an approved simulated voice that resembles the type of communication employees might encounter and then measure what happens next. Does the employee follow the callback procedure? Do they request independent verification? Or does the urgency of the conversation cause them to bypass the control?

That last point is particularly important. Urgency override is often a bigger weakness than voice-cloning technology itself. An employee may know exactly what the policy says but still skip it because the caller says a payment must be completed in five minutes, an executive is waiting, or a business deal is at risk.

So my assessment question is simple: when a familiar voice asks for an unusual action under pressure, does our verification process still work? If the answer is yes, the organisation has a control that doesn’t depend on recognising whether a voice is real. If the answer is no, that’s the process weakness I want to fix.

Practical AI Vishing Verification Checklist

When I receive an unexpected call asking me to transfer money, reveal information, change credentials or bypass a normal process, I use this checklist before taking action:

• Pause the request. I don’t allow urgency, seniority or pressure from the caller to change the verification process.
• Don’t trust the voice alone. A familiar voice can be convincing, but voice recognition is not sufficient proof of identity for a high-risk request.
• End and call back. I terminate the call and contact the person through a trusted number already stored in the corporate directory or approved contact system.
• Verify the request independently. I confirm exactly what action is being requested, why it is required and whether it follows the normal business process.
• Use dual approval. For financial transfers, credential changes or sensitive disclosures, I involve the second authorised person required by company policy.
• Watch for control-bypass language. Statements such as “keep this confidential,” “don’t call anyone else,” or “we don’t have time for the normal process” are reasons to strengthen verification, not weaken it.
• Use a pre-agreed verification method. For particularly sensitive scenarios, I use an organisation-approved challenge, callback procedure or other independent verification mechanism that isn’t communicated during the suspicious call.
• Report the attempt. I preserve the relevant call details and report the incident through the established security channel, even when I stopped the attack successfully.

My rule: if the request is high-risk, I verify the request independently — not just the voice. A convincing voice can imitate a person; it cannot independently authorise a transaction or override the organisation’s approval process.

AI Assistant Manipulation

This is where I want you to think beyond traditional phishing. AI assistants built into tools such as email, collaboration platforms and productivity suites introduce a new layer of trust. The employee isn’t necessarily interacting directly with the attacker’s instructions. Instead, the attacker may try to influence the AI assistant that the employee already trusts.

Here’s the pattern I want you to understand. An attacker sends an email that looks legitimate but contains hidden or embedded instructions intended for an AI assistant. The employee then asks the assistant to summarise the message, analyse it or help prepare a response. The assistant processes the email content along with those instructions. If the assistant has access to the employee’s data or tools, the malicious instructions may influence what it recommends or attempts to do.

Imagine the assistant produces a perfectly normal-looking summary and suggests preparing a response or creating an action. The employee may approve it because the result appears routine. The important question is: where did that instruction actually originate? If the assistant treated untrusted email content as instructions, the attacker may have influenced the workflow without ever persuading the employee directly.

This is essentially the indirect prompt injection problem we studied earlier in the course, but now it is connected to an enterprise productivity environment. The risk becomes much greater when an AI assistant can read sensitive information, access internal systems or perform actions using the employee’s permissions.

When I assess an AI assistant, I therefore don’t stop at asking whether the model can recognise malicious instructions. I also ask: What data can it access? What actions can it perform? Which actions require human approval? Can untrusted content influence those actions? Those questions help me determine the potential impact even when the assistant’s underlying model behaves exactly as designed.

My defensive principle here is simple: content from an email, document or webpage should never automatically become an instruction with the same authority as a trusted user command. The assistant needs clear trust boundaries, least-privilege access, confirmation for consequential actions and controls that separate untrusted content from instructions.

Deepfake Business Email Compromise

This is where I want you to stop thinking about BEC as simply a fake email from the CEO. Modern BEC can combine several impersonation techniques into one believable business conversation. The attacker may begin with an email, move the conversation to a phone call or messaging application, and then use synthetic audio or video to make the supposed executive appear even more legitimate.

The FBI has documented BEC scenarios where criminals used virtual meeting platforms to impersonate executives, including situations involving a still image, deepfake audio or a combination of compromised email and video-conferencing accounts to persuade employees to initiate financial transfers.

The important lesson for me is that each additional channel can strengthen the deception without independently proving identity. An employee might receive an email that appears to come from the CFO, receive a follow-up call from someone using the CFO’s voice, and then join a video meeting where the person appears to be the CFO. Everything seems to agree — but the evidence may ultimately come from the same attacker-controlled chain.

How the Attack Works

I break a deepfake BEC scenario into five stages:

1. Research: The attacker learns who has authority over payments, who reports to whom, which projects are active and what communication style the executives use.
2. Initial contact: A convincing email, message or other communication establishes the business context. AI can help generate polished and highly personalised content.
3. Trust escalation: The attacker introduces another communication channel — perhaps a phone call, voice message or video meeting — to make the identity appear more credible.
4. Urgent action: The supposed executive requests a payment, change to bank details, disclosure of information or another consequential action.
5. Verification bypass: The attacker attempts to prevent independent verification by creating urgency, confidentiality requirements or claims that the executive is unavailable through normal channels.

The last stage is the one I pay particular attention to during an assessment. A technically convincing deepfake is not necessarily enough to complete the fraud. The attacker still needs the organisation’s process to accept the request. If the company requires independent verification before a large transfer, the deepfake has to overcome that control as well.

Why Traditional Detection Is Not Enough

I don’t want employees trying to become forensic experts during a five-minute payment request. The FBI itself warns that AI-generated content has become difficult to identify reliably and recommends independent verification rather than relying only on visual or audio clues.

That changes the defensive question. Instead of asking, “Can I tell whether this video is fake?”, I ask, “What independent evidence would I require before taking this action?”

For example, a CFO appearing on a video call may genuinely look and sound like the CFO. That still doesn’t independently authorise a wire transfer. I want the payment request verified through an established corporate procedure, such as a trusted callback, an approved second approver or another control defined by the organisation’s financial process.

Deepfake BEC Verification Checklist

• Don’t rely on the face or voice. Treat synthetic media as a possibility whenever the requested action is high-risk.
• Check the original communication channel. Verify the sender address, domain, phone number and account independently.
• Use a trusted callback. Contact the executive using a number already stored in the corporate directory — never a number supplied during the suspicious interaction.
• Confirm the transaction independently. Verify the beneficiary, amount, purpose and account details through the normal financial-control process.
• Require dual approval. High-value or unusual transactions should not depend on one person’s confirmation.
• Challenge unusual process changes. A request to bypass normal approval, keep the transaction secret or avoid contacting another employee should trigger additional verification.
• Preserve evidence. Keep the original email, message, call information, meeting details and transaction information if an attempt occurs.

The Assessment I Want You to Run

In an authorised assessment, I don’t need to create a convincing deepfake of a real executive. I can test the organisation’s control using a clearly approved simulation that measures whether employees follow the verification process when presented with a realistic high-pressure scenario.

I measure four things: recognition, verification, escalation and control adherence. Did the employee notice something unusual? Did they independently verify the request? Did they know where to report it? Most importantly, did the organisation’s financial controls prevent the simulated request from becoming an unauthorised action?

My main lesson: I don’t want a security programme that depends on employees detecting perfect deepfakes. I want a process that assumes impersonation is possible and makes independent verification mandatory before high-impact actions. AI can imitate the person, but it should not be able to imitate the organisation’s approval process.

Updating Awareness Training for AI-Quality Attacks

One thing I want you to rethink is traditional phishing awareness training. A lot of training still teaches employees to look for poor grammar, generic greetings, strange wording, suspicious domains or obviously unrealistic stories. Those are useful signals, but they are becoming less reliable as AI-generated attacks become more polished.

If an employee has been trained to think, “The email looks professional, so it is probably legitimate,” we have created a problem. AI can produce professional-looking communication very easily. The goal of modern awareness training should therefore be to teach employees what to do, not just what to spot.

That’s why I teach a verification-first approach. When an email asks for money, credentials, sensitive information or a change to an established process, I don’t make the decision based solely on how convincing the message looks. I verify the request through an independent channel that I initiate myself.

For example, if an email supposedly comes from the CFO asking me to make an urgent payment, I don’t call the number included in the email. I use the trusted corporate directory, an established contact number or the organisation’s approved verification process. If the real CFO confirms that they sent the request, I continue through the normal approval process. If they say they didn’t, the attempted attack has been stopped regardless of how convincing the original email was.

This doesn’t mean detection becomes useless. I still want employees to recognise suspicious links, unexpected attachments, unusual sender addresses and other warning signs. But I don’t want those signals to be the only defence. Detection helps me notice the problem; verification prevents the problem from becoming an incident.

That is the shift I want you to take away from this section: don’t train people to identify every possible AI-generated attack. Train them to follow a verification process that remains effective even when the attacker gets everything else right.

Simple Employee Verification Checklist

I keep the employee version simple. When a message asks me to do something sensitive, I use this five-step check before I act:

1. STOP. Don’t click, pay, share credentials or change account details just because the request sounds urgent.
2. CHECK. Look at the sender, request and context. Ask yourself: is this normal for this person, process and situation?
3. VERIFY. Contact the person through a trusted channel you already have. Don’t use the phone number, link or contact details provided in the suspicious message.
4. CONFIRM. For money, credentials or sensitive information, confirm exactly what is being requested and follow the normal approval process.
5. REPORT. If something doesn’t feel right, report it through your organisation’s security process. If you already clicked or responded, report it immediately — don’t wait.

Easy rule to remember: Stop → Check → Verify → Confirm → Report. I don’t need to prove that a message is AI-generated. I just need to make sure the request is independently verified before I act.

Emergency Response — If You Already Acted

Sometimes the warning signs only become obvious after I’ve already clicked, replied or approved something. The most important rule at that point is: don’t hide the mistake and don’t wait. Fast reporting can give the security team time to contain the damage.

1. STOP. Don’t continue the conversation or perform any additional action requested by the attacker.
2. REPORT IMMEDIATELY. Contact the organisation’s security or incident-response team using the established internal channel. If money was involved, notify the appropriate finance or fraud team immediately as well.
3. SECURE YOUR ACCOUNT. If you entered credentials into a suspicious site, follow your organisation’s account-compromise procedure and change the affected credentials through the legitimate service.
4. DON’T DELETE EVIDENCE. Preserve the original email, messages, URLs, screenshots, call details and transaction information. Don’t forward suspicious content unnecessarily.
5. DOCUMENT WHAT HAPPENED. Record what you clicked, what information you provided, when it happened and what actions you took afterward. Accurate timing helps responders investigate and contain the incident.

My emergency rule is simple: Stop → Report → Secure → Preserve → Document. Making a mistake is recoverable. Delaying the report because you’re embarrassed or unsure can give the attacker more time to act.