EEM — Embedded Event Manager Scripting
Every network engineer has experienced the same situation: an interface flaps at 2 AM, a CPU spike causes packet drops, or a BGP peer resets — and by the time anyone is alerted and investigates, the transient event is gone and the evidence has rolled out of the syslog buffer. Embedded Event Manager (EEM) is Cisco IOS's built-in automation engine that eliminates this problem. EEM allows you to define policy rules directly on the router — when a specified event occurs, IOS automatically executes a set of actions without any human intervention, without an external NMS, and without waiting for someone to notice an alert. For the broader automation landscape see Network Automation Overview.
EEM policies are called applets — small, self-contained event-action rules written directly in IOS global configuration. An applet has two parts: an event clause that defines what triggers it (a syslog message pattern, an interface state change, a CLI command being entered, a CPU threshold, an IP SLA state change, or a timer), and one or more action clauses that define what happens in response (send a syslog message, execute CLI commands, send an email, reload the device, or capture show command output to a file).
Before starting this lab, ensure you understand syslog message formats at Syslog Severity Levels and syslog server configuration at Syslog Configuration. For IP SLA probes that EEM can monitor as event triggers, see IP SLA Configuration & Tracking. For SSH access used by EEM to send alerts, see SSH Configuration.
1. EEM Architecture — Core Concepts
EEM Components
| Component | Description | IOS Keyword |
|---|---|---|
| Event Detector | Monitors the IOS subsystem for a specific condition — syslog message pattern, interface state, CPU level, timer expiry, CLI command, IP SLA state, SNMP OID value | event [detector-type] |
| Policy (Applet) | The complete event-action rule. Named by the administrator. Stored in running-config and survives reload when saved. An applet can have one event and multiple ordered actions | event manager applet [name] |
| Action | What the router does when the event fires — send syslog, run CLI commands, send email, set a variable, evaluate an expression, or call another applet | action [label] [action-type] |
| Environment Variable | A global key-value variable available to all EEM applets — used to store email recipients, threshold values, or frequently changed parameters without editing the applet itself | event manager environment [name] [value] |
| Action Label | A number (or string) that defines the execution order of actions within an applet. Actions execute in ascending label order. Labels are typically 1.0, 2.0, 3.0 — decimals allow inserting actions between existing ones | action 1.0 ... |
EEM Event Detectors
| Event Detector | Trigger Condition | Common Use Case |
|---|---|---|
event syslog |
A syslog message matching a specified pattern (regex) is generated by IOS | Interface down/up, BGP peer reset, authentication failure, any IOS-generated log message |
event interface |
An interface counter crosses a threshold (input errors, output drops, bandwidth utilisation) | Detect high error rates, CRC errors, or queue drops on a WAN interface |
event track |
An IP SLA tracking object changes state (Up to Down, or Down to Up) | Take action when a tracked WAN link or default route becomes unavailable |
event cli |
A specific CLI command is entered by a user in exec or config mode | Audit sensitive commands (no shut, write erase), alert on unauthorised changes, or block specific commands |
event timer |
A timer expires — periodic (every N seconds), absolute (specific date/time), or watchdog (fires unless reset) | Periodic configuration backup, scheduled reporting, periodic health-check commands |
event snmp |
An SNMP OID value crosses a threshold | CPU utilisation above 80%, memory free below threshold, interface errors exceed rate |
event ipsla |
An IP SLA operation returns a specific result (timeout, RTT threshold exceeded, jitter threshold exceeded) | Detect WAN latency degradation, trigger path switchover when probe fails |
event none |
No automatic trigger — applet runs only when manually triggered with event manager run [name] |
On-demand maintenance scripts, test applets, manually invoked diagnostics |
EEM Action Types
| Action Type | What It Does | Example |
|---|---|---|
action syslog |
Generates a custom syslog message at a specified severity level | Log "Interface down alert" at severity 2 (Critical) |
action cli command |
Executes a CLI command in exec or configuration mode as if typed by an operator | Run show interfaces, clear ip bgp, or any config command |
action mail |
Sends an email via a configured SMTP server with a custom subject and body | Email NOC team when an interface goes down |
action puts |
Writes a string to the console, a file, or a buffer — useful for logging applet output | Write "EEM triggered at [time]" to flash:/eem-log.txt |
action set |
Sets a local EEM variable to a value or expression result | Store the current interface status in a variable for conditional logic |
action info type |
Retrieves event-specific information — interface name, syslog message text, track state — and stores it in a variable | Capture which interface triggered the event, then use it in subsequent actions |
action reload |
Reloads the device — use with extreme caution | Automated recovery from a known-bad config state (rarely appropriate) |
EEM Applet Execution Flow
2. Lab Topology & Scenario
NetsTuts_R1 is an edge router with two WAN interfaces (Gi0/0 to ISP-A, Gi0/1 to ISP-B) and one LAN interface (Gi0/2). The NOC team needs automated alerts and diagnostic capture when WAN interfaces fail. Four EEM applets will be built in this lab:
3. Step 1 — Prerequisites and EEM Global Settings
Before writing applets, configure the global EEM prerequisites: an EEM user for CLI action execution, the SMTP server for email actions, and environment variables for reusable parameters.
NetsTuts_R1>en NetsTuts_R1#conf t Enter configuration commands, one per line. End with CNTL/Z. ! ══════════════════════════════════════════════════════════ ! EEM CLI Execution User ! Required so "action cli command" can run privileged commands ! ══════════════════════════════════════════════════════════ ! ── Create a local user for EEM to impersonate ──────────── NetsTuts_R1(config)#username eem-user privilege 15 secret EEM$ecret99 ! ── Tell EEM to use this user for cli action sessions ───── NetsTuts_R1(config)#event manager session cli username eem-user ! ══════════════════════════════════════════════════════════ ! SMTP Mail Server (for action mail) ! ══════════════════════════════════════════════════════════ NetsTuts_R1(config)#event manager environment _email_server 10.0.0.25 NetsTuts_R1(config)#event manager environment _email_from [email protected] NetsTuts_R1(config)#event manager environment _email_to [email protected] NetsTuts_R1(config)#event manager environment _email_cc [email protected] ! ══════════════════════════════════════════════════════════ ! EEM Environment Variables — reusable parameters ! Change these without editing each applet individually ! ══════════════════════════════════════════════════════════ NetsTuts_R1(config)#event manager environment _cpu_threshold 80 NetsTuts_R1(config)#event manager environment _hostname NetsTuts_R1 NetsTuts_R1(config)#event manager environment _log_dir flash:/eem-logs/
The EEM session CLI username is critical for any
applet that uses
action cli command. Without it, the
CLI action runs with minimal privilege and cannot execute
show commands, enter configuration mode, or clear
processes. The environment variables prefixed with
an underscore (a Cisco convention for EEM variables) centralise
values that appear in multiple applets — changing the NOC email
address only requires updating the environment variable, not
editing each applet. For user account configuration, see
Hostname,
Banner & Password Configuration.
4. Step 2 — Applet 1: Interface Down Syslog Alert
This applet fires whenever any interface transitions to a down state.
The event trigger is a syslog detector matching the
standard IOS message generated for an interface going down.
Actions send a custom syslog alert and execute
show interfaces to capture a snapshot at the moment
of failure.
NetsTuts_R1(config)#event manager applet INTF-DOWN-ALERT NetsTuts_R1(config-applet)# description "Alert and capture diag when any intf goes down" ! ── EVENT: match IOS syslog message for interface down ──── ! ── %LINK-3-UPDOWN: Interface [intf], changed state to down NetsTuts_R1(config-applet)# event syslog pattern "%LINK-3-UPDOWN.*down" NetsTuts_R1(config-applet)# maxrun 60 ! ── ACTION 1: Send custom syslog at Critical severity ───── NetsTuts_R1(config-applet)# action 1.0 syslog priority critical msg \ "EEM ALERT: Interface down event detected on $_hostname" ! ── ACTION 2: Get the syslog message that triggered the event NetsTuts_R1(config-applet)# action 2.0 info type routername NetsTuts_R1(config-applet)# action 2.1 set _rname "$_info_routername" ! ── ACTION 3: Run show interfaces brief — capture state ─── NetsTuts_R1(config-applet)# action 3.0 cli command "enable" NetsTuts_R1(config-applet)# action 3.1 cli command \ "show ip interface brief | redirect flash:/eem-logs/intf-brief.txt" ! ── ACTION 4: Run show interfaces for the triggering event─ NetsTuts_R1(config-applet)# action 4.0 cli command \ "show interfaces | redirect flash:/eem-logs/intf-detail.txt" ! ── ACTION 5: Log completion message ────────────────────── NetsTuts_R1(config-applet)# action 5.0 syslog priority informational \ msg "EEM INTF-DOWN-ALERT: diagnostic files saved to flash:/eem-logs/" NetsTuts_R1(config-applet)#exit
The
event syslog pattern uses a regular expression
matching the standard IOS link-state change message. The pattern
%LINK-3-UPDOWN.*down matches the facility
(LINK), severity (3), mnemonic
(UPDOWN), and the word down at the end —
it fires on any interface going down but not on interfaces coming
back up (which would say up). The maxrun 60
sets the maximum applet runtime to 60 seconds — if actions take
longer than this, EEM terminates the applet. The | redirect
pipe writes CLI output to a file — useful for post-mortem analysis
of what the router's state was at the exact moment of the fault.
See show interfaces for the
output captured by action 4.0.
For syslog message formats and severity levels, see
Syslog Severity Levels.
Understanding the Syslog Pattern Match
IOS syslog message format:
%FACILITY-SEVERITY-MNEMONIC: message text
Example messages that trigger INTF-DOWN-ALERT:
%LINK-3-UPDOWN: Interface GigabitEthernet0/0, changed state to down
%LINK-3-UPDOWN: Interface GigabitEthernet0/1, changed state to down
%LINK-3-UPDOWN: Interface Serial0/0/0, changed state to down
Pattern: "%LINK-3-UPDOWN.*down"
%LINK-3-UPDOWN ← literal match of facility-severity-mnemonic
.* ← any characters (interface name varies)
down ← ends with "down" — not "up"
Messages that do NOT trigger (interface coming back up):
%LINK-3-UPDOWN: Interface GigabitEthernet0/0, changed state to up
^^^ "up" not "down"
More specific pattern (only Gi0/0):
"%LINK-3-UPDOWN: Interface GigabitEthernet0/0.*down"
Related messages to consider:
%LINEPROTO-5-UPDOWN: Line protocol on Interface ... changed state to down
↑ Layer 3 protocol down (after Layer 1 is already down)
Use severity 5 and LINEPROTO mnemonic for Layer 3 events
5. Step 3 — Applet 2: Interface Down Email Notification
This applet sends an email to the NOC team when the primary WAN interface (Gi0/0) specifically goes down. It uses EEM environment variables for the mail parameters — no email address is hard-coded in the applet itself.
NetsTuts_R1(config)#event manager applet INTF-DOWN-EMAIL NetsTuts_R1(config-applet)# description "Email NOC when primary WAN Gi0/0 goes down" ! ── EVENT: match only Gi0/0 going down ─────────────────── NetsTuts_R1(config-applet)# event syslog pattern \ "%LINK-3-UPDOWN: Interface GigabitEthernet0/0.*down" NetsTuts_R1(config-applet)# maxrun 30 ! ── ACTION 1: Syslog alert ──────────────────────────────── NetsTuts_R1(config-applet)# action 1.0 syslog priority critical \ msg "EEM EMAIL: Primary WAN Gi0/0 DOWN — alerting NOC via email" ! ── ACTION 2: Capture current time for email body ───────── NetsTuts_R1(config-applet)# action 2.0 info type snmp getid oid sysUpTime.0 NetsTuts_R1(config-applet)# action 2.1 set _uptime "$_info_snmp_value" ! ── ACTION 3: Send email using environment variables ────── NetsTuts_R1(config-applet)# action 3.0 mail server "$_email_server" \ to "$_email_to" \ from "$_email_from" \ cc "$_email_cc" \ subject "ALERT: Primary WAN Gi0/0 DOWN on $_hostname" \ body "Router $_hostname: Primary WAN interface GigabitEthernet0/0 \ has gone DOWN. Immediate investigation required. \ Router uptime: $_uptime. \ Check syslog server for full event details." ! ── ACTION 4: Log that email was sent ───────────────────── NetsTuts_R1(config-applet)# action 4.0 syslog priority informational \ msg "EEM EMAIL: Alert email sent to $_email_to for Gi0/0 down event" NetsTuts_R1(config-applet)#exit
The
action mail command requires the IOS device to have
IP connectivity to the SMTP server (_email_server).
The SMTP server must accept unauthenticated relay from the router's
source IP, or SMTP authentication credentials must be configured.
In most enterprise environments, a local relay (Postfix, Exchange
relay connector, or a syslog server with email forwarding) is easier
to configure than sending directly to an external SMTP server. Note
that the email server variable ($_email_server) and
recipient variables all reference the global environment variables
set in Step 1 — changing the NOC email address means updating only
the environment variable, not editing this applet. For syslog
server setup, see Syslog
Server Configuration.
6. Step 4 — Applet 3: CPU Threshold Alert
This applet uses the snmp event detector to monitor
the router's CPU utilisation OID. When the 5-second CPU average
exceeds 80%, it logs an alert and captures show processes cpu
sorted to identify the offending process.
NetsTuts_R1(config)#event manager applet CPU-THRESHOLD-ALERT NetsTuts_R1(config-applet)# description "Alert when 5s CPU exceeds 80 percent" ! ── EVENT: SNMP OID for 5-second CPU average ───────────── ! ── OID 1.3.6.1.4.1.9.2.1.56 = Cisco 5-second CPU % NetsTuts_R1(config-applet)# event snmp oid 1.3.6.1.4.1.9.2.1.56 \ get-type exact \ entry-op gt entry-val 80 \ poll-interval 10 NetsTuts_R1(config-applet)# maxrun 60 ! ── ACTION 1: Log critical syslog ───────────────────────── NetsTuts_R1(config-applet)# action 1.0 syslog priority critical \ msg "EEM CPU-ALERT: CPU utilisation exceeded 80% on $_hostname" ! ── ACTION 2: Capture CPU process list ─────────────────── NetsTuts_R1(config-applet)# action 2.0 cli command "enable" NetsTuts_R1(config-applet)# action 2.1 cli command \ "show processes cpu sorted | head 30 | \ redirect flash:/eem-logs/cpu-high.txt" ! ── ACTION 3: Capture memory summary ───────────────────── NetsTuts_R1(config-applet)# action 3.0 cli command \ "show processes memory sorted | head 20 | \ redirect flash:/eem-logs/mem-high.txt" ! ── ACTION 4: Log file location ─────────────────────────── NetsTuts_R1(config-applet)# action 4.0 syslog priority warning \ msg "EEM CPU-ALERT: Diagnostics saved to flash:/eem-logs/cpu-high.txt" NetsTuts_R1(config-applet)#exit
The
event snmp detector polls the SNMP OID every
poll-interval 10 seconds. entry-op gt
means "fire when the value is greater than" the
entry-val 80 threshold. SNMP must be enabled on the
router for the SNMP event detector to function — it uses the local
SNMP agent to read the OID, so a remote NMS is not required.
An alternative approach using the syslog detector:
IOS generates %SYS-4-CPUHOG messages when a process
runs for too long — matching this pattern with
event syslog pattern "%SYS-4-CPUHOG" works on devices
without SNMP configured. For SNMP configuration, see
SNMP v2c & v3
Configuration.
Alternative: CPU Alert Using Syslog Pattern (no SNMP required)
! ── Alternative CPU applet using syslog event detector ─── NetsTuts_R1(config)#event manager applet CPU-HOG-SYSLOG NetsTuts_R1(config-applet)# description "Alert on CPU hog process via syslog" NetsTuts_R1(config-applet)# event syslog pattern "%SYS-3-CPUHOG" NetsTuts_R1(config-applet)# maxrun 30 NetsTuts_R1(config-applet)# action 1.0 syslog priority critical \ msg "EEM: CPU HOG process detected on $_hostname — check processes" NetsTuts_R1(config-applet)# action 2.0 cli command "enable" NetsTuts_R1(config-applet)# action 2.1 cli command \ "show processes cpu sorted | redirect flash:/eem-logs/cpuhog.txt" NetsTuts_R1(config-applet)#exit
7. Step 5 — Applet 4: Configuration Change Audit
This applet uses the cli event detector to fire
whenever a user enters global configuration mode. It logs the
username and terminal line, then captures a configuration diff
showing exactly what changed. This is a lightweight change-audit
trail without requiring an external change management system.
NetsTuts_R1(config)#event manager applet CONFIG-CHANGE-AUDIT NetsTuts_R1(config-applet)# description "Log who entered config mode and capture diff" ! ── EVENT: fire when 'configure terminal' is entered ───── NetsTuts_R1(config-applet)# event cli pattern "conf.*t" \ sync no \ skip no NetsTuts_R1(config-applet)# maxrun 30 ! ── ACTION 1: Retrieve who entered the command ──────────── NetsTuts_R1(config-applet)# action 1.0 info type cli username NetsTuts_R1(config-applet)# action 1.1 set _user "$_info_cli_username" ! ── ACTION 2: Get the terminal line information ─────────── NetsTuts_R1(config-applet)# action 2.0 info type tty_name NetsTuts_R1(config-applet)# action 2.1 set _tty "$_info_tty_name" ! ── ACTION 3: Log the configuration entry event ────────── NetsTuts_R1(config-applet)# action 3.0 syslog priority warning \ msg "EEM AUDIT: User '$_user' entered config mode via $_tty on $_hostname" ! ── ACTION 4: Capture running config snapshot before changes NetsTuts_R1(config-applet)# action 4.0 cli command "enable" NetsTuts_R1(config-applet)# action 4.1 cli command \ "show running-config | redirect flash:/eem-logs/config-before.txt" ! ── ACTION 5: Log the file location ────────────────────── NetsTuts_R1(config-applet)# action 5.0 syslog priority informational \ msg "EEM AUDIT: Pre-change config snapshot saved to flash" NetsTuts_R1(config-applet)#exit
The
event cli pattern "conf.*t" matches any CLI
command beginning with conf and containing
t — which covers configure terminal and
common abbreviations (conf t, config t).
sync no means the applet runs asynchronously — the
user's configure terminal command proceeds immediately
without waiting for the applet to finish (important so the audit
does not delay the operator). skip no means the
original command still executes — the applet observes it without
blocking it. Use skip yes to prevent the
command from executing (useful to block specific dangerous commands).
For running-config management, see
show running-config
and Saving
& Managing Cisco Configurations.
8. Additional Practical Applet Examples
Applet 5: Periodic Configuration Backup (Timer Event)
! ── Back up running-config to TFTP every 24 hours ───────── NetsTuts_R1(config)#event manager applet DAILY-CONFIG-BACKUP NetsTuts_R1(config-applet)# description "Copy running-config to TFTP daily" ! ── EVENT: periodic timer — fires every 86400 seconds (24h) NetsTuts_R1(config-applet)# event timer periodic interval 86400 NetsTuts_R1(config-applet)# maxrun 60 ! ── ACTION 1: Log the backup start ─────────────────────── NetsTuts_R1(config-applet)# action 1.0 syslog priority informational \ msg "EEM BACKUP: Starting daily config backup on $_hostname" ! ── ACTION 2: Copy to TFTP server ───────────────────────── NetsTuts_R1(config-applet)# action 2.0 cli command "enable" NetsTuts_R1(config-applet)# action 2.1 cli command \ "copy running-config tftp://10.0.0.100/backups/NetsTuts_R1-backup.cfg" NetsTuts_R1(config-applet)# action 2.2 cli command "" ! ── Empty string confirms any TFTP dialog prompts ───────── ! ── ACTION 3: Log completion ────────────────────────────── NetsTuts_R1(config-applet)# action 3.0 syslog priority informational \ msg "EEM BACKUP: Daily config backup completed for $_hostname" NetsTuts_R1(config-applet)#exit
The
event timer periodic interval 86400 fires every
86,400 seconds (24 hours) from when the applet was last triggered.
On the first trigger after a reload, the timer starts from the
moment the applet was registered. The empty string action 2.2
cli command "" sends a carriage return to confirm any
interactive prompts the copy command might display
(such as destination filename confirmation). For configuration
backup and management, see
Saving &
Managing Cisco Configurations.
Applet 6: BGP Peer Down Alert
! ── Alert when any BGP peer goes down ───────────────────── NetsTuts_R1(config)#event manager applet BGP-PEER-DOWN NetsTuts_R1(config-applet)# description "Alert when BGP neighbour resets" ! ── EVENT: match BGP neighbour state change syslog ──────── ! ── %BGP-5-ADJCHANGE: neighbor x.x.x.x Down NetsTuts_R1(config-applet)# event syslog pattern \ "%BGP-5-ADJCHANGE.*Down" NetsTuts_R1(config-applet)# maxrun 30 ! ── ACTION 1: Critical syslog ───────────────────────────── NetsTuts_R1(config-applet)# action 1.0 syslog priority critical \ msg "EEM BGP-ALERT: BGP neighbour down on $_hostname — check immediately" ! ── ACTION 2: Capture BGP summary ──────────────────────── NetsTuts_R1(config-applet)# action 2.0 cli command "enable" NetsTuts_R1(config-applet)# action 2.1 cli command \ "show ip bgp summary | redirect flash:/eem-logs/bgp-down.txt" NetsTuts_R1(config-applet)# action 2.2 cli command \ "show ip bgp neighbors | redirect flash:/eem-logs/bgp-neighbors.txt" ! ── ACTION 3: Email NOC ─────────────────────────────────── NetsTuts_R1(config-applet)# action 3.0 mail server "$_email_server" \ to "$_email_to" \ from "$_email_from" \ subject "ALERT: BGP Peer DOWN on $_hostname" \ body "A BGP neighbour adjacency has gone down on $_hostname. \ BGP diagnostics saved to flash:/eem-logs/. Check BGP summary immediately." NetsTuts_R1(config-applet)#exit
Applet 7: IP SLA Tracking Object State Change
! ── React when IP SLA tracking object changes state ─────── ! ── Assumes: ip sla 1 configured to probe WAN gateway ! ── track 1 ip sla 1 reachability NetsTuts_R1(config)#event manager applet WAN-SLA-DOWN NetsTuts_R1(config-applet)# description "Alert when WAN SLA track object goes down" ! ── EVENT: track object 1 transitions to "down" state ──── NetsTuts_R1(config-applet)# event track 1 state down NetsTuts_R1(config-applet)# maxrun 30 ! ── ACTION 1: Log the WAN failure ───────────────────────── NetsTuts_R1(config-applet)# action 1.0 syslog priority critical \ msg "EEM WAN-SLA: Track object 1 DOWN — WAN probe failing on $_hostname" ! ── ACTION 2: Show IP SLA statistics ───────────────────── NetsTuts_R1(config-applet)# action 2.0 cli command "enable" NetsTuts_R1(config-applet)# action 2.1 cli command \ "show ip sla statistics 1 | redirect flash:/eem-logs/sla-down.txt" ! ── ACTION 3: Show routing table — verify failover ──────── NetsTuts_R1(config-applet)# action 3.0 cli command \ "show ip route | redirect flash:/eem-logs/route-at-failure.txt" NetsTuts_R1(config-applet)#exit
The
event track [number] state down detector integrates
directly with IOS Object Tracking — the same tracking objects used
by IP SLA, static route tracking, and HSRP. When the tracked object
transitions to down (probe fails), the applet fires immediately —
no syslog pattern matching delay. This is more precise than a
syslog pattern match for IP SLA events. A companion applet with
event track 1 state up can send an "all clear" alert
when the WAN recovers. For IP SLA and tracking configuration, see
IP SLA Configuration
& Tracking. For the combination of IP SLA + EEM for
automated WAN monitoring, see
IP SLA with Syslog Alerting.
9. Verification
show event manager policy registered — List All Applets
NetsTuts_R1#show event manager policy registered
No. Class Type Event Type Trap Time Registered
--- -------- ------ ------------------ ---- ----------------
1 applet system syslog Off Wed Oct 16 09:00:01 2024
Name : INTF-DOWN-ALERT
Event : pattern {%LINK-3-UPDOWN.*down}
Description : Alert and capture diag when any intf goes down
2 applet system syslog Off Wed Oct 16 09:00:02 2024
Name : INTF-DOWN-EMAIL
Event : pattern {%LINK-3-UPDOWN: Interface GigabitEthernet0/0.*down}
Description : Email NOC when primary WAN Gi0/0 goes down
3 applet system snmp Off Wed Oct 16 09:00:03 2024
Name : CPU-THRESHOLD-ALERT
Event : OID {1.3.6.1.4.1.9.2.1.56} op gt val {80}
Description : Alert when 5s CPU exceeds 80 percent
4 applet system cli Off Wed Oct 16 09:00:04 2024
Name : CONFIG-CHANGE-AUDIT
Event : pattern {conf.*t} sync {no} skip {no}
Description : Log who entered config mode and capture diff
5 applet system timer Off Wed Oct 16 09:00:05 2024
Name : DAILY-CONFIG-BACKUP
Event : periodic interval {86400}
Description : Copy running-config to TFTP daily
show event manager policy active — Applets Currently Running
NetsTuts_R1#show event manager policy active No. Applet name Class Event Type State --- ------------------- -------- ------------ ------- (none currently running) ! ── If an applet is mid-execution, it appears here ──────── ! ── with execution time and current action label ──────────
show event manager history events — Execution History
NetsTuts_R1#show event manager history events No. Time Event Type Name --- ------------------- ------------------ ------------------- 1 Wed Oct 16 14:32:01 syslog INTF-DOWN-ALERT 2 Wed Oct 16 14:32:01 syslog INTF-DOWN-EMAIL 3 Wed Oct 16 14:32:05 syslog INTF-DOWN-ALERT 4 Wed Oct 16 09:00:00 timer DAILY-CONFIG-BACKUP
The history shows applets 1 and 2 both fired at 14:32:01 — the same
syslog event triggered both applets simultaneously (both matched the
Gi0/0 down message). The history is stored in a ring buffer —
the oldest events are overwritten when the buffer fills. This is
the first place to check when an applet is not triggering — if
the expected event is not in the history, the event detector
pattern is not matching. Adjust the syslog pattern and re-test.
show event manager history traps — SNMP Trap History
NetsTuts_R1#show event manager history traps No. Time Name Status --- ------------------- -------------------- ------- 1 Wed Oct 16 14:32:01 INTF-DOWN-ALERT success 2 Wed Oct 16 14:32:01 INTF-DOWN-EMAIL success
show event manager environment — Verify Environment Variables
NetsTuts_R1#show event manager environment Name Value -------------------------- --------------------------------- _email_server 10.0.0.25 _email_from [email protected] _email_to [email protected] _email_cc [email protected] _cpu_threshold 80 _hostname NetsTuts_R1 _log_dir flash:/eem-logs/
Manually Trigger an Applet — Testing with event none
! ── Step 1: temporarily change the event to "none" for testing NetsTuts_R1(config)#event manager applet INTF-DOWN-ALERT NetsTuts_R1(config-applet)# event none NetsTuts_R1(config-applet)#exit ! ── Step 2: manually run the applet ─────────────────────── NetsTuts_R1#event manager run INTF-DOWN-ALERT % EEM policy successfully executed ! ── Step 3: verify files were created ──────────────────── NetsTuts_R1#dir flash:/eem-logs/ Directory of flash:/eem-logs/ 1 -rw- 3512 Oct 16 2024 14:35:22 intf-brief.txt 2 -rw- 45672 Oct 16 2024 14:35:22 intf-detail.txt ! ── Step 4: restore the original syslog event trigger ───── NetsTuts_R1(config)#event manager applet INTF-DOWN-ALERT NetsTuts_R1(config-applet)# event syslog pattern "%LINK-3-UPDOWN.*down" NetsTuts_R1(config-applet)#exit ! ── Step 5: verify logs show the manual execution ───────── NetsTuts_R1#show event manager history events No. Time Event Type Name --- ------------------- ------------------ ------------------- 1 Wed Oct 16 14:35:22 none INTF-DOWN-ALERT
Always test new applets with
event none before
attaching a live trigger — this prevents the applet from firing
on real events during development and allows you to execute
event manager run at a controlled time. Verify all
actions complete correctly, check that files are created in the
expected locations, and confirm syslog messages appear with
show logging before switching to the production
event trigger.
show logging — Confirm EEM Syslog Actions
NetsTuts_R1#show logging | include EEM Oct 16 14:32:01: %HA_EM-2-LOG: INTF-DOWN-ALERT: EEM ALERT: Interface down event detected on NetsTuts_R1 Oct 16 14:32:01: %HA_EM-6-LOG: INTF-DOWN-ALERT: EEM INTF-DOWN-ALERT: diagnostic files saved to flash:/eem-logs/ Oct 16 14:32:01: %HA_EM-2-LOG: INTF-DOWN-EMAIL: EEM EMAIL: Primary WAN Gi0/0 DOWN — alerting NOC via email Oct 16 14:32:02: %HA_EM-6-LOG: INTF-DOWN-EMAIL: EEM EMAIL: Alert email sent to [email protected]
EEM syslog actions appear in the log with the facility
%HA_EM (High Availability Embedded Manager) and the
severity level specified in the action syslog priority
command. The applet name is included in the log message — making
it easy to filter and identify EEM-generated messages with
show logging | include EEM or
show logging | include HA_EM. For log buffer
management and forwarding EEM syslog to a central server, see
show logging and
Syslog Server
Configuration.
Verification Command Summary
| Command | What It Shows | Primary Use |
|---|---|---|
show event manager policy registered |
All registered applets — name, event type, pattern/threshold, registration time | Primary verification — confirm applet is registered with correct event trigger |
show event manager policy active |
Applets currently executing — name, state, runtime, current action label | Debug a running applet — see if it is stuck on a slow action (e.g., copy to TFTP timing out) |
show event manager history events |
Historical applet executions — which applet, when, and what event type triggered it | Confirm an applet fired when expected, or diagnose why it did not fire |
show event manager history traps |
SNMP trap history from EEM — applet name and success/failure status | Confirm SNMP-triggered applets executed and check for execution errors |
show event manager environment |
All configured EEM environment variables and their current values | Verify environment variables are set correctly before testing email or file path actions |
event manager run [name] |
Manually triggers an applet with event none configured |
Test applet actions safely without waiting for a real event to occur |
show logging | include HA_EM |
All EEM-generated syslog messages in the log buffer | Confirm action syslog commands are executing and producing the expected log messages |
dir flash:/eem-logs/ |
Files written to flash by EEM action cli command with | redirect |
Confirm diagnostic capture files are being created with correct timestamps |
10. Troubleshooting EEM Applets
| Problem | Symptom | Cause | Fix |
|---|---|---|---|
| Applet never triggers despite the event occurring | show event manager history events shows no executions for the applet — the interface went down but INTF-DOWN-ALERT never fired |
The syslog pattern in the event syslog pattern clause does not match the actual syslog message IOS generates. IOS syslog message text varies slightly between IOS versions and platforms |
Capture the exact syslog message from show logging at the moment the event occurs. Compare it character-by-character with the pattern in the applet. The pattern is a regular expression — test it with event none and manually verify the regex matches the log line. Remember that the % character in the pattern matches the literal % in the syslog message. Use debug event manager to see real-time EEM pattern matching activity |
action cli command fails — access denied or incomplete output |
Applet fires but the CLI commands produce no output files, or show event manager history shows the applet executed with errors. dir flash:/eem-logs/ shows no files created |
The EEM session CLI user is not configured (event manager session cli username is missing), or the configured user does not have privilege level 15, or the flash directory does not exist |
Verify event manager session cli username eem-user is in the running config. Confirm the user has privilege 15 with show running-config | include username eem-user. Create the log directory manually first: mkdir flash:/eem-logs from the router CLI. Re-test with event manager run [name] |
| Email action never sends emails | Applet fires and logs show action 3.0 mail executing, but no email arrives at the NOC inbox |
The SMTP server (_email_server) is not reachable from the router, the SMTP server does not accept relay from the router's source IP, or the environment variable contains an incorrect IP address |
Verify SMTP server reachability: ping 10.0.0.25 from the router. Confirm the SMTP server allows relay from the router's IP. Check show event manager environment to verify _email_server is correct. Try a simple Telnet test: telnet 10.0.0.25 25 — if port 25 does not respond, SMTP is blocked. In many environments it is easier to use a syslog-to-email gateway instead of direct SMTP |
| Applet triggers too frequently — log flooding | The applet fires dozens of times per minute — syslog is flooded with EEM messages and flash storage fills up quickly with captured files | The triggering event is recurring very rapidly (e.g., an interface flapping up and down repeatedly) and the applet has no rate limiting or cooldown period | Add a rate limit to the event detector: event syslog pattern "..." maxrun 60 ratelimit 300 — the ratelimit 300 prevents the applet from firing again for 300 seconds (5 minutes) after its last execution. Alternatively use a event timer watchdog pattern to only fire once until manually reset. Investigate the root cause of the flapping interface separately |
| Applet actions execute out of order or skip actions | The applet fires but only some actions execute — for example, action 1.0 syslog works but action 2.0 cli command and 3.0 cli command do not run | The maxrun timer expired before all actions completed — IOS terminates the applet at the maxrun limit. CLI commands that capture large outputs (e.g., show processes cpu sorted | redirect) can take 5–15 seconds each |
Increase the maxrun value: for applets with multiple CLI capture commands, set maxrun 120 or higher. Verify the actual runtime with show event manager policy active during execution or review show event manager history events for timeout indicators. Remove non-critical actions to reduce runtime |
Config-audit applet blocks the operator's configure terminal |
When an engineer types conf t, the prompt hangs for 20–30 seconds before entering config mode — the EEM applet is running synchronously and blocking IOS |
The CLI event applet was configured with sync yes (or the default sync behaviour) — the applet runs synchronously, meaning IOS waits for the applet to complete before processing the matched command |
Change to asynchronous execution: event cli pattern "conf.*t" sync no skip no. With sync no, the operator's command executes immediately and the applet runs in the background. This is the correct setting for audit applets that should observe without blocking. Reserve sync yes only for applets that must complete before the matched command is processed (e.g., an applet that validates or modifies the command) |
Key Points & Exam Tips
- EEM (Embedded Event Manager) is an IOS-native automation engine that allows policy rules called applets to be defined directly on the router. Each applet has one
eventclause (trigger) and one or moreactionclauses (responses), executed in ascending label order. - The most common event detector is
event syslog pattern, which uses a regular expression to match IOS-generated syslog messages. The pattern must match the exact format of the syslog message including facility, severity, and mnemonic (e.g.,%LINK-3-UPDOWN.*down). - For
action cli commandto execute privileged commands,event manager session cli username [user]must be configured globally, and the referenced user must have privilege level 15. Without this, CLI actions silently fail or run with user-level privilege. - Action labels (1.0, 2.0, 3.0) determine execution order — actions run in ascending numerical order. Use decimal labels (1.5, 2.5) to insert actions between existing ones without renumbering. All actions within an applet run as a single atomic sequence unless
maxrunterminates the applet early. - Environment variables (
event manager environment [name] [value]) allow reusable parameters to be defined once and referenced in multiple applets with$_varnamesyntax. Change email addresses, thresholds, or server IPs without editing individual applets. - Always test new applets using
event noneandevent manager run [name]before attaching a live event trigger. This allows safe testing of all actions without waiting for a real fault event to occur. - The
ratelimit [seconds]option on the event clause prevents applet re-triggering for the specified duration after execution — essential for events that may recur rapidly (interface flapping, repeated syslog patterns) to prevent log flooding and flash exhaustion. - For CLI event applets,
sync no skip nomeans the applet runs asynchronously (does not block the operator's command) and the original command still executes.sync yes skip yesmeans the applet blocks the command and the original command is prevented from executing — use this pattern to block dangerous commands. show event manager policy registeredis the primary verification command — it lists all applets with their event type and pattern.show event manager history eventsshows execution history — if an expected event is missing from the history, the event detector pattern is not matching the actual syslog message.- On the exam: know the EEM applet structure (
event manager applet→event→action), the four most common event detectors (syslog, track, timer, cli), the key actions (syslog, cli command, mail), the purpose of environment variables, and thesync/skipoptions for CLI events. Also review debug commands and show logging as the primary tools for diagnosing why an EEM applet is or is not triggering.
Next Steps: For combining EEM with IP SLA probes
to generate automatic WAN monitoring alerts, see
IP SLA Configuration
& Tracking and
IP SLA with Syslog Alerting.
For sending EEM syslog output to a central logging server, see
Syslog Configuration and
Syslog Server Configuration.
For the SNMP infrastructure that backs the
event snmp
detector, see SNMP v2c
& v3 Configuration and
SNMP Traps.
For Python-based automation that can replace or supplement EEM for
more complex multi-device workflows, see
Python Netmiko Show
Commands.
For traffic capture at the time of an EEM-detected event, see
SPAN & RSPAN Port Mirroring.
For device hardening that complements EEM configuration-change
auditing, see
Login Security
& Brute-Force Protection.
TEST WHAT YOU LEARNED
What are the two mandatory components of every EEM applet, and what happens if an applet has actions but no event clause?
Correct answer is C. An EEM applet is defined with
event manager applet [name], and within it requires exactly one event statement (the trigger) and one or more action statements (what to do). The description is optional but highly recommended for documentation. If you write actions but forget the event clause, IOS accepts the partial configuration but the applet is not registered with the EEM scheduler — it never executes automatically. show event manager policy registered will not show it, or will show it without an event type. The event none configuration is a special deliberate pattern used specifically for testing: it tells EEM that this applet has no automatic trigger and can only be invoked manually with event manager run [name]. This allows safe incremental testing of all actions before attaching the production event trigger.An EEM applet is configured with event syslog pattern "%LINK-3-UPDOWN.*down". An interface goes down and IOS generates the message %LINK-3-UPDOWN: Interface GigabitEthernet0/0, changed state to down. The applet does not fire. What is the most likely cause?
Correct answer is A. The EEM syslog event detector works by monitoring the IOS syslog message stream. If the logging buffer is configured with a severity filter that excludes the message (e.g.,
logging buffered informational would filter out severity 3 errors since informational is severity 6, but actually severity 3 is errors which is more severe and would be included — the common issue is logging buffered notifications level 5 or higher which would exclude severity 3). More precisely: IOS logging levels work numerically — a lower number means higher severity. If logging buffered warnings (severity 4) is configured, severity 3 (errors) messages ARE included. But if no logging is configured or logging is completely disabled, EEM cannot see any syslog messages. The key diagnostic is show logging — verify the message actually appears in the buffer. If it does not appear in the buffer, EEM's syslog detector cannot match it. EEM syslog monitoring is independent of whether messages go to a remote syslog server, but it does depend on the local logging subsystem being active. Option B is incorrect — EEM syslog patterns fully support standard POSIX-style regular expressions including .*.Why is event manager session cli username eem-user required for applets that use action cli command, and what happens if it is missing?
Correct answer is D. EEM's
action cli command creates a virtual VTY-like session within IOS to execute commands. Unlike a human operator who authenticates via SSH or console and establishes a privilege level, EEM's virtual session has no inherent authentication context. The event manager session cli username [user] global command tells EEM which local user identity to use for this virtual session — the user's configured privilege level (set with privilege 15 in the username definition) determines what commands the EEM session can execute. Without it, EEM either fails to create the CLI session at all or creates one at privilege 1 (user EXEC only) which cannot run show processes cpu, copy running-config, or any privileged command. The failure is particularly insidious because the applet shows as "executed successfully" in the history (the action statement itself did not fail), but no output files are created and no commands actually ran with meaningful effect. The diagnostic is to check dir flash:/eem-logs/ for expected output files and compare with show event manager history events for execution confirmation.What is the difference between event cli pattern "..." sync yes skip yes and event cli pattern "..." sync no skip no, and give an example use case for each?
Correct answer is B. The
sync and skip parameters on CLI event detectors are among the most powerful and frequently misunderstood EEM options. Sync controls the execution relationship between the applet and the original command: sync yes means the CLI command is held pending while the applet runs to completion; sync no means the CLI command processes immediately and the applet runs concurrently in the background. Skip controls whether the original command actually executes after the applet runs: skip yes suppresses the original command (the user sees it being accepted but nothing happens); skip no allows it to execute normally. The combination sync yes skip yes effectively "intercepts" the command — useful for creating command blocks or confirmations. For example, an applet matching write erase with sync yes skip yes would log the unauthorised attempt and prevent the config from being erased. An applet with sync no skip no runs in pure observation mode — invisible to the operator, no delay, full command execution — appropriate for audit logging. Note: skip yes with sync no would be contradictory in most use cases since the command fires before the applet completes.An EEM applet with five action cli command statements fires but only the first two actions execute. show event manager history events shows the applet ran. What is the most likely cause?
Correct answer is C. The
maxrun parameter sets the maximum wall-clock time EEM allows for an applet to complete. If the applet is still running when maxrun expires, IOS terminates it immediately — any remaining actions are never executed. The default maxrun is 20 seconds on most IOS versions, which is often insufficient for applets that run multiple show commands and redirect their output to flash. show processes cpu sorted | redirect flash:/file.txt on a busy router can take 8–12 seconds alone. Five such commands would require 40–60 seconds. The fix is to increase maxrun: within the applet configuration, add maxrun 120 (or higher) to give the applet 2 minutes to complete. Verify the actual runtime needed by testing with event none and event manager run while watching show event manager policy active to see the elapsed time. Labels use decimal numbers (1.0, 2.0) which is valid IOS syntax — option B is incorrect.What is the purpose of EEM environment variables, and how do they differ from local variables set within an applet using action set?
Correct answer is D. Understanding the scope difference between environment variables and local variables is essential for writing maintainable EEM policies. Environment variables are the EEM equivalent of global constants: defined once in global config with
event manager environment _email_to [email protected], they are available to every applet and every applet execution. Change the NOC email address in one place and all applets immediately use the new value. They are also visible in the running-config and shown by show event manager environment. Local variables (action set) are runtime scratchpad variables: they are created during an applet execution, hold a value for use by subsequent actions in the same run, and are destroyed when the applet completes. They are typically used to store data retrieved from the triggering event (the interface name from a syslog message, the SNMP value that triggered the threshold) for use in subsequent actions within that same run. A key practical point: the info type actions retrieve event-specific data from the EEM event context and store it in built-in variables (like $_info_routername) — these are also local to the applet execution.How does the event track [number] state down detector differ from using event syslog pattern "%TRACK.*Down" for the same IP SLA failure event?
Correct answer is A. This distinction is important in production deployments where the timeliness of automated response matters. The IOS Object Tracking subsystem maintains the state of tracked objects (IP SLA reachability, interface line protocol, route existence) and notifies subscribers synchronously when state changes. EEM's
event track detector registers as a subscriber — it is called directly by the tracking subsystem at the instant the state changes, with no intermediate processing. The event syslog approach depends on IOS generating a log message about the track state change, the message passing through the syslog infrastructure (which may apply rate limiting, buffering, or severity filtering), and then the EEM syslog detector scanning the message for a pattern match. In normal operation this happens within seconds, but under high CPU load or with syslog rate limiting active, the syslog-based approach can be delayed by 30+ seconds or miss the event entirely. For critical events like WAN failover detection where automated rerouting or alerting must happen within seconds, event track is the correct choice. See IP SLA Configuration & Tracking for the tracking object setup that event track monitors.An EEM applet designed to send an email when a BGP peer goes down is triggering 40 times per hour during a BGP flap event, flooding the NOC inbox and filling flash with diagnostic files. How should the applet be modified to prevent this?
Correct answer is B. The
ratelimit option is the correct, purpose-built mechanism for preventing applet flooding. When ratelimit [seconds] is added to the event clause, EEM tracks the last execution time of the applet and ignores all matching events that occur within the ratelimit window after execution. The applet fires on the first matching event, then will not fire again until the ratelimit period expires — even if hundreds of matching events occur during that window. For BGP flapping at 40 events per hour, ratelimit 300 (5 minutes) would reduce execution to at most once per 5-minute period, cutting the flood from 40 to 12 per hour maximum. A ratelimit of 900 (15 minutes) would further reduce it to 4 per hour. The ratelimit does not prevent the alert from eventually being sent — it just ensures the NOC is not overwhelmed with repetitive notifications about the same ongoing event. Option C is incorrect — maxrun controls the maximum duration of a single applet execution, not the interval between executions. Using maxrun 3600 would mean a single applet run could take up to one hour, not that subsequent triggers are blocked.A security team wants an EEM applet that prevents any user from running write erase on production routers, logs the attempt with the username, and sends a syslog alert. Which event and skip configuration achieves this?
Correct answer is C. This is the only combination that correctly blocks the command AND allows the applet to perform audit actions. The key insight is the interaction between sync and skip:
sync yes means IOS holds the matched command in a pending state while the applet runs to completion. Only after the applet finishes does IOS decide whether to execute or skip the original command. skip yes tells IOS to skip (not execute) the original command after the applet completes. Together: the operator types write erase, IOS intercepts it via EEM, the applet logs the username and sends a syslog alert, and then IOS suppresses the original write erase — the config is never erased. Option B (sync no skip yes) is contradictory and does not work as intended: with sync no, the command executes concurrently with the applet — skip yes cannot retroactively prevent a command that has already been dispatched for execution. Option D (sync yes skip no) would log the attempt but then execute the command — the running-config would still be erased. This applet pattern is a practical and commonly deployed security control on Cisco devices where AAA is not fully implemented.show event manager history events shows an applet named INTF-DOWN-ALERT executing 12 times in the last hour. The interface actually went down only once. What are two possible causes and how are they identified?
Correct answer is D. Multiple triggers from a single real event is one of the most common EEM operational surprises. When a Gigabit Ethernet interface goes down, IOS does not generate just one syslog message — it typically generates a cascade:
%LINK-3-UPDOWN: Interface ... down (Layer 1), %LINEPROTO-5-UPDOWN: Line protocol ... down (Layer 3), possibly %OSPF-5-ADJCHG: Process ... neighbor down, %BGP-5-ADJCHANGE: neighbor ... Down, STP topology change notifications, and more. If the pattern %LINK-3-UPDOWN.*down is crafted carefully to match only the Layer 1 event, it fires once. But if the pattern were .*down (too broad), it would match all of these related messages and fire 10+ times for one physical link failure. The diagnostic is straightforward: capture show logging output during a test interface shutdown and count how many lines match the applet's pattern regex. Each matching line represents one potential applet execution. Then add ratelimit as a belt-and-suspenders protection regardless. Both root causes (overly broad pattern and missing ratelimit) are common in production and should both be addressed for a robust applet.Related Topics & Step-by-Step Tutorials
Related concepts and next steps:
- Syslog – Logging Overview — syslog events that EEM can trigger on
- Network Automation Overview — automation overview — where EEM fits
- Syslog Configuration
- IP SLA with Syslog Alerting
- Python Netmiko — Connect and Run Show Commands