EEM — Embedded Event Manager Scripting

Every network engineer has experienced the same situation: an interface flaps at 2 AM, a CPU spike causes packet drops, or a BGP peer resets — and by the time anyone is alerted and investigates, the transient event is gone and the evidence has rolled out of the syslog buffer. Embedded Event Manager (EEM) is Cisco IOS's built-in automation engine that eliminates this problem. EEM allows you to define policy rules directly on the router — when a specified event occurs, IOS automatically executes a set of actions without any human intervention, without an external NMS, and without waiting for someone to notice an alert. For the broader automation landscape see Network Automation Overview.

EEM policies are called applets — small, self-contained event-action rules written directly in IOS global configuration. An applet has two parts: an event clause that defines what triggers it (a syslog message pattern, an interface state change, a CLI command being entered, a CPU threshold, an IP SLA state change, or a timer), and one or more action clauses that define what happens in response (send a syslog message, execute CLI commands, send an email, reload the device, or capture show command output to a file).

Before starting this lab, ensure you understand syslog message formats at Syslog Severity Levels and syslog server configuration at Syslog Configuration. For IP SLA probes that EEM can monitor as event triggers, see IP SLA Configuration & Tracking. For SSH access used by EEM to send alerts, see SSH Configuration.

1. EEM Architecture — Core Concepts

EEM Components

Component Description IOS Keyword
Event Detector Monitors the IOS subsystem for a specific condition — syslog message pattern, interface state, CPU level, timer expiry, CLI command, IP SLA state, SNMP OID value event [detector-type]
Policy (Applet) The complete event-action rule. Named by the administrator. Stored in running-config and survives reload when saved. An applet can have one event and multiple ordered actions event manager applet [name]
Action What the router does when the event fires — send syslog, run CLI commands, send email, set a variable, evaluate an expression, or call another applet action [label] [action-type]
Environment Variable A global key-value variable available to all EEM applets — used to store email recipients, threshold values, or frequently changed parameters without editing the applet itself event manager environment [name] [value]
Action Label A number (or string) that defines the execution order of actions within an applet. Actions execute in ascending label order. Labels are typically 1.0, 2.0, 3.0 — decimals allow inserting actions between existing ones action 1.0 ...

EEM Event Detectors

Event Detector Trigger Condition Common Use Case
event syslog A syslog message matching a specified pattern (regex) is generated by IOS Interface down/up, BGP peer reset, authentication failure, any IOS-generated log message
event interface An interface counter crosses a threshold (input errors, output drops, bandwidth utilisation) Detect high error rates, CRC errors, or queue drops on a WAN interface
event track An IP SLA tracking object changes state (Up to Down, or Down to Up) Take action when a tracked WAN link or default route becomes unavailable
event cli A specific CLI command is entered by a user in exec or config mode Audit sensitive commands (no shut, write erase), alert on unauthorised changes, or block specific commands
event timer A timer expires — periodic (every N seconds), absolute (specific date/time), or watchdog (fires unless reset) Periodic configuration backup, scheduled reporting, periodic health-check commands
event snmp An SNMP OID value crosses a threshold CPU utilisation above 80%, memory free below threshold, interface errors exceed rate
event ipsla An IP SLA operation returns a specific result (timeout, RTT threshold exceeded, jitter threshold exceeded) Detect WAN latency degradation, trigger path switchover when probe fails
event none No automatic trigger — applet runs only when manually triggered with event manager run [name] On-demand maintenance scripts, test applets, manually invoked diagnostics

EEM Action Types

Action Type What It Does Example
action syslog Generates a custom syslog message at a specified severity level Log "Interface down alert" at severity 2 (Critical)
action cli command Executes a CLI command in exec or configuration mode as if typed by an operator Run show interfaces, clear ip bgp, or any config command
action mail Sends an email via a configured SMTP server with a custom subject and body Email NOC team when an interface goes down
action puts Writes a string to the console, a file, or a buffer — useful for logging applet output Write "EEM triggered at [time]" to flash:/eem-log.txt
action set Sets a local EEM variable to a value or expression result Store the current interface status in a variable for conditional logic
action info type Retrieves event-specific information — interface name, syslog message text, track state — and stores it in a variable Capture which interface triggered the event, then use it in subsequent actions
action reload Reloads the device — use with extreme caution Automated recovery from a known-bad config state (rarely appropriate)

EEM Applet Execution Flow

  IOS generates event (e.g., interface goes down)
          │
          ▼
  EEM Event Detector scans all registered applets
  for a matching event clause
          │
          ▼ (match found)
  EEM Scheduler queues the applet for execution
          │
          ▼
  Actions execute in ascending label order:
    action 1.0 → action 2.0 → action 3.0 → ...
          │
          ▼
  Applet completes — EEM returns to monitoring state

  IMPORTANT NOTES:
  ─────────────────────────────────────────────────────────
  ● By default, applets run synchronously in the IOS main
    scheduler. Use "event [type] sync no" for async execution
    on heavy applets to avoid blocking IOS processes.
  ● "action cli command" runs in a separate Tcl shell context.
    It requires "event manager session cli username [user]"
    or a configured privilege level to execute privileged commands.
  ● Actions that modify configuration take effect immediately —
    there is no "review before apply" step. Test applets with
    "event none" before attaching a live event trigger.
  

2. Lab Topology & Scenario

NetsTuts_R1 is an edge router with two WAN interfaces (Gi0/0 to ISP-A, Gi0/1 to ISP-B) and one LAN interface (Gi0/2). The NOC team needs automated alerts and diagnostic capture when WAN interfaces fail. Four EEM applets will be built in this lab:

         ISP-A                  ISP-B
      203.0.113.1             198.51.100.1
           |                       |
        Gi0/0                   Gi0/1
           \                       /
            \                     /
             [  NetsTuts_R1  ]
              Gi0/2 (LAN)
              10.0.0.1/24

  EEM Applets to build:
  ┌───┬───────────────────────────────┬────────────────────────────────────┐
  │ # │ Applet Name                   │ Purpose                            │
  ├───┼───────────────────────────────┼────────────────────────────────────┤
  │ 1 │ INTF-DOWN-ALERT               │ Syslog + show commands when        │
  │   │                               │ any interface goes down            │
  ├───┼───────────────────────────────┼────────────────────────────────────┤
  │ 2 │ INTF-DOWN-EMAIL               │ Email NOC team when WAN interface  │
  │   │                               │ Gi0/0 goes down                    │
  ├───┼───────────────────────────────┼────────────────────────────────────┤
  │ 3 │ CPU-THRESHOLD-ALERT           │ Log and capture diagnostics when   │
  │   │                               │ CPU exceeds 80% for 5 seconds      │
  ├───┼───────────────────────────────┼────────────────────────────────────┤
  │ 4 │ CONFIG-CHANGE-AUDIT           │ Log who entered config mode and    │
  │   │                               │ capture a config diff              │
  └───┴───────────────────────────────┴────────────────────────────────────┘
  

3. Step 1 — Prerequisites and EEM Global Settings

Before writing applets, configure the global EEM prerequisites: an EEM user for CLI action execution, the SMTP server for email actions, and environment variables for reusable parameters.

NetsTuts_R1>en
NetsTuts_R1#conf t
Enter configuration commands, one per line.  End with CNTL/Z.

! ══════════════════════════════════════════════════════════
! EEM CLI Execution User
! Required so "action cli command" can run privileged commands
! ══════════════════════════════════════════════════════════
! ── Create a local user for EEM to impersonate ────────────
NetsTuts_R1(config)#username eem-user privilege 15 secret EEM$ecret99

! ── Tell EEM to use this user for cli action sessions ─────
NetsTuts_R1(config)#event manager session cli username eem-user

! ══════════════════════════════════════════════════════════
! SMTP Mail Server (for action mail)
! ══════════════════════════════════════════════════════════
NetsTuts_R1(config)#event manager environment _email_server  10.0.0.25
NetsTuts_R1(config)#event manager environment _email_from    [email protected]
NetsTuts_R1(config)#event manager environment _email_to      [email protected]
NetsTuts_R1(config)#event manager environment _email_cc      [email protected]

! ══════════════════════════════════════════════════════════
! EEM Environment Variables — reusable parameters
! Change these without editing each applet individually
! ══════════════════════════════════════════════════════════
NetsTuts_R1(config)#event manager environment _cpu_threshold  80
NetsTuts_R1(config)#event manager environment _hostname       NetsTuts_R1
NetsTuts_R1(config)#event manager environment _log_dir        flash:/eem-logs/
  
The EEM session CLI username is critical for any applet that uses action cli command. Without it, the CLI action runs with minimal privilege and cannot execute show commands, enter configuration mode, or clear processes. The environment variables prefixed with an underscore (a Cisco convention for EEM variables) centralise values that appear in multiple applets — changing the NOC email address only requires updating the environment variable, not editing each applet. For user account configuration, see Hostname, Banner & Password Configuration.

4. Step 2 — Applet 1: Interface Down Syslog Alert

This applet fires whenever any interface transitions to a down state. The event trigger is a syslog detector matching the standard IOS message generated for an interface going down. Actions send a custom syslog alert and execute show interfaces to capture a snapshot at the moment of failure.

NetsTuts_R1(config)#event manager applet INTF-DOWN-ALERT
NetsTuts_R1(config-applet)# description "Alert and capture diag when any intf goes down"

! ── EVENT: match IOS syslog message for interface down ────
! ── %LINK-3-UPDOWN: Interface [intf], changed state to down
NetsTuts_R1(config-applet)# event syslog pattern "%LINK-3-UPDOWN.*down"
NetsTuts_R1(config-applet)#  maxrun 60

! ── ACTION 1: Send custom syslog at Critical severity ─────
NetsTuts_R1(config-applet)# action 1.0 syslog priority critical msg \
   "EEM ALERT: Interface down event detected on $_hostname"

! ── ACTION 2: Get the syslog message that triggered the event
NetsTuts_R1(config-applet)# action 2.0 info type routername
NetsTuts_R1(config-applet)# action 2.1 set _rname "$_info_routername"

! ── ACTION 3: Run show interfaces brief — capture state ───
NetsTuts_R1(config-applet)# action 3.0 cli command "enable"
NetsTuts_R1(config-applet)# action 3.1 cli command \
   "show ip interface brief | redirect flash:/eem-logs/intf-brief.txt"

! ── ACTION 4: Run show interfaces for the triggering event─
NetsTuts_R1(config-applet)# action 4.0 cli command \
   "show interfaces | redirect flash:/eem-logs/intf-detail.txt"

! ── ACTION 5: Log completion message ──────────────────────
NetsTuts_R1(config-applet)# action 5.0 syslog priority informational \
   msg "EEM INTF-DOWN-ALERT: diagnostic files saved to flash:/eem-logs/"

NetsTuts_R1(config-applet)#exit
  
The event syslog pattern uses a regular expression matching the standard IOS link-state change message. The pattern %LINK-3-UPDOWN.*down matches the facility (LINK), severity (3), mnemonic (UPDOWN), and the word down at the end — it fires on any interface going down but not on interfaces coming back up (which would say up). The maxrun 60 sets the maximum applet runtime to 60 seconds — if actions take longer than this, EEM terminates the applet. The | redirect pipe writes CLI output to a file — useful for post-mortem analysis of what the router's state was at the exact moment of the fault. See show interfaces for the output captured by action 4.0. For syslog message formats and severity levels, see Syslog Severity Levels.

Understanding the Syslog Pattern Match

  IOS syslog message format:
  %FACILITY-SEVERITY-MNEMONIC: message text

  Example messages that trigger INTF-DOWN-ALERT:
  %LINK-3-UPDOWN: Interface GigabitEthernet0/0, changed state to down
  %LINK-3-UPDOWN: Interface GigabitEthernet0/1, changed state to down
  %LINK-3-UPDOWN: Interface Serial0/0/0, changed state to down

  Pattern: "%LINK-3-UPDOWN.*down"
    %LINK-3-UPDOWN  ← literal match of facility-severity-mnemonic
    .*              ← any characters (interface name varies)
    down            ← ends with "down" — not "up"

  Messages that do NOT trigger (interface coming back up):
  %LINK-3-UPDOWN: Interface GigabitEthernet0/0, changed state to up
                                                                  ^^^ "up" not "down"

  More specific pattern (only Gi0/0):
  "%LINK-3-UPDOWN: Interface GigabitEthernet0/0.*down"

  Related messages to consider:
  %LINEPROTO-5-UPDOWN: Line protocol on Interface ... changed state to down
  ↑ Layer 3 protocol down (after Layer 1 is already down)
  Use severity 5 and LINEPROTO mnemonic for Layer 3 events
  

5. Step 3 — Applet 2: Interface Down Email Notification

This applet sends an email to the NOC team when the primary WAN interface (Gi0/0) specifically goes down. It uses EEM environment variables for the mail parameters — no email address is hard-coded in the applet itself.

NetsTuts_R1(config)#event manager applet INTF-DOWN-EMAIL
NetsTuts_R1(config-applet)# description "Email NOC when primary WAN Gi0/0 goes down"

! ── EVENT: match only Gi0/0 going down ───────────────────
NetsTuts_R1(config-applet)# event syslog pattern \
   "%LINK-3-UPDOWN: Interface GigabitEthernet0/0.*down"
NetsTuts_R1(config-applet)#  maxrun 30

! ── ACTION 1: Syslog alert ────────────────────────────────
NetsTuts_R1(config-applet)# action 1.0 syslog priority critical \
   msg "EEM EMAIL: Primary WAN Gi0/0 DOWN — alerting NOC via email"

! ── ACTION 2: Capture current time for email body ─────────
NetsTuts_R1(config-applet)# action 2.0 info type snmp getid oid sysUpTime.0
NetsTuts_R1(config-applet)# action 2.1 set _uptime "$_info_snmp_value"

! ── ACTION 3: Send email using environment variables ──────
NetsTuts_R1(config-applet)# action 3.0 mail server "$_email_server" \
   to "$_email_to" \
   from "$_email_from" \
   cc "$_email_cc" \
   subject "ALERT: Primary WAN Gi0/0 DOWN on $_hostname" \
   body "Router $_hostname: Primary WAN interface GigabitEthernet0/0 \
   has gone DOWN. Immediate investigation required. \
   Router uptime: $_uptime. \
   Check syslog server for full event details."

! ── ACTION 4: Log that email was sent ─────────────────────
NetsTuts_R1(config-applet)# action 4.0 syslog priority informational \
   msg "EEM EMAIL: Alert email sent to $_email_to for Gi0/0 down event"

NetsTuts_R1(config-applet)#exit
  
The action mail command requires the IOS device to have IP connectivity to the SMTP server (_email_server). The SMTP server must accept unauthenticated relay from the router's source IP, or SMTP authentication credentials must be configured. In most enterprise environments, a local relay (Postfix, Exchange relay connector, or a syslog server with email forwarding) is easier to configure than sending directly to an external SMTP server. Note that the email server variable ($_email_server) and recipient variables all reference the global environment variables set in Step 1 — changing the NOC email address means updating only the environment variable, not editing this applet. For syslog server setup, see Syslog Server Configuration.

6. Step 4 — Applet 3: CPU Threshold Alert

This applet uses the snmp event detector to monitor the router's CPU utilisation OID. When the 5-second CPU average exceeds 80%, it logs an alert and captures show processes cpu sorted to identify the offending process.

NetsTuts_R1(config)#event manager applet CPU-THRESHOLD-ALERT
NetsTuts_R1(config-applet)# description "Alert when 5s CPU exceeds 80 percent"

! ── EVENT: SNMP OID for 5-second CPU average ─────────────
! ── OID 1.3.6.1.4.1.9.2.1.56 = Cisco 5-second CPU %
NetsTuts_R1(config-applet)# event snmp oid 1.3.6.1.4.1.9.2.1.56 \
   get-type exact \
   entry-op gt entry-val 80 \
   poll-interval 10

NetsTuts_R1(config-applet)#  maxrun 60

! ── ACTION 1: Log critical syslog ─────────────────────────
NetsTuts_R1(config-applet)# action 1.0 syslog priority critical \
   msg "EEM CPU-ALERT: CPU utilisation exceeded 80% on $_hostname"

! ── ACTION 2: Capture CPU process list ───────────────────
NetsTuts_R1(config-applet)# action 2.0 cli command "enable"
NetsTuts_R1(config-applet)# action 2.1 cli command \
   "show processes cpu sorted | head 30 | \
   redirect flash:/eem-logs/cpu-high.txt"

! ── ACTION 3: Capture memory summary ─────────────────────
NetsTuts_R1(config-applet)# action 3.0 cli command \
   "show processes memory sorted | head 20 | \
   redirect flash:/eem-logs/mem-high.txt"

! ── ACTION 4: Log file location ───────────────────────────
NetsTuts_R1(config-applet)# action 4.0 syslog priority warning \
   msg "EEM CPU-ALERT: Diagnostics saved to flash:/eem-logs/cpu-high.txt"

NetsTuts_R1(config-applet)#exit
  
The event snmp detector polls the SNMP OID every poll-interval 10 seconds. entry-op gt means "fire when the value is greater than" the entry-val 80 threshold. SNMP must be enabled on the router for the SNMP event detector to function — it uses the local SNMP agent to read the OID, so a remote NMS is not required. An alternative approach using the syslog detector: IOS generates %SYS-4-CPUHOG messages when a process runs for too long — matching this pattern with event syslog pattern "%SYS-4-CPUHOG" works on devices without SNMP configured. For SNMP configuration, see SNMP v2c & v3 Configuration.

Alternative: CPU Alert Using Syslog Pattern (no SNMP required)

! ── Alternative CPU applet using syslog event detector ───
NetsTuts_R1(config)#event manager applet CPU-HOG-SYSLOG
NetsTuts_R1(config-applet)# description "Alert on CPU hog process via syslog"
NetsTuts_R1(config-applet)# event syslog pattern "%SYS-3-CPUHOG"
NetsTuts_R1(config-applet)#  maxrun 30
NetsTuts_R1(config-applet)# action 1.0 syslog priority critical \
   msg "EEM: CPU HOG process detected on $_hostname — check processes"
NetsTuts_R1(config-applet)# action 2.0 cli command "enable"
NetsTuts_R1(config-applet)# action 2.1 cli command \
   "show processes cpu sorted | redirect flash:/eem-logs/cpuhog.txt"
NetsTuts_R1(config-applet)#exit
  

7. Step 5 — Applet 4: Configuration Change Audit

This applet uses the cli event detector to fire whenever a user enters global configuration mode. It logs the username and terminal line, then captures a configuration diff showing exactly what changed. This is a lightweight change-audit trail without requiring an external change management system.

NetsTuts_R1(config)#event manager applet CONFIG-CHANGE-AUDIT
NetsTuts_R1(config-applet)# description "Log who entered config mode and capture diff"

! ── EVENT: fire when 'configure terminal' is entered ─────
NetsTuts_R1(config-applet)# event cli pattern "conf.*t" \
   sync no \
   skip no

NetsTuts_R1(config-applet)#  maxrun 30

! ── ACTION 1: Retrieve who entered the command ────────────
NetsTuts_R1(config-applet)# action 1.0 info type cli username
NetsTuts_R1(config-applet)# action 1.1 set _user "$_info_cli_username"

! ── ACTION 2: Get the terminal line information ───────────
NetsTuts_R1(config-applet)# action 2.0 info type tty_name
NetsTuts_R1(config-applet)# action 2.1 set _tty "$_info_tty_name"

! ── ACTION 3: Log the configuration entry event ──────────
NetsTuts_R1(config-applet)# action 3.0 syslog priority warning \
   msg "EEM AUDIT: User '$_user' entered config mode via $_tty on $_hostname"

! ── ACTION 4: Capture running config snapshot before changes
NetsTuts_R1(config-applet)# action 4.0 cli command "enable"
NetsTuts_R1(config-applet)# action 4.1 cli command \
   "show running-config | redirect flash:/eem-logs/config-before.txt"

! ── ACTION 5: Log the file location ──────────────────────
NetsTuts_R1(config-applet)# action 5.0 syslog priority informational \
   msg "EEM AUDIT: Pre-change config snapshot saved to flash"

NetsTuts_R1(config-applet)#exit
  
The event cli pattern "conf.*t" matches any CLI command beginning with conf and containing t — which covers configure terminal and common abbreviations (conf t, config t). sync no means the applet runs asynchronously — the user's configure terminal command proceeds immediately without waiting for the applet to finish (important so the audit does not delay the operator). skip no means the original command still executes — the applet observes it without blocking it. Use skip yes to prevent the command from executing (useful to block specific dangerous commands). For running-config management, see show running-config and Saving & Managing Cisco Configurations.

8. Additional Practical Applet Examples

Applet 5: Periodic Configuration Backup (Timer Event)

! ── Back up running-config to TFTP every 24 hours ─────────
NetsTuts_R1(config)#event manager applet DAILY-CONFIG-BACKUP
NetsTuts_R1(config-applet)# description "Copy running-config to TFTP daily"

! ── EVENT: periodic timer — fires every 86400 seconds (24h)
NetsTuts_R1(config-applet)# event timer periodic interval 86400

NetsTuts_R1(config-applet)#  maxrun 60

! ── ACTION 1: Log the backup start ───────────────────────
NetsTuts_R1(config-applet)# action 1.0 syslog priority informational \
   msg "EEM BACKUP: Starting daily config backup on $_hostname"

! ── ACTION 2: Copy to TFTP server ─────────────────────────
NetsTuts_R1(config-applet)# action 2.0 cli command "enable"
NetsTuts_R1(config-applet)# action 2.1 cli command \
   "copy running-config tftp://10.0.0.100/backups/NetsTuts_R1-backup.cfg"
NetsTuts_R1(config-applet)# action 2.2 cli command ""
! ── Empty string confirms any TFTP dialog prompts ─────────

! ── ACTION 3: Log completion ──────────────────────────────
NetsTuts_R1(config-applet)# action 3.0 syslog priority informational \
   msg "EEM BACKUP: Daily config backup completed for $_hostname"

NetsTuts_R1(config-applet)#exit
  
The event timer periodic interval 86400 fires every 86,400 seconds (24 hours) from when the applet was last triggered. On the first trigger after a reload, the timer starts from the moment the applet was registered. The empty string action 2.2 cli command "" sends a carriage return to confirm any interactive prompts the copy command might display (such as destination filename confirmation). For configuration backup and management, see Saving & Managing Cisco Configurations.

Applet 6: BGP Peer Down Alert

! ── Alert when any BGP peer goes down ─────────────────────
NetsTuts_R1(config)#event manager applet BGP-PEER-DOWN
NetsTuts_R1(config-applet)# description "Alert when BGP neighbour resets"

! ── EVENT: match BGP neighbour state change syslog ────────
! ── %BGP-5-ADJCHANGE: neighbor x.x.x.x Down
NetsTuts_R1(config-applet)# event syslog pattern \
   "%BGP-5-ADJCHANGE.*Down"
NetsTuts_R1(config-applet)#  maxrun 30

! ── ACTION 1: Critical syslog ─────────────────────────────
NetsTuts_R1(config-applet)# action 1.0 syslog priority critical \
   msg "EEM BGP-ALERT: BGP neighbour down on $_hostname — check immediately"

! ── ACTION 2: Capture BGP summary ────────────────────────
NetsTuts_R1(config-applet)# action 2.0 cli command "enable"
NetsTuts_R1(config-applet)# action 2.1 cli command \
   "show ip bgp summary | redirect flash:/eem-logs/bgp-down.txt"
NetsTuts_R1(config-applet)# action 2.2 cli command \
   "show ip bgp neighbors | redirect flash:/eem-logs/bgp-neighbors.txt"

! ── ACTION 3: Email NOC ───────────────────────────────────
NetsTuts_R1(config-applet)# action 3.0 mail server "$_email_server" \
   to "$_email_to" \
   from "$_email_from" \
   subject "ALERT: BGP Peer DOWN on $_hostname" \
   body "A BGP neighbour adjacency has gone down on $_hostname. \
   BGP diagnostics saved to flash:/eem-logs/. Check BGP summary immediately."

NetsTuts_R1(config-applet)#exit
  

Applet 7: IP SLA Tracking Object State Change

! ── React when IP SLA tracking object changes state ───────
! ── Assumes: ip sla 1 configured to probe WAN gateway
! ──          track 1 ip sla 1 reachability
NetsTuts_R1(config)#event manager applet WAN-SLA-DOWN
NetsTuts_R1(config-applet)# description "Alert when WAN SLA track object goes down"

! ── EVENT: track object 1 transitions to "down" state ────
NetsTuts_R1(config-applet)# event track 1 state down

NetsTuts_R1(config-applet)#  maxrun 30

! ── ACTION 1: Log the WAN failure ─────────────────────────
NetsTuts_R1(config-applet)# action 1.0 syslog priority critical \
   msg "EEM WAN-SLA: Track object 1 DOWN — WAN probe failing on $_hostname"

! ── ACTION 2: Show IP SLA statistics ─────────────────────
NetsTuts_R1(config-applet)# action 2.0 cli command "enable"
NetsTuts_R1(config-applet)# action 2.1 cli command \
   "show ip sla statistics 1 | redirect flash:/eem-logs/sla-down.txt"

! ── ACTION 3: Show routing table — verify failover ────────
NetsTuts_R1(config-applet)# action 3.0 cli command \
   "show ip route | redirect flash:/eem-logs/route-at-failure.txt"

NetsTuts_R1(config-applet)#exit
  
The event track [number] state down detector integrates directly with IOS Object Tracking — the same tracking objects used by IP SLA, static route tracking, and HSRP. When the tracked object transitions to down (probe fails), the applet fires immediately — no syslog pattern matching delay. This is more precise than a syslog pattern match for IP SLA events. A companion applet with event track 1 state up can send an "all clear" alert when the WAN recovers. For IP SLA and tracking configuration, see IP SLA Configuration & Tracking. For the combination of IP SLA + EEM for automated WAN monitoring, see IP SLA with Syslog Alerting.

9. Verification

show event manager policy registered — List All Applets

NetsTuts_R1#show event manager policy registered

No.  Class     Type    Event Type          Trap  Time Registered
---  --------  ------  ------------------  ----  ----------------
1    applet    system  syslog              Off   Wed Oct 16 09:00:01 2024
  Name        : INTF-DOWN-ALERT
  Event       : pattern {%LINK-3-UPDOWN.*down}
  Description : Alert and capture diag when any intf goes down

2    applet    system  syslog              Off   Wed Oct 16 09:00:02 2024
  Name        : INTF-DOWN-EMAIL
  Event       : pattern {%LINK-3-UPDOWN: Interface GigabitEthernet0/0.*down}
  Description : Email NOC when primary WAN Gi0/0 goes down

3    applet    system  snmp                Off   Wed Oct 16 09:00:03 2024
  Name        : CPU-THRESHOLD-ALERT
  Event       : OID {1.3.6.1.4.1.9.2.1.56} op gt val {80}
  Description : Alert when 5s CPU exceeds 80 percent

4    applet    system  cli                 Off   Wed Oct 16 09:00:04 2024
  Name        : CONFIG-CHANGE-AUDIT
  Event       : pattern {conf.*t} sync {no} skip {no}
  Description : Log who entered config mode and capture diff

5    applet    system  timer               Off   Wed Oct 16 09:00:05 2024
  Name        : DAILY-CONFIG-BACKUP
  Event       : periodic interval {86400}
  Description : Copy running-config to TFTP daily
  

show event manager policy active — Applets Currently Running

NetsTuts_R1#show event manager policy active

No.  Applet name          Class     Event Type    State
---  -------------------  --------  ------------  -------
(none currently running)

! ── If an applet is mid-execution, it appears here ────────
! ── with execution time and current action label ──────────
  

show event manager history events — Execution History

NetsTuts_R1#show event manager history events

No.  Time                 Event Type          Name
---  -------------------  ------------------  -------------------
1    Wed Oct 16 14:32:01  syslog              INTF-DOWN-ALERT
2    Wed Oct 16 14:32:01  syslog              INTF-DOWN-EMAIL
3    Wed Oct 16 14:32:05  syslog              INTF-DOWN-ALERT
4    Wed Oct 16 09:00:00  timer               DAILY-CONFIG-BACKUP
  
The history shows applets 1 and 2 both fired at 14:32:01 — the same syslog event triggered both applets simultaneously (both matched the Gi0/0 down message). The history is stored in a ring buffer — the oldest events are overwritten when the buffer fills. This is the first place to check when an applet is not triggering — if the expected event is not in the history, the event detector pattern is not matching. Adjust the syslog pattern and re-test.

show event manager history traps — SNMP Trap History

NetsTuts_R1#show event manager history traps

No.  Time                 Name                  Status
---  -------------------  --------------------  -------
1    Wed Oct 16 14:32:01  INTF-DOWN-ALERT        success
2    Wed Oct 16 14:32:01  INTF-DOWN-EMAIL        success
  

show event manager environment — Verify Environment Variables

NetsTuts_R1#show event manager environment

Name                        Value
--------------------------  ---------------------------------
_email_server               10.0.0.25
_email_from                 [email protected]
_email_to                   [email protected]
_email_cc                   [email protected]
_cpu_threshold              80
_hostname                   NetsTuts_R1
_log_dir                    flash:/eem-logs/
  

Manually Trigger an Applet — Testing with event none

! ── Step 1: temporarily change the event to "none" for testing
NetsTuts_R1(config)#event manager applet INTF-DOWN-ALERT
NetsTuts_R1(config-applet)# event none
NetsTuts_R1(config-applet)#exit

! ── Step 2: manually run the applet ───────────────────────
NetsTuts_R1#event manager run INTF-DOWN-ALERT
% EEM policy successfully executed

! ── Step 3: verify files were created ────────────────────
NetsTuts_R1#dir flash:/eem-logs/
Directory of flash:/eem-logs/
   1  -rw-      3512   Oct 16 2024 14:35:22  intf-brief.txt
   2  -rw-     45672   Oct 16 2024 14:35:22  intf-detail.txt

! ── Step 4: restore the original syslog event trigger ─────
NetsTuts_R1(config)#event manager applet INTF-DOWN-ALERT
NetsTuts_R1(config-applet)# event syslog pattern "%LINK-3-UPDOWN.*down"
NetsTuts_R1(config-applet)#exit

! ── Step 5: verify logs show the manual execution ─────────
NetsTuts_R1#show event manager history events
No.  Time                 Event Type          Name
---  -------------------  ------------------  -------------------
1    Wed Oct 16 14:35:22  none                INTF-DOWN-ALERT
  
Always test new applets with event none before attaching a live trigger — this prevents the applet from firing on real events during development and allows you to execute event manager run at a controlled time. Verify all actions complete correctly, check that files are created in the expected locations, and confirm syslog messages appear with show logging before switching to the production event trigger.

show logging — Confirm EEM Syslog Actions

NetsTuts_R1#show logging | include EEM

Oct 16 14:32:01: %HA_EM-2-LOG: INTF-DOWN-ALERT: EEM ALERT: Interface down event detected on NetsTuts_R1
Oct 16 14:32:01: %HA_EM-6-LOG: INTF-DOWN-ALERT: EEM INTF-DOWN-ALERT: diagnostic files saved to flash:/eem-logs/
Oct 16 14:32:01: %HA_EM-2-LOG: INTF-DOWN-EMAIL: EEM EMAIL: Primary WAN Gi0/0 DOWN — alerting NOC via email
Oct 16 14:32:02: %HA_EM-6-LOG: INTF-DOWN-EMAIL: EEM EMAIL: Alert email sent to [email protected]
  
EEM syslog actions appear in the log with the facility %HA_EM (High Availability Embedded Manager) and the severity level specified in the action syslog priority command. The applet name is included in the log message — making it easy to filter and identify EEM-generated messages with show logging | include EEM or show logging | include HA_EM. For log buffer management and forwarding EEM syslog to a central server, see show logging and Syslog Server Configuration.

Verification Command Summary

Command What It Shows Primary Use
show event manager policy registered All registered applets — name, event type, pattern/threshold, registration time Primary verification — confirm applet is registered with correct event trigger
show event manager policy active Applets currently executing — name, state, runtime, current action label Debug a running applet — see if it is stuck on a slow action (e.g., copy to TFTP timing out)
show event manager history events Historical applet executions — which applet, when, and what event type triggered it Confirm an applet fired when expected, or diagnose why it did not fire
show event manager history traps SNMP trap history from EEM — applet name and success/failure status Confirm SNMP-triggered applets executed and check for execution errors
show event manager environment All configured EEM environment variables and their current values Verify environment variables are set correctly before testing email or file path actions
event manager run [name] Manually triggers an applet with event none configured Test applet actions safely without waiting for a real event to occur
show logging | include HA_EM All EEM-generated syslog messages in the log buffer Confirm action syslog commands are executing and producing the expected log messages
dir flash:/eem-logs/ Files written to flash by EEM action cli command with | redirect Confirm diagnostic capture files are being created with correct timestamps

10. Troubleshooting EEM Applets

Problem Symptom Cause Fix
Applet never triggers despite the event occurring show event manager history events shows no executions for the applet — the interface went down but INTF-DOWN-ALERT never fired The syslog pattern in the event syslog pattern clause does not match the actual syslog message IOS generates. IOS syslog message text varies slightly between IOS versions and platforms Capture the exact syslog message from show logging at the moment the event occurs. Compare it character-by-character with the pattern in the applet. The pattern is a regular expression — test it with event none and manually verify the regex matches the log line. Remember that the % character in the pattern matches the literal % in the syslog message. Use debug event manager to see real-time EEM pattern matching activity
action cli command fails — access denied or incomplete output Applet fires but the CLI commands produce no output files, or show event manager history shows the applet executed with errors. dir flash:/eem-logs/ shows no files created The EEM session CLI user is not configured (event manager session cli username is missing), or the configured user does not have privilege level 15, or the flash directory does not exist Verify event manager session cli username eem-user is in the running config. Confirm the user has privilege 15 with show running-config | include username eem-user. Create the log directory manually first: mkdir flash:/eem-logs from the router CLI. Re-test with event manager run [name]
Email action never sends emails Applet fires and logs show action 3.0 mail executing, but no email arrives at the NOC inbox The SMTP server (_email_server) is not reachable from the router, the SMTP server does not accept relay from the router's source IP, or the environment variable contains an incorrect IP address Verify SMTP server reachability: ping 10.0.0.25 from the router. Confirm the SMTP server allows relay from the router's IP. Check show event manager environment to verify _email_server is correct. Try a simple Telnet test: telnet 10.0.0.25 25 — if port 25 does not respond, SMTP is blocked. In many environments it is easier to use a syslog-to-email gateway instead of direct SMTP
Applet triggers too frequently — log flooding The applet fires dozens of times per minute — syslog is flooded with EEM messages and flash storage fills up quickly with captured files The triggering event is recurring very rapidly (e.g., an interface flapping up and down repeatedly) and the applet has no rate limiting or cooldown period Add a rate limit to the event detector: event syslog pattern "..." maxrun 60 ratelimit 300 — the ratelimit 300 prevents the applet from firing again for 300 seconds (5 minutes) after its last execution. Alternatively use a event timer watchdog pattern to only fire once until manually reset. Investigate the root cause of the flapping interface separately
Applet actions execute out of order or skip actions The applet fires but only some actions execute — for example, action 1.0 syslog works but action 2.0 cli command and 3.0 cli command do not run The maxrun timer expired before all actions completed — IOS terminates the applet at the maxrun limit. CLI commands that capture large outputs (e.g., show processes cpu sorted | redirect) can take 5–15 seconds each Increase the maxrun value: for applets with multiple CLI capture commands, set maxrun 120 or higher. Verify the actual runtime with show event manager policy active during execution or review show event manager history events for timeout indicators. Remove non-critical actions to reduce runtime
Config-audit applet blocks the operator's configure terminal When an engineer types conf t, the prompt hangs for 20–30 seconds before entering config mode — the EEM applet is running synchronously and blocking IOS The CLI event applet was configured with sync yes (or the default sync behaviour) — the applet runs synchronously, meaning IOS waits for the applet to complete before processing the matched command Change to asynchronous execution: event cli pattern "conf.*t" sync no skip no. With sync no, the operator's command executes immediately and the applet runs in the background. This is the correct setting for audit applets that should observe without blocking. Reserve sync yes only for applets that must complete before the matched command is processed (e.g., an applet that validates or modifies the command)

Key Points & Exam Tips

  • EEM (Embedded Event Manager) is an IOS-native automation engine that allows policy rules called applets to be defined directly on the router. Each applet has one event clause (trigger) and one or more action clauses (responses), executed in ascending label order.
  • The most common event detector is event syslog pattern, which uses a regular expression to match IOS-generated syslog messages. The pattern must match the exact format of the syslog message including facility, severity, and mnemonic (e.g., %LINK-3-UPDOWN.*down).
  • For action cli command to execute privileged commands, event manager session cli username [user] must be configured globally, and the referenced user must have privilege level 15. Without this, CLI actions silently fail or run with user-level privilege.
  • Action labels (1.0, 2.0, 3.0) determine execution order — actions run in ascending numerical order. Use decimal labels (1.5, 2.5) to insert actions between existing ones without renumbering. All actions within an applet run as a single atomic sequence unless maxrun terminates the applet early.
  • Environment variables (event manager environment [name] [value]) allow reusable parameters to be defined once and referenced in multiple applets with $_varname syntax. Change email addresses, thresholds, or server IPs without editing individual applets.
  • Always test new applets using event none and event manager run [name] before attaching a live event trigger. This allows safe testing of all actions without waiting for a real fault event to occur.
  • The ratelimit [seconds] option on the event clause prevents applet re-triggering for the specified duration after execution — essential for events that may recur rapidly (interface flapping, repeated syslog patterns) to prevent log flooding and flash exhaustion.
  • For CLI event applets, sync no skip no means the applet runs asynchronously (does not block the operator's command) and the original command still executes. sync yes skip yes means the applet blocks the command and the original command is prevented from executing — use this pattern to block dangerous commands.
  • show event manager policy registered is the primary verification command — it lists all applets with their event type and pattern. show event manager history events shows execution history — if an expected event is missing from the history, the event detector pattern is not matching the actual syslog message.
  • On the exam: know the EEM applet structure (event manager appleteventaction), the four most common event detectors (syslog, track, timer, cli), the key actions (syslog, cli command, mail), the purpose of environment variables, and the sync/skip options for CLI events. Also review debug commands and show logging as the primary tools for diagnosing why an EEM applet is or is not triggering.
Next Steps: For combining EEM with IP SLA probes to generate automatic WAN monitoring alerts, see IP SLA Configuration & Tracking and IP SLA with Syslog Alerting. For sending EEM syslog output to a central logging server, see Syslog Configuration and Syslog Server Configuration. For the SNMP infrastructure that backs the event snmp detector, see SNMP v2c & v3 Configuration and SNMP Traps. For Python-based automation that can replace or supplement EEM for more complex multi-device workflows, see Python Netmiko Show Commands. For traffic capture at the time of an EEM-detected event, see SPAN & RSPAN Port Mirroring. For device hardening that complements EEM configuration-change auditing, see Login Security & Brute-Force Protection.

TEST WHAT YOU LEARNED

1. What are the two mandatory components of every EEM applet, and what happens if an applet has actions but no event clause?

Correct answer is C. An EEM applet is defined with event manager applet [name], and within it requires exactly one event statement (the trigger) and one or more action statements (what to do). The description is optional but highly recommended for documentation. If you write actions but forget the event clause, IOS accepts the partial configuration but the applet is not registered with the EEM scheduler — it never executes automatically. show event manager policy registered will not show it, or will show it without an event type. The event none configuration is a special deliberate pattern used specifically for testing: it tells EEM that this applet has no automatic trigger and can only be invoked manually with event manager run [name]. This allows safe incremental testing of all actions before attaching the production event trigger.

2. An EEM applet is configured with event syslog pattern "%LINK-3-UPDOWN.*down". An interface goes down and IOS generates the message %LINK-3-UPDOWN: Interface GigabitEthernet0/0, changed state to down. The applet does not fire. What is the most likely cause?

Correct answer is A. The EEM syslog event detector works by monitoring the IOS syslog message stream. If the logging buffer is configured with a severity filter that excludes the message (e.g., logging buffered informational would filter out severity 3 errors since informational is severity 6, but actually severity 3 is errors which is more severe and would be included — the common issue is logging buffered notifications level 5 or higher which would exclude severity 3). More precisely: IOS logging levels work numerically — a lower number means higher severity. If logging buffered warnings (severity 4) is configured, severity 3 (errors) messages ARE included. But if no logging is configured or logging is completely disabled, EEM cannot see any syslog messages. The key diagnostic is show logging — verify the message actually appears in the buffer. If it does not appear in the buffer, EEM's syslog detector cannot match it. EEM syslog monitoring is independent of whether messages go to a remote syslog server, but it does depend on the local logging subsystem being active. Option B is incorrect — EEM syslog patterns fully support standard POSIX-style regular expressions including .*.

3. Why is event manager session cli username eem-user required for applets that use action cli command, and what happens if it is missing?

Correct answer is D. EEM's action cli command creates a virtual VTY-like session within IOS to execute commands. Unlike a human operator who authenticates via SSH or console and establishes a privilege level, EEM's virtual session has no inherent authentication context. The event manager session cli username [user] global command tells EEM which local user identity to use for this virtual session — the user's configured privilege level (set with privilege 15 in the username definition) determines what commands the EEM session can execute. Without it, EEM either fails to create the CLI session at all or creates one at privilege 1 (user EXEC only) which cannot run show processes cpu, copy running-config, or any privileged command. The failure is particularly insidious because the applet shows as "executed successfully" in the history (the action statement itself did not fail), but no output files are created and no commands actually ran with meaningful effect. The diagnostic is to check dir flash:/eem-logs/ for expected output files and compare with show event manager history events for execution confirmation.

4. What is the difference between event cli pattern "..." sync yes skip yes and event cli pattern "..." sync no skip no, and give an example use case for each?

Correct answer is B. The sync and skip parameters on CLI event detectors are among the most powerful and frequently misunderstood EEM options. Sync controls the execution relationship between the applet and the original command: sync yes means the CLI command is held pending while the applet runs to completion; sync no means the CLI command processes immediately and the applet runs concurrently in the background. Skip controls whether the original command actually executes after the applet runs: skip yes suppresses the original command (the user sees it being accepted but nothing happens); skip no allows it to execute normally. The combination sync yes skip yes effectively "intercepts" the command — useful for creating command blocks or confirmations. For example, an applet matching write erase with sync yes skip yes would log the unauthorised attempt and prevent the config from being erased. An applet with sync no skip no runs in pure observation mode — invisible to the operator, no delay, full command execution — appropriate for audit logging. Note: skip yes with sync no would be contradictory in most use cases since the command fires before the applet completes.

5. An EEM applet with five action cli command statements fires but only the first two actions execute. show event manager history events shows the applet ran. What is the most likely cause?

Correct answer is C. The maxrun parameter sets the maximum wall-clock time EEM allows for an applet to complete. If the applet is still running when maxrun expires, IOS terminates it immediately — any remaining actions are never executed. The default maxrun is 20 seconds on most IOS versions, which is often insufficient for applets that run multiple show commands and redirect their output to flash. show processes cpu sorted | redirect flash:/file.txt on a busy router can take 8–12 seconds alone. Five such commands would require 40–60 seconds. The fix is to increase maxrun: within the applet configuration, add maxrun 120 (or higher) to give the applet 2 minutes to complete. Verify the actual runtime needed by testing with event none and event manager run while watching show event manager policy active to see the elapsed time. Labels use decimal numbers (1.0, 2.0) which is valid IOS syntax — option B is incorrect.

6. What is the purpose of EEM environment variables, and how do they differ from local variables set within an applet using action set?

Correct answer is D. Understanding the scope difference between environment variables and local variables is essential for writing maintainable EEM policies. Environment variables are the EEM equivalent of global constants: defined once in global config with event manager environment _email_to [email protected], they are available to every applet and every applet execution. Change the NOC email address in one place and all applets immediately use the new value. They are also visible in the running-config and shown by show event manager environment. Local variables (action set) are runtime scratchpad variables: they are created during an applet execution, hold a value for use by subsequent actions in the same run, and are destroyed when the applet completes. They are typically used to store data retrieved from the triggering event (the interface name from a syslog message, the SNMP value that triggered the threshold) for use in subsequent actions within that same run. A key practical point: the info type actions retrieve event-specific data from the EEM event context and store it in built-in variables (like $_info_routername) — these are also local to the applet execution.

7. How does the event track [number] state down detector differ from using event syslog pattern "%TRACK.*Down" for the same IP SLA failure event?

Correct answer is A. This distinction is important in production deployments where the timeliness of automated response matters. The IOS Object Tracking subsystem maintains the state of tracked objects (IP SLA reachability, interface line protocol, route existence) and notifies subscribers synchronously when state changes. EEM's event track detector registers as a subscriber — it is called directly by the tracking subsystem at the instant the state changes, with no intermediate processing. The event syslog approach depends on IOS generating a log message about the track state change, the message passing through the syslog infrastructure (which may apply rate limiting, buffering, or severity filtering), and then the EEM syslog detector scanning the message for a pattern match. In normal operation this happens within seconds, but under high CPU load or with syslog rate limiting active, the syslog-based approach can be delayed by 30+ seconds or miss the event entirely. For critical events like WAN failover detection where automated rerouting or alerting must happen within seconds, event track is the correct choice. See IP SLA Configuration & Tracking for the tracking object setup that event track monitors.

8. An EEM applet designed to send an email when a BGP peer goes down is triggering 40 times per hour during a BGP flap event, flooding the NOC inbox and filling flash with diagnostic files. How should the applet be modified to prevent this?

Correct answer is B. The ratelimit option is the correct, purpose-built mechanism for preventing applet flooding. When ratelimit [seconds] is added to the event clause, EEM tracks the last execution time of the applet and ignores all matching events that occur within the ratelimit window after execution. The applet fires on the first matching event, then will not fire again until the ratelimit period expires — even if hundreds of matching events occur during that window. For BGP flapping at 40 events per hour, ratelimit 300 (5 minutes) would reduce execution to at most once per 5-minute period, cutting the flood from 40 to 12 per hour maximum. A ratelimit of 900 (15 minutes) would further reduce it to 4 per hour. The ratelimit does not prevent the alert from eventually being sent — it just ensures the NOC is not overwhelmed with repetitive notifications about the same ongoing event. Option C is incorrect — maxrun controls the maximum duration of a single applet execution, not the interval between executions. Using maxrun 3600 would mean a single applet run could take up to one hour, not that subsequent triggers are blocked.

9. A security team wants an EEM applet that prevents any user from running write erase on production routers, logs the attempt with the username, and sends a syslog alert. Which event and skip configuration achieves this?

Correct answer is C. This is the only combination that correctly blocks the command AND allows the applet to perform audit actions. The key insight is the interaction between sync and skip: sync yes means IOS holds the matched command in a pending state while the applet runs to completion. Only after the applet finishes does IOS decide whether to execute or skip the original command. skip yes tells IOS to skip (not execute) the original command after the applet completes. Together: the operator types write erase, IOS intercepts it via EEM, the applet logs the username and sends a syslog alert, and then IOS suppresses the original write erase — the config is never erased. Option B (sync no skip yes) is contradictory and does not work as intended: with sync no, the command executes concurrently with the applet — skip yes cannot retroactively prevent a command that has already been dispatched for execution. Option D (sync yes skip no) would log the attempt but then execute the command — the running-config would still be erased. This applet pattern is a practical and commonly deployed security control on Cisco devices where AAA is not fully implemented.

10. show event manager history events shows an applet named INTF-DOWN-ALERT executing 12 times in the last hour. The interface actually went down only once. What are two possible causes and how are they identified?

Correct answer is D. Multiple triggers from a single real event is one of the most common EEM operational surprises. When a Gigabit Ethernet interface goes down, IOS does not generate just one syslog message — it typically generates a cascade: %LINK-3-UPDOWN: Interface ... down (Layer 1), %LINEPROTO-5-UPDOWN: Line protocol ... down (Layer 3), possibly %OSPF-5-ADJCHG: Process ... neighbor down, %BGP-5-ADJCHANGE: neighbor ... Down, STP topology change notifications, and more. If the pattern %LINK-3-UPDOWN.*down is crafted carefully to match only the Layer 1 event, it fires once. But if the pattern were .*down (too broad), it would match all of these related messages and fire 10+ times for one physical link failure. The diagnostic is straightforward: capture show logging output during a test interface shutdown and count how many lines match the applet's pattern regex. Each matching line represents one potential applet execution. Then add ratelimit as a belt-and-suspenders protection regardless. Both root causes (overly broad pattern and missing ratelimit) are common in production and should both be addressed for a robust applet.