esc
  • WorkflowsPages
  • Case studiesPages
  • Machine Learning FoundationsLearn
  • AI EngineeringLearn
  • AI Guardrails & Red TeamingLearn
  • UI/UXLearn
  • Design SystemsLearn
  • Production System DesignLearn
  • Book a callContact
  • Download ResumeContact
Let’s talk

I’ll build the agent for it.

Security Monitoring: Watching Signals to Spot Attacks Early

How teams collect login, access, and traffic signals, turn them into alerts, and act on them, with a worked AWS example.

In short

Security monitoring collects records such as logins and access events, keeps them safe, and reviews them so teams can spot attacks early.

11 min readFree

What You Will Learn

Security monitoring means collecting signals from a system all the time and watching them. The goal is to notice attacks, misuse, and policy breaks quickly, so someone can act. It answers one question: is someone doing something they should not, and would we know? This lesson covers what to watch, how raw records become useful alerts, and how an alert leads to action.

Monitoring belongs in system design because the design decides what gets recorded, where those records are kept, and who gets told. If a design records nothing, nobody can check what happened later.

Key words

Signal
A piece of recorded information about something that happened in a system, such as one login attempt.
Event
One recorded action, with a time, an actor, and a result.
Log
A stored list of events, kept in time order.
Authentication
Proving who you are, usually with a password, a code, or both.
Authorization
Checking what an identified user is allowed to do.
Baseline
A picture of normal behavior, used to spot what is unusual.
Threshold
A set number that, once passed, triggers a finding or an alert.
False positive
An alert raised for activity that turned out to be harmless.
Anomaly
Something that clearly differs from the normal baseline.
Incident
A security event that causes, or may cause, harm and needs a planned response.
On-call
The person assigned to respond when an alert fires, often outside normal work hours.
Tamper-resistant
Hard to change or delete without leaving a trace.
SIEM
A central tool that gathers logs from many sources and runs detection rules on them.

Why Does Monitoring Matter?

A running system produces a steady stream of records. Each login, file read, or permission change can leave one. Security monitoring picks the right records, keeps them safe, and reviews them for trouble.

According to OWASP guidance, a team that keeps good records can notice attacks, study incidents after they happen, and show the facts afterward. OWASP also lists logging and monitoring failures among its top ten web application risks, because weak records make attacks harder to see and prove.

Tip

Collecting records is not the same as monitoring them. A log that nobody reads cannot stop anything.

What Should You Watch?

Not every record matters equally. The signals below show up again and again in security guidance.

  • Login events

    Record successful and failed logins. Failed logins can show someone guessing passwords. A success that follows many failures can show a break-in.

  • Authorization failures

    Requests that break an access rule on the server should be recorded. A user trying to open something outside their rights is a clue.

  • Permission and configuration changes

    Administrative actions, including changes to security configuration settings, should be logged. A changed role can open a door for an attacker.

  • Unusual traffic

    Malformed requests and validation failures may signal probing or injection attempts. A sudden surge from a few addresses also stands out.

  • Access to sensitive data

    Many systems leave unauthorized access to sensitive data unrecorded, which is a gap. Reads, exports, and deletes of private data should leave a trail.

Be careful what goes into a log. Logs should not contain sensitive details such as session identifiers or passwords. A log full of secrets becomes a new target.

Tip

For each signal, ask two questions: who did it, and would we know if they did it at 3 a.m.?

Keep Logs Central and Hard to Change

Logs are useful only if they survive an attack. Weak logging systems can let attackers reach or change the logs, which makes their activity hard to trace. That is why logs need protection of their own.

A common fix is to gather logs from every service into one central store. Developer guides recommend sending logs to a central system and protecting them from editing. An attacker who takes over one machine then cannot quietly erase its history.

Tamper-resistant logs also need a way to prove they were not edited. If the text changes, the fingerprint no longer matches.

Entries must also be written safely. Dangerous characters should be encoded before logging, so that an attacker cannot forge fake lines. Finally, keep logs long enough. Logs that are deleted too soon make it hard to investigate older incidents.

How Does Detection Work?

Raw events are not findings. A finding is a conclusion that something needs attention. Detection is the step that turns many events into a small number of findings. Three tools do most of this work: baselines with rules and thresholds, anomaly detection, and human review.

Baselines, Rules, and Thresholds

A baseline is a measure of normal, such as how many logins an app sees on a typical day. A rule is an if-then check, and a threshold is the number that triggers it. A classic rule is a burst of many failures on one account, or from one address, within a few minutes.

Rules are cheap and easy to explain, but they only catch what someone thought to write down.

The sample below is a small rule in Python. It uses only the standard library. It counts failed logins for each address and flags any address that reaches a threshold of five. Real systems use time windows and far more data, but the idea is the same.

Python
from collections import Counter

# Each event is (source IP, result of a login attempt)
events = [
    ('203.0.113.7', 'failed'),
    ('203.0.113.7', 'failed'),
    ('203.0.113.7', 'failed'),
    ('203.0.113.7', 'failed'),
    ('198.51.100.4', 'success'),
    ('192.0.2.9', 'failed'),
    ('203.0.113.7', 'failed'),
]

THRESHOLD = 5

failures = Counter(ip for ip, result in events if result == 'failed')
for ip, count in sorted(failures.items()):
    if count >= THRESHOLD:
        print(f'ALERT {ip}: {count} failed logins')
    else:
        print(f'ok {ip}: {count} failed logins')

Output

ok 192.0.2.9: 1 failed logins
ALERT 203.0.113.7: 5 failed logins

Quick check

Your web app's logs show 5,000 failed login attempts in ten minutes across hundreds of usernames, all from a few IP addresses. Then one login succeeds. Which reading and next step fit best?

Anomaly Detection

Anomaly detection compares what is happening now with a learned baseline and flags large differences. It can catch a new pattern that nobody wrote a rule for. Cloud tools use this idea.

The model is not just about location. This helps cut noise, because a new address alone may not be enough.

False Alarms vs Missed Attacks

Every threshold is a trade-off. Set it too low, and the team gets flooded with false positives, which are alerts for harmless activity. Guides warn that alert fatigue can cause real warning signs to be missed. Set it too high, and a real attack may never trigger anything.

A single odd event rarely proves a breach. One strange login is not enough to confirm a breach. A traveler may log in from a hotel abroad. Good designs therefore look for a second signal before they treat the event as an attack.

Tip

Start with a few high-value rules, such as brute force, privilege increases, and known-bad addresses. Tune their thresholds with real data before you add more rules.

Worked Example: A New Login and New Access Keys

Here is a real AWS setup. The goal: if someone signs in to the AWS web console from a country the account never uses, and then creates new access keys, an alert should appear. A person can then investigate and lock the access down.

  1. 1

    Turn on CloudTrail

    CloudTrail provides a history of AWS API calls for the account, including calls made through the console, SDKs, and command-line tools. This is the record of what happened.

  2. 2

    Turn on GuardDuty

    GuardDuty analyzes data to detect unwanted activity. It does the detection so your team does not write every rule.

  3. 3

    Watch the login

    GuardDuty can flag a console login when a user signs in from a client it has never used before, or from an unusual location. The login from the new country becomes the first signal.

  4. 4

    Watch the new keys

    An attacker creates access keys to solidify their foothold in the account. In CloudTrail, the new key shows up as a record of the key being created, right after the login.

  5. 5

    Raise the alert

    Team members can examine and manage GuardDuty findings in the console or through the API.

  6. 6

    Investigate

    GuardDuty cannot answer every question on its own. Check the activity right after the login, such as key creation or policy changes, and confirm whether the real user signed in.

  7. 7

    Lock down

    If the access was not authorized, revoke the user's console sessions, reset the password and multi-factor authentication (a second proof of identity), and rotate the access keys.

Tip

AWS has retired some older GuardDuty finding types, so check the current list in AWS's documentation before you write rules that depend on one name.

How Do You Respond to an Alert?

An alert that nobody owns is just noise. Good monitoring names who receives each alert and what that person must do. An incident is a security event that causes, or may cause, harm and needs a planned response.

Who Owns the Alert?

Each alert needs an owner, often the person on call for that shift. Silence is also an alert.

A Basic Incident Flow

The steps below follow the plan this lesson uses.

  1. 1

    Detect

    Effective detection tools are the foundation of a timely response. The sooner a signal reaches a person, the sooner the response starts.

  2. 2

    Triage

    Triage means sorting alerts quickly. Ask whether the event is real and assess the incident's scope, impact, and severity.

  3. 3

    Contain

    Containment limits the impact and stops the incident from spreading. In the AWS example, that means revoking sessions and freezing the keys.

  4. 4

    Recover

    Recovery can include resetting passwords and restoring systems. Only return to normal once the attacker's access is gone.

  5. 5

    Review

    Lessons from the incident feed back into preparation for the next one. Ask which signal helped, and which one was missing.

NIST groups containment, eradication, and recovery into one phase, so the steps above are a simpler view of the same work. Taken together, the phases form a full lifecycle for handling security incidents.

Quick check

A staff member logs in from a hotel abroad during a trip. The alert fires at 3 a.m. The on-call engineer sees one new login, with no new keys and no policy changes. What is the best next step?

Feeding Monitoring Back Into Design

Monitoring does not end when an incident closes. Each review should ask whether the design made the attack easy to hide. If it did, change the design. Log the missing event, add a rule, limit failed login attempts, or split admin rights into smaller roles.

Testing helps too. Compare each test attack with the alerts it produced, and track how many attacks were caught. Gaps found this way show where the design needs work.

Quick check

A payment service usually sends security events every hour. For a full day, the dashboard shows none from it. What should the team check first?

Key Takeaways

Security monitoring asks whether someone is doing something they should not, and whether you would know. Watch the right signals, keep logs central and hard to change, and turn events into findings with baselines, rules, and anomaly checks. Balance false alarms against missed attacks. Give every alert an owner, follow a clear incident flow, and feed what you learn back into the design.

Sources

How is this lesson?

Last updated on 18 November 2026

Learn with me

Every lesson is free. I add new ones as I learn. Have a question, or want help with your own work? Book a short call.