Ariadne’s Network Notes

My notes are mostly Cisco things centered around these certification exams: CCNA, ENCOR, ENARSI, ENSLD and CCIE EI.

I’ve been in the Networking Industry since 2012, I’ve been an IT Person since 1999.

I’ve worked at Cisco on-and-off over the last 10y+, on teams like TAC for IOS-XR, or HTTS PS TAC for Route/Switch.

• Linkedin.
• Resume.

Push, Pop, and Swap are MPLS operations.

Email: ariadne@haske.org

License

This work is shared under Creative Commons Attribution-ShareAlike 4.0 International.

How to use this site

Search

Click the 🔍 button.

Set a theme

I like Ayu.

This site isn’t an authoritative

I usually cite my sources, this website is shorthand for better materials.

Parts of it are certainly wrong.

… what does that mean?

I usually do something like this

• Open a web search
  • site:cisco.com ios xe ospf

An alternative

• Ask your favorite LLM for the definitive Cisco Live / NANOG / IETF / Industry group slides
• Never let the LLM explain, only let it explain why you should read a specific source

Every topic has its expert. Find that person’s presentation materials.

I rarely reference someone else’s blog. I have too much distrust of how easy it is to get stuff wrong, myself included.

Why does this site exist?

• I like linking it to people on Discord
• I am very forgetful

How to Study

I do something like:

1. Skim material (This is a vibe check).
2. Go back over material, slower, with an eye towards the exam topics.
3. Take basic notes.
4. Pick a topic, create a lab with a working example.
5. Find source materials, or more examples of a given technology.
6. Talk with others about the topic.
7. Understand the existing lab, look at debugs, logs, packets.
8. Read more source Material.
9. Take more advanced notes documenting observations about behavior.
10. Understand where this concept fits with other related concepts …

Image by Tidema - CC BY 4.0

References

Bloom’s taxonomy - Wikipedia

How to Take Notes

The process

Consistency

If all your notes are in spiral bound notebooks or 3x5 cards, that’s OK, but pick a system and stick with it.

Detail

Cite your sources. You need enough to remember where something comes from.

Summarize

Do not re-create source materials.

• Record facts
• Summarize and reduce scope
• Fit your notes into your whole understanding (synthesis)

An hour long meeting? One page of notes. Don’t overdo it.

Myths

I don’t need to take notes. I can remember everything just fine

Note down what you will not remember

• Bit counts
• Packet headers
• Tiny specific interactions

You can find these details via

• Reading white-papers
• Reading RFCs
• Reading forum posts
• Doing labs
• Finding outputs
• Collecting packets
• Collecting debugs

I don’t look at my notes after I take them

Don’t take notes on stuff you’ll remember, take notes on stuff you’ll forget.

Learn what kinds of details you forget.

The notes aren’t for you six months from now, the notes are for you, six years from now.

I can’t take notes digitally, I need to write them out

You can take handwritten notes, just OCR them into digitals. The AI is decent at this.

If typing is painful consider a much nicer keyboard and something like Dvorak.

References

MIT - Notes and Notebooks - Mayfield Handbook of Technical and Scientific Writing

Berkeley - Academic Skills Resource Library | Athletic Study Center

Harvard - Note-Taking – Academic Resource Center

How to Make a Blog Like This

Be correct

I always ask myself these questions:

• Who is the authority on this?
• Is someone being paid to be correct?
• Does this activity have a following, group, conference, forum, or discord?
• Can this be tested in the lab?
• Can this behavior be observed?
• Can the behavior be documented?
• Can the behavior be diagrammed?
• Is effort tracked by a standards body? e.g., the IETF, IEEE, etc.

See The Correctness Hierarchy.

Documentation as code

This blog follows the Documentation as Code ethos.

• Create an account on GitHub
• Write articles in Markdown
• Learn Git Fundamentals
• Use Git
• Use a development environment
• Use GitHub Pages and Actions to Deploy mdBook
  • My Deploy script on GitHub

My development environment

I use Notepad++ with Syncthing.

When I save a file in Notepad++, Syncthing is notified of the change and propagates it to tesseract. mdBook sees the modification and rebuilds the webpage.

ariadne@tesseract:~/git/pushpopswap$ mdbook serve -n 0.0.0.0
 INFO Book building has started
 INFO Running the html backend
 INFO HTML book written to `/home/ariadne/git/pushpopswap/book`
 INFO Serving on: http://0.0.0.0:3000
 INFO Watching for changes...
 INFO Files changed: ["/home/ariadne/git/pushpopswap/src/how-to-make-a-blog-like-this.md"]
 INFO Book building has started
 INFO Running the html backend
 INFO HTML book written to `/home/ariadne/git/pushpopswap/book`
 INFO Files changed: ["/home/ariadne/git/pushpopswap/src/how-to-make-a-blog-like-this.md"]
 INFO Book building has started
 INFO Running the html backend
 INFO HTML book written to `/home/ariadne/git/pushpopswap/book`

Processing

mdBook is used to turn CommonMark into html.

These tools extend the html features, typically with JavaScript.

Preprocessing

These are tasks mdBook cannot do without addons.

Mermaid

• Javascript
• Turns text into diagrams
• Allows version control for diagrams
• SVG Adaptive
  • Resize Nicely
  • Light and Dark theme adaptive

Mermaid is a binary js file that gets copied to the root of the repo.

curl -sL https://cdn.jsdelivr.net/npm/mermaid@11.5.0/dist/mermaid.min.js -o mermaid.min.js

Then mdbook needs to be rebuilt:

mdbook build

GitInfo

• Injects Git Metadata into the rendered HTML articles
  • Commit
  • Date of Commit
  • Link to Commit
  • Onto every webpage

SVGBOB

• buggy
• Converts ASCII Art to a SVG
• It does this on build

mdbook-svgbob uses the above embed svgbob into mdBook

Custom Scripts

Javascript here modifies mdbook or extends mdbook to allow a better site viewing experience.

Mermaid

Mermaid lazyload

mermaid-lazyload.js

Lighthouse penalizes pages based on first content paint times.

Mermaid comes with a large binary, ~3MB: mermaid-11.15.0.min.js.

This script is put into the render blocking path. The big mermaid file is only loaded on pages with Mermaid.

Connected to:

• mermaid-11.15.0.min.js
• svg-zoomer.js

SVG zoomer

svg-zoomer.js

There are some very large diagrams on this site.

SVGs and Mermaid diagrams can’t be zoom’d natively on desktop. This script creates a fullscreen pan/zoom viewer.

• Desktop only: on mobile the browser’s own pinch-to-zoom already works; this script is disabled.
• Self-contained (Pointer Events), no libraries

Connected to:

• mermaid-lazyload.js

Zoom unstick

zoom-unstick.js

On touch devices a sticky menu bar rides along magnified during pinch-zoom, covering the content being zoomed. This tags <html> with pinch-zoomed while zoomed in so the menu bar unsticks, then restores it when zoomed back out.

• Uses visualViewport scale, no libraries
• Keeps the search button reachable at normal zoom

SEO

This site doesn’t do great on Bing.

Stamp page mtimes

stamp-page-mtimes.sh

Necessary for accurate dates on pages before submitting with IndexNow.

This is run right before build.

Use-ability

Things to make the notes experience nicer.

Editable extras

editable-extras.js

• Extends mdBook’s editable code block behavior after book.js runs
  • Adds undo controls for editable non-Rust examples
  • Uses code fences: ```text,editable

AI Use Statement

There is some AI use here, I generally write all the articles in my own voice.

Some diagrams are AI assisted, I go over the theory by hand.

I token maxx, balancing between $20 on Anthropic and $20 on OpenAI.

Where LLMs are used

• OCR tasks
  • Diagram-to-Code tasks
• Reforming tables
• Lint
• Spelling and grammar
• Finding potential technical errors
• Rubber Ducking
• Vibe Coding
• Debugging
• Lab Implementation

Where LLMs are not used

• Theory
• Debugging
• Lab Design
• Note Taking
• Prose
• Technical Writing

The Correctness Hierarchy

This is an ordered list of how correct something tends to be, from the standpoint of a Network Engineer.

Ariadne’s maxim

The more correct a material tends to be, the harder it becomes to understand and/or access.

The list

A tier list ranking sources of technical correctness from S (most authoritative) to F (least): S is observed behavior, A is published standards, B is inventor sources, C is vendor sources, D is Wikipedia, E is content creator sources, F is vendor marketing and social media.

Terms

Published Standards — IETF Documents or IEEE documents

Inventor — The person who created it

Distinguished Engineer — Someone famous in the Industry

Content Creator — Someone making content

Examples

Using the AI to be correct is a failing grade at F-tier. Same goes for social media (let’s say, TikTok, Linkedin, Facebook, et. al)

Vendors lie when they market, so they also go to F-tier.

Textbooks are E-tier, there is no independent auditing of textbook correctness, and they are very difficult to correct once published.

If the vendor has a textbook, that’s D-tier. Usually, a certification is tied to it, so they are a touch better edited.

The vendor community is interesting, people post both very stupid questions, but also get brilliant answers. Solid C-tier.

Vendor materials can’t get above C-tier because money is always involved. Getting material published at a vendor is difficult enough.

John Moy’s book on OSPF is B-tier. Ivan’s site https://blog.ipspace.net/ is also B-tier, he’s been in the Industry forever.

The RFC for OSPFv2 is A-tier, it’s what we all refer to to check the behavior, otherwise we need to get onto a mailing list.

A setup running OSPFv2 is S-tier. It might be ancient, broken, or incorrect, but we cannot argue that it did not happen.

Git

Git is a collaborative code tool. It allows distributed development, with yourself or others.

The database that holds the code, the history of changes to the code is a repository.

Previous version control tools used diffs, which are changes or deltas between files “lines 38 to 42 have changed”, Git is snapshot based.

Every file being tracked is hashed with SHA-1, each and every time a commit is made. Git always knows if files have been changed.

The list of files and their hashes is called the Index.

Terms

Repository

• .git/
• AKA, Repo
• AKA, the object store
• Content Addressable Filesystem
• Key-value based. All of Git is key-value
• De-Duplicated. If two files have the same SHA-1, git stores 1 blob
  • Git stores: blobs, trees, commits, and tags
• Interacted with almost purely via git

Commits

• Takes a snapshot of the current Index
• SHA-1 entity that points to a specific tree
• Metadata: author, commiter, timestamp, messages, parent commit
• Commits are backwards chains
  • All commits store the hash of previous commits

Tree

• A directory snapshot
  • Mode (file type and permissions)
  • Name (file or directory)
  • SHA-1 (the hash)
  • Trees point to blobs and other trees

Blob

• A key-value pair that represents a file
• 82da472f6d00dc5f0a651f33ebb320aa9c7b08d0 LICENSE
• The SHA-1 is used to find the compressed content of LICENSE

Branch

• A named pointer to a specific commit
• Commits are stored as nodes on a DAG
• Merge commits have two or more parents
• The init commit has no parents

HEAD

• A named pointer to the current branch
• In detached HEAD state, points directly to a commit
• Works like the playhead on a tapedeck

Working tree

• AKA, project directory
• AKA, user directory
• AKA, your files
• This is where project files are modified
• Directory inside git init was ran

The index

• .git/index
• AKA, staging
• AKA, cache
• AKA, pre-commit
• AKA, git’s files
• Invoked with git add
• Adding a file does two things:
  • SHA-1 of the file, adding it to the Index
  • Writes the blob to .git/objects/
• Files in the Index are tracked

Local files overview

Git tries to avoid touching working tree files. When a file is added, a snapshot is taken at that time.

Image courtesy of Derrick Stolee.

Branch example

gitGraph
   commit id: "A"
   commit id: "B"
   commit id: "C"
   commit id: "D"
   commit id: "HEAD → E"

A, B, C, D are previous commits.

The working tree contains the files from commit E.

I think of it like a tape deck, the HEAD can be played backwards or forwards.

commit moves the HEAD forward, and reset moves the HEAD backwards.

Remote and local overview

A basic example: a local repo, and a remote repo.

flowchart TD
    Remote --> Local
    Local --> Remote

The default reference for the remote repo is origin.

Local branches are referenced by name e.g. main

flowchart TD
    Remote[origin/main] --> Local
    Local[main] --> Remote

Pull

To update the local repo to match the remote, use pull

flowchart TD
    Remote[origin/main] -->|pull| Local
    Local[main] --> Remote

git pull

Push

To update the remote repo to match the local, use push

flowchart TD
    Remote[origin/main] --> Local
    Local[main] --> |push|Remote

git push

Git operations

image courtesy of Reddit

References

Git - User-Manual Documentation

Git Merge

Warning

Do not perform pull or a merge with uncommitted changes.

merge is designed to combine committed changes. You can lose work.

How does merge work

E is the last shared commit.

Merge will play back the commits on both branches. If neither set of commits touches the same parts of the file, the branches are merged, and HEAD moves to the end.

gitGraph
   commit id: "D"
   commit id: "E"
   branch feature
   checkout feature
   commit id: "A"
   commit id: "B"
   commit id: "MERGE_HEAD → C"
   checkout main
   commit id: "F"
   commit id: "HEAD → G"

After the merge

gitGraph
   commit id: "D"
   commit id: "E"
   branch feature
   checkout feature
   commit id: "A"
   commit id: "B"
   commit id: "C"
   checkout main
   commit id: "F"
   commit id: "G"
   merge feature id: "HEAD → H"

Merge types

Fast forward

The default merge is called fast forward, or FF.

FF can be used when there are no local changes or local commits.

The repo was cloned previously.

Before fetch.

gitGraph
   commit id: "A"
   commit id: "B"
   commit id: "HEAD → C"

After fetch.

gitGraph
   commit id: "A"
   commit id: "B"
   commit id: "HEAD → C"
   commit id: "D"
   commit id: "E"

After merge.

git merge origin/main

gitGraph
   commit id: "A"
   commit id: "B"
   commit id: "C"
   commit id: "D"
   commit id: "HEAD → E"

Pull

pull combines fetch and merge. pull assumes a fast forward merge.

Before pull.

gitGraph
   commit id: "A"
   commit id: "B"
   commit id: "HEAD → C"

After pull.

gitGraph
   commit id: "A"
   commit id: "B"
   commit id: "C"
   commit id: "D"
   commit id: "HEAD → E"

True merge

This happens trying to update the part of a file, someone else has already updated.

1. HEAD pointer stays the same

2. MERGE_HEAD pointer on the branch to be merged.

3. What can be merged cleanly is merged.

4. Index records three versions of the files. Ancestor, HEAD and MERGE_HEAD

   • Files are merged in the working directory with conflict markers <<<<<<, =======, >>>>>>`

5. A ref called AUTO_MERGE gets created.

Conflict resolution

After the merge, Git will say what files need to be resolved. This is called Conflict Resolution.

To resolve:

1. Open the file.
2. Find the Markers.
3. Delete lines you don’t want.
4. Delete the markers.

Two way conflict

In Conflict.

                   Here are lines that are either unchanged from the common        
                   ancestor, or cleanly resolved because only one side changed,    
                   or cleanly resolved because both sides changed the same way.    
                   <<<<<<< yours:sample.txt                                        
Your change  ┌───  Conflict resolution is hard;                                    
             └───  let's go shopping.                                              
                   =======                                                         
    Theirs    ───  Git makes conflict resolution easy.                             
                   >>>>>>> theirs:sample.txt                                       
                   And here is another line that is cleanly resolved or unmodified.

Resolved, kept local change.

                   Here are lines that are either unchanged from the common        
                   ancestor, or cleanly resolved because only one side changed,    
                   or cleanly resolved because both sides changed the same way.    
Your change  ┌───  Conflict resolution is hard;                                    
             └───  let's go shopping.                                              
                   And here is another line that is cleanly resolved or unmodified.

Three way conflict

zdiff3 shows the conflict with the original text, adding the ||||||| marker.

In Conflict

                 Here are lines that are either unchanged from the common        
                 ancestor, or cleanly resolved because only one side changed,    
                 or cleanly resolved because both sides changed the same way.    
                 <<<<<<< yours:sample.txt                                        
      Yours ┌──  Conflict resolution is hard;                                    
            └──  let's go shopping.                                              
                 ||||||| base:sample.txt                                         
Original    ┌──  or cleanly resolved because both sides changed identically.     
 (Ancestor) └──  Conflict resolution is hard.                                    
                 =======                                                         
     Theirs  ──  Git makes conflict resolution easy.                             
                 >>>>>>> theirs:sample.txt                                       
                 And here is another line that is cleanly resolved or unmodified.

Resolved, kept their lines.

                 Here are lines that are either unchanged from the common        
                 ancestor, or cleanly resolved because only one side changed,    
                 or cleanly resolved because both sides changed the same way.    
     Theirs  ──  Git makes conflict resolution easy.                             
                 And here is another line that is cleanly resolved or unmodified.

Conflicts resolved

This will also check the merge status.

git merge --continue

Abort A merge

merge --abort

References

Git - git-merge Documentation

Git Branch

Always name branches as lowercase, as case-sensitive branches do not work across all operating systems.

Create a new branch

git branch --copy new-branch

Create a branch from a specific branch

git branch --copy new-branch old-branch

Delete a branch

Only works if it’s fully merged.

git branch --delete dev

Delete a remote branch

Only works if it’s fully merged.

git branch --delete --remotes dev

Move a branch

git branch --move old-branch new-branch

See all branches

git branch --all

References

github - Git branch name - case sensitive or insensitive? - Stack Overflow

Git - git-branch Documentation

Git Switch

This command was added to Git in 2019 (2.23), after user feedback that checkout was destructive in some cases.

switch updates the working tree, and index to match the new branch.

Important

The destructive form of this command is --discard-changes.

Make A new branch

git switch -c newbranch

Make A new branch from A specific commit

git switch -c newbranch <commit>

References

Git - git-switch Documentation

Git Push

Git has a default setup for where it plans to perform the push. This is the push.default.

The push.default contains the refspec.

Refspec

The refspec maps a local branch to a remote branch.

This is inside of .git/config

[remote "origin"]
        url = git@github.com:ariadne-notes/network-notes.git
        fetch = +refs/heads/*:refs/remotes/origin/*

Format of refspec

This is the fetch line from above.

• Format: +<src>:<dst>
  • + force update (non-fast-forward allowed)
  • <src> source ref
  • <dst> destination ref

Updating history

Making a change on the remote, to the commit history itself requires force.

The safer way to do this is per-branch, not the entire repo.

git push origin +main

References

Git - git-push Documentation

Git - The Refspec

Git Log

Top down, the normal view.

git log --oneline

Bottom up, good for rebasing.

git log --oneline --reverse

Decorate with active branches.

git log --oneline --decorate --graph

Decorate with all branches, even rebased ones.

git log --oneline --decorate --graph --all

Before rebasing

This can help find a commit in the event you need it again.

git log --reverse --oneline --date=short --format="%ad | %h | %an | %s" > commit-before-rebasing.txt

References

Git - git-log Documentation

Git Rebase

Caution

rebase modifies history.

Do not rebase commits that others may have based work on.

Two branches, the start.

---
config:
  gitGraph:
    parallelCommits: true
---
gitGraph LR:
   commit id: "A"
   commit id: "B"
   commit id: "C"
   branch dev
   checkout dev
   commit id: "HEAD → E"
   checkout main
   commit id: "HEAD → D"

Normally, with two branches, we’d do a merge

Merging

---
config:
  gitGraph:
    parallelCommits: true
---
gitGraph LR:
   commit id: "A"
   commit id: "B"
   commit id: "C"
   branch dev
   checkout dev
   commit id: "HEAD → E"
   checkout main
   commit id: "D"
   merge dev id: "HEAD → F"

F is the commit the combines the diff of the endpoints of the branches main and dev.

The dev branch is just hanging out.

Rebasing

---
config:
  gitGraph:
    parallelCommits: true
---
gitGraph LR:
   commit id: "A"
   commit id: "B"
   commit id: "C"
   branch dev
   checkout dev
   commit id: "HEAD → E"
   checkout main
   commit id: "HEAD → D"

Finding the common ancestor to both branches, C we go “This is the common base, just play the diffs forward from both branches onto main”

git checkout dev
git rebase main

---
config:
  gitGraph:
    parallelCommits: true
---
gitGraph LR:
   commit id: "A"
   commit id: "B"
   commit id: "C"
   commit id: "HEAD/main → D"
   commit id: "HEAD/dev → E"

Now the HEAD for two branches main and dev are on the same branch. Main can be FF’d to move the HEAD.

git checkout main
git merge dev

---
config:
  gitGraph:
    parallelCommits: true
---
gitGraph LR:
   commit id: "A"
   commit id: "B"
   commit id: "C"
   commit id: "D"
   commit id: "HEAD/main, dev → E"

References

Git - Rebasing

Git - git-rebase Documentation

Git Reset

Reset changes what commit HEAD points to. This is one way to undo a bad local commit, or a series of bad local commits.

Reset always modifies history.

Soft reset

AKA, Squashing

• Move HEAD.
• Do not modify the Index.
• Do not modify the Working Tree.

This is useful to undo local commits then re-play them as one commit.

e.g, you don’t need to git add the files are already in the Index.

git reset --soft HEAD~3
git commit

Mixed reset

Creates Intermediate Commits.

• Move HEAD.
• Reset Index to HEAD
• Do not modify the Working Tree.

Maybe you made 5 local commits but you’d prefer if it was 2.

git reset --mixed HEAD~5
git add file1.c
git add file2.c
git commit -m "Intermediate Commit 1"
git add file3.c
git add file4.c
git commit -m "Final commit"

Hard reset

Caution

This erases uncommitted work. This creates dangling local commits.

• Move HEAD
• Reset Index to HEAD
• Reset Working Tree to HEAD

git fetch origin
git reset --hard origin/main

References

Git - git-reset Documentation

Git Remote

Necessary when cloning a repo I own usually.

Show current remotes

ariadne@tesseract:~/git/pushpopswap$ git remote -v
origin  git@github.com:ariadne-notes/pushpopswap.git (fetch)
origin  git@github.com:ariadne-notes/pushpopswap.git (push)

Set remote

git remote set-url origin git@github.com:ariadne-notes/pushpopswap.git

References

Git - git-remote Documentation

Git RM

Used to remove files from the Index, in the event they are deleted from the working tree, and need to be deleted from future commits.

Example

I’ve already deleted this file from the Working Tree, now I want to remove it from the Index.

ariadne@tesseract:~/git/network-notes$ git status
On branch main
Your branch is up to date with 'origin/main'.

Changes not staged for commit:
  (use "git add/rm <file>..." to update what will be committed)
  (use "git restore <file>..." to discard changes in working directory)
        modified:   .github/workflows/mdbook.yml
        modified:   book.toml
        deleted:    src/New Text Document.txt

git rm --cached src/New Text Document.txt

References

Git - git-rm Documentation

Git Commit

Ordinary commit

git commit -m "Here is what I did"

Amend - change the last commit message

Warning

Do not amend commits that have already been published.

git commit --amend -m "This is what I wished it said"

All

• Automatically stages all files known to the Index
• Does not add new files

git commit --all -m "Used when you don't want to manually add files

References

Git - git-commit Documentation

Git Restore

This command was added to Git in 2019 (2.23), after user feedback that checkout was destructive in some cases.

Important

These are destructive commands.

These commands should be used with -- otherwise Git doesn’t know if you want a branch, directory, or file.

-- means file.

Restore from index

• Copy a file to the working tree from the Index.

A file is staged.

git add hello.c

Work happens (in the working tree) now this file is broken.

git restore -- hello.c

Restore the index

• Copy a file to the Index from the current commit..

This “unstages” a file, the version in HEAD is restored to Index.

Equivalent to undoing git add

git restore --staged -- hello.c

Restore a file to working tree from origin

git restore --source origin/main -- hello.c

References

Git - git-restore Documentation

What is git restore and how is it different from git reset? - Stack Overflow

Git Ignore

The .gitignore file is useful for things that should not be included in Git.

# - Comments

/ - Directories, the first / is the root of the repo.

Example

This ignores the file itself, and the directory /book which is the build artifacts when mdbook is run.

ariadne@tesseract:~/git/network-notes$ cat .gitignore 
.gitignore
/book

References

Git - gitignore Documentation

Git Hub Commands

Creating the repo on GitHub

This is done inside the directory after it’s created.

Auth needs to be already working.

gh repo create ariadne-notes/wifi-charts  --remote=origin --public --push --source=.

References

GitHub CLI | Take GitHub to the command line

TCP

Connection oriented.

Header

 0                   1                   2                   3    
 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1  
┌───────────────────────────────┬───────────────────────────────┐ 
│          Source Port          │       Destination Port        │ 
├───────────────────────────────┴───────────────────────────────┤ 
│                        Sequence Number                        │ 
├───────────────────────────────────────────────────────────────┤ 
│                    Acknowledgment Number                      │ 
├───────┬───────┬─┬─┬─┬─┬─┬─┬─┬─┬───────────────────────────────┤ 
│  Data │       │C│E│U│A│P│R│S│F│                               │ 
│ Offset│ Rsrvd │W│C│R│C│S│S│Y│I│            Window             │ 
│       │       │R│E│G│K│H│T│N│N│                               │ 
├───────┴───────┴─┴─┴─┴─┴─┴─┴─┴─┼───────────────────────────────┤ 
│           Checksum            │         Urgent Pointer        │ 
├───────────────────────────────┴───────────────────────────────┤ 
│                           [Options]                           │ 
├───────────────────────────────────────────────────────────────┤ 
│                                                               │ 
│                             Data                              │ 
│                                                               │ 
└───────────────────────────────────────────────────────────────┘ 

Flags

• Congestion Window Reduced
• ECN-Echo
• Urgent
• Acknowledgment
• Push
• Reset
• Synchronize
• Final

References

RFC 9293: Transmission Control Protocol (TCP) | RFC Editor

UDP

UDP checksum

UDP does try to send error-free packets by including a checksum, the below via the RFC

Checksum is the 16-bit one’s complement of the one’s complement sum of a pseudo header of information from the IP header, the UDP header, and the data, padded with zero octets at the end (if necessary) to make a multiple of two octets.

…

If the computed checksum is zero, it is transmitted as all ones (the equivalent in one’s complement arithmetic). An all zero transmitted checksum value means that the transmitter generated no checksum (for debugging or for higher level protocols that don’t care).

UDP header

 0                   1                   2                   3
 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1
┌────────────────────────────────┬───────────────────────────────┐
│          Source Port           │       Destination Port        │
├────────────────────────────────┼───────────────────────────────┤
│            Length              │           Checksum            │
├────────────────────────────────┴───────────────────────────────┘
│          Data Octets
└────────────────────────────────►

TFTP read request

Frame 115: 69 bytes on wire (552 bits), 69 bytes captured (552 bits) on interface -, id 0
    Internet Protocol Version 4, Src: 10.0.10.22, Dst: 10.0.10.33
    User Datagram Protocol, Src Port: 52775, Dst Port: 69
        Source Port: 52775
        Destination Port: 69
        Length: 31
        Checksum: 0x4aed [correct]
        [Checksum Status: Good]
        [Stream index: 0]
        [Timestamps]
        UDP payload (23 bytes)
    Trivial File Transfer Protocol
        Opcode: Read Request (1)
        Source File: startup-config
        Type: octet

TFTP data packet

Frame 116: 562 bytes on wire (4496 bits), 562 bytes captured (4496 bits) on interface
    Internet Protocol Version 4, Src: 10.0.10.33, Dst: 10.0.10.22
    User Datagram Protocol, Src Port: 52590, Dst Port: 52775
        Source Port: 52590
        Destination Port: 52775
        Length: 524
        Checksum: 0xde83 [correct]
        [Checksum Status: Good]
        [Stream index: 1]
        [Timestamps]
        UDP payload (516 bytes)
    Trivial File Transfer Protocol
        Opcode: Data Packet (3)
        [Destination File: startup-config]
        [Read Request in frame 115]
        Block: 1
        [Full Block Number: 1]
    Data (512 bytes)
    
    0000  0a 21 0a 21 20 4c 61 73 74 20 63 6f 6e 66 69 67   .!.! Last config
    0010  75 72 61 74 69 6f 6e 20 63 68 61 6e 67 65 20 61   uration change a
    0020  74 20 30 35 3a 31 31 3a 31 35 20 55 54 43 20 53   t 05:11:15 UTC S
    0030  61 74 20 4a 75 6c 20 38 20 32 30 32 33 0a 21 0a   at Jul 8 2023.!.
    0040  76 65 72 73 69 6f 6e 20 31 35 2e 32 0a 73 65 72   version 15.2.ser
    0050  76 69 63 65 20 74 69 6d 65 73 74 61 6d 70 73 20   vice timestamps 
    0060  64 65 62 75 67 20 64 61 74 65 74 69 6d 65 20 6d   debug datetime m
    0070  73 65 63 0a 73 65 72 76 69 63 65 20 74 69 6d 65   sec.service time
    0080  73 74 61 6d 70 73 20 6c 6f 67 20 64 61 74 65 74   stamps log datet
    0090  69 6d 65 20 6d 73 65 63 0a 6e 6f 20 73 65 72 76   ime msec.no serv
    00a0  69 63 65 20 70 61 73 73 77 6f 72 64 2d 65 6e 63   ice password-enc
    00b0  72 79 70 74 69 6f 6e 0a 73 65 72 76 69 63 65 20   ryption.service 
    00c0  63 6f 6d 70 72 65 73 73 2d 63 6f 6e 66 69 67 0a   compress-config.
    00d0  21 0a 68 6f 73 74 6e 61 6d 65 20 53 57 33 0a 21   !.hostname SW3.!
    00e0  0a 62 6f 6f 74 2d 73 74 61 72 74 2d 6d 61 72 6b   .boot-start-mark
    00f0  65 72 0a 62 6f 6f 74 2d 65 6e 64 2d 6d 61 72 6b   er.boot-end-mark
    0100  65 72 0a 21 0a 21 0a 6c 6f 67 67 69 6e 67 20 64   er.!.!.logging d
    0110  69 73 63 72 69 6d 69 6e 61 74 6f 72 20 45 58 43   iscriminator EXC
    0120  45 53 53 20 73 65 76 65 72 69 74 79 20 64 72 6f   ESS severity dro
    0130  70 73 20 36 20 6d 73 67 2d 62 6f 64 79 20 64 72   ps 6 msg-body dr
    0140  6f 70 73 20 45 58 43 45 53 53 43 4f 4c 4c 20 0a   ops EXCESSCOLL .
    0150  6c 6f 67 67 69 6e 67 20 62 75 66 66 65 72 65 64   logging buffered
    0160  20 35 30 30 30 30 0a 6c 6f 67 67 69 6e 67 20 63    50000.logging c
    0170  6f 6e 73 6f 6c 65 20 64 69 73 63 72 69 6d 69 6e   onsole discrimin
    0180  61 74 6f 72 20 45 58 43 45 53 53 0a 21 0a 6e 6f   ator EXCESS.!.no
    0190  20 61 61 61 20 6e 65 77 2d 6d 6f 64 65 6c 0a 21    aaa new-model.!
    01a0  0a 21 0a 21 0a 21 0a 21 0a 6e 6f 20 69 70 20 69   .!.!.!.!.no ip i
    01b0  63 6d 70 20 72 61 74 65 2d 6c 69 6d 69 74 20 75   cmp rate-limit u
    01c0  6e 72 65 61 63 68 61 62 6c 65 0a 21 0a 21 0a 21   nreachable.!.!.!
    01d0  0a 6e 6f 20 69 70 20 64 6f 6d 61 69 6e 2d 6c 6f   .no ip domain-lo
    01e0  6f 6b 75 70 0a 69 70 20 63 65 66 0a 6e 6f 20 69   okup.ip cef.no i
    01f0  70 76 36 20 63 65 66 0a 21 0a 21 0a 21 0a 73 70   pv6 cef.!.!.!.sp

References

User Datagram Protocol - RFC 768

Cisco Traceroute

I wrote this trying to understand why my lab equipment had missing ICMP packets.

The answer is ICMP Rate Limiting, but it also covers UDP behavior and how the ports are allocated.

Missing ICMP replies

Topology

10.0.0.1/32     10.0.0.2/32     10.0.0.3/32      10.0.0.4/32       10.0.0.5/32 
   │               │               │                │                 │        
   │ 10.1.12.0/24  │ 10.3.23.0/24  │  10.1.34.0/24  │  10.3.45.0/24   │        
   ▼       │       ▼       │       ▼        │       ▼        │        ▼        
┌────┐.1   ▼  .2┌────┐.2   ▼  .3┌────┐.3    ▼  .4┌────┐.4    ▼   .5┌────┐      
│    ├──────────┤    ├──────────┤    ├───────────┤    ├────────────┤    │      
│ R1 │          │ R2 │          │ R3 │           │ R4 │            │ R5 │      
│    ├──────────┤    ├──────────┤    ├───────────┤    ├────────────┤    │      
└────┘.1   ▲  .2└────┘.2   ▲  .3└────┘.3    ▲  .4└────┘.4    ▲   .5└────┘      
           │               │                │                │                 
     10.2.12.0/24    10.4.23.0/24     10.2.34.0/24     10.4.45.0/24            

• SW: IOS-XE 17.13.01a
• HW: C8000v
• Backend: Enterprise CML
• Single area OSPFv2
• 13 subnets
• Ping works

The Drops

R1# traceroute 10.0.0.5 source 10.0.0.1 probe 5

  1 10.2.12.2 2 msec
    10.1.12.2 1 msec
    10.2.12.2 2 msec
    10.1.12.2 2 msec
    10.2.12.2 1 msec
  2 10.4.23.3 4 msec 3 msec 2 msec 2 msec 3 msec
  3 10.2.34.4 4 msec 4 msec 4 msec 4 msec 4 msec
  4 10.4.45.5 5 msec 5 msec *  5 msec * 

Captures are taken between R1 and R2. The output above is from a probe 5 run; the captures use the default 3 probes per hop.

• cisco-traceroute-link-1-dns-missing.pcap
• cisco-traceroute-link-2-dns-missing.pcap

Cisco traceroute uses UDP

The router does not send ICMP packets. It’s purely UDP.

Sent

Frame 1: Packet, 42 bytes on wire (336 bits), 42 bytes captured (336 bits)
Ethernet II, Src: 52:54:00:d8:d7:63 (52:54:00:d8:d7:63), Dst: 52:54:00:81:12:6f (52:54:00:81:12:6f)
Internet Protocol Version 4, Src: 10.0.0.1 (10.0.0.1), Dst: 10.0.0.5 (10.0.0.5)
User Datagram Protocol, Src Port: 49435 (49435), Dst Port: mtrace (33435)

In response it gets an ICMP TTL message.

Received

Frame 2: Packet, 70 bytes on wire (560 bits), 70 bytes captured (560 bits)
Ethernet II, Src: 52:54:00:81:12:6f (52:54:00:81:12:6f), Dst: 52:54:00:d8:d7:63 (52:54:00:d8:d7:63)
Internet Protocol Version 4, Src: 10.1.12.2 (10.1.12.2), Dst: 10.0.0.1 (10.0.0.1)
Internet Control Message Protocol
    Type: Time-to-live exceeded (11)
    Code: 0 (Time to live exceeded in transit)
    Checksum: 0x091f [correct]
    [Checksum Status: Good]
    Unused: 00000000
    Internet Protocol Version 4, Src: 10.0.0.1 (10.0.0.1), Dst: 10.0.0.5 (10.0.0.5)
    User Datagram Protocol, Src Port: 49435 (49435), Dst Port: mtrace (33435)

The payload of the ICMP message is the original UDP packet.

It increments the src and dst UDP ports

• udp.srcport looks like it starts at a randomized ephemeral port and increments for each probe
• udp.dstport starts at 33434 and increments for each probe
• udp.dstport as a filter will find both the UDP probe and the ICMP reply

This means that each probe pair (the probe and its reply) has a unique and sequential port number.

Each probe set has 1 added to its TTL.

!
! Set 1
!
Probe 1 - udp.dstport == 33434 && udp.srcport == 49305 && ip.ttl == 1
Probe 2 - udp.dstport == 33435 && udp.srcport == 49306 && ip.ttl == 1
Probe 3 - udp.dstport == 33436 && udp.srcport == 49307 && ip.ttl == 1
!
! Set 2
!
Probe 1 - udp.dstport == 33437 && udp.srcport == 49308 && ip.ttl == 2
Probe 2 - udp.dstport == 33438 && udp.srcport == 49309 && ip.ttl == 2
Probe 3 - udp.dstport == 33439 && udp.srcport == 49310 && ip.ttl == 2
!
! Set 3
!
Probe 1 - udp.dstport == 33440 && udp.srcport == 49311 && ip.ttl == 3
Probe 2 - udp.dstport == 33441 && udp.srcport == 49312 && ip.ttl == 3
Probe 3 - udp.dstport == 33442 && udp.srcport == 49313 && ip.ttl == 3
!
! Set 4
!
Probe 1 - udp.dstport == 33443 && udp.srcport == 49314 && ip.ttl == 4
Probe 2 - udp.dstport == 33444 && udp.srcport == 49315 && ip.ttl == 4
Probe 3 - udp.dstport == 33445 && udp.srcport == 49316 && ip.ttl == 4

Pathing

The UDP traffic has unique udp.srcport and udp.dstport per probe.

The returning ICMP traffic (Type 11 from the hops, Type 3 from the destination) is much more constrained.

Important

Each router decides per flow how to route both kinds of packets: udp & icmp

Packets may not return via the path they arrived.

Cisco Traceroute Uses DNS

It works differently depending on DNS.

DNS is on, but broken

The first four packets are 8 seconds apart. The router requests PTR records for each IP.

It waits for those PTR queries.

12 probes takes about 50 seconds. I get the * in the results.

• cisco-traceroute-link-1-dns-missing.pcap
• cisco-traceroute-link-2-dns-missing.pcap
• cisco-traceroute-merged-dns-missing.pcap

Towards 8.8.8.8 with DNS enabled and working

Note

Exact same router.

The behavior changes when I connect the router to my lab network, and it gets a DHCP address and can reach the open Internet.

30+ probes complete in a little under 2 seconds. DNS is involved.

Disabling DNS lookups

The delay is reverse DNS: the captures show PTR queries for each newly seen hop address timing out between probes.

I added no ip domain lookup, it’s much faster but I still get the * in the results.

• cisco-traceroute-link-1-dns-disabled.pcap
• cisco-traceroute-link-2-dns-disabled.pcap
• cisco-traceroute-merged-dns-disabled.pcap

ICMP Type 3 Rate Limiting

IOS-XE defaults to 1 ICMP type 3 reply every 500 ms.

These replies are what traceroute needs to show a time result.

R1# show run all | i icmp rate-limit
ip icmp rate-limit unreachable 500

Show dropped replies

Looking at the captures, cisco-traceroute-merged-dns-missing.pcap and cisco-traceroute-merged-dns-disabled.pcap we do see the UDP probes arrive very close together.

If two probes come in too close together, the router will not reply.

Note

These drops will not show up with debug icmp

R5# show ip icmp rate-limit 

                           DF bit unreachables       All other unreachables   
Interval (millisecond)     500                       500                      

Interface                  # DF bit unreachables     # All other unreachables 
---------                  ---------------------     ------------------------ 
GigabitEthernet3           0                         0                        
GigabitEthernet4           0                         1                        
Loopback0                  0                         0   

References

Use the Traceroute Command on Operating Systems - Cisco

IPv4

The 2004 Turing award was given to Vinton Cerf and Robert Kahn for the invention of TCP/IP.

Concepts from the 1974 IEEE paper:

• Gateways
  • Separate different L2 networks
• TCP Port Multiplexing
• Connection Oriented
• Sequence Numbers
• Flow Control
• Retransmission

References

History of the Internet - Internet Society

IEN 54 - Internetwork Protocol Specification Version 4

A Protocol for Packet Network Intercommunication - IEEE Transactions on Communications

IPv4 - Wikipedia

RFC 791: Internet Protocol | RFC Editor

IPv4 Addressing

Address ranges

This Network

The 0.0.0.0 address literally means “when an app requests connectivity to 0.0.0.0, bind to every interface running IP and make it work”.

References

How Class E addresses solve for IP address exhaustion in GKE | Google Cloud Blog

IPv4 Packet Header

 0                   1                   2                   3
 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1
┌───────┬───────┬───────────────┬───────────────────────────────┐
│Version│  IHL  │Type of Service│          Total Length         │
├───────┴───────┴───────────────┼─────┬─────────────────────────┤
│         Identification        │Flags│     Fragment Offset     │
├───────────────┬───────────────┼─────┴─────────────────────────┤
│  Time to Live │    Protocol   │        Header Checksum        │
├───────────────┴───────────────┴───────────────────────────────┤
│                         Source Address                        │
├───────────────────────────────────────────────────────────────┤
│                      Destination Address                      │
├───────────────────────────────────────────────┬───────────────┤
│                    Options                    │    Padding    │
└───────────────────────────────────────────────┴───────────────┘

Fields

Flags

  Flags:  3 bits

    Various Control Flags.

      Bit 0: reserved, must be zero
      Bit 1: (DF) 0 = May Fragment,  1 = Don't Fragment.
      Bit 2: (MF) 0 = Last Fragment, 1 = More Fragments.

          0   1   2
        +---+---+---+
        |   | D | M |
        | 0 | F | F |
        +---+---+---+

References

RFC 791: Internet Protocol | RFC Editor

IPv4 Address Planning

1. Define the requirements
2. Plan the ipv4 range
3. Document the plan

Example standards

• Statically assign network infrastructure
• User devices are DHCP.
• .1 is the default gateway
• Third Octet: The vlan ID. 10.0.150.0/24 is vlan 150.
• Forth Octet: Address Assignment type. 1 to 99 are static IPs, 100-200 are DHCP.

Figure out NAT

• How many IPs are needed?
• Do any devices need to be reachable from the outside?

Best practices

• Address internal hosts with RFC1918 addresses.

Recommended networks

References

IPv4 Interactive Map - bgp.tools

IPv4 Subnets

All IPv4 addresses are 32-bits, or four groups of eight.

We call each group of eight an octet.

An IPv4 Address in binary looks like this.

0000 0000 . 0000 0000 . 0000 0000 . 0000 0000

Some portion of those 32-bits are network bits N, and another portion are host bits H.

A /24 is 24 n-bits and 8 h-bits.

nnnn nnnn . nnnn nnnn . nnnn nnnn . hhhh hhhh

Since classful networking is dead, unless we are told the subnet mask, we cannot guess what the subnet size is from an IP

Terms

N-bit — Network Bit

• Identifies the network portion of a v4 address

H-bit — Host Bit

• Identifies the host portion of a v4 address

Via octet

We can subnet anywhere from /0 to /32. Even /31 is valid.

When a portion of the 32-bit space is used to make a subnet mask, we call these bits borrowed.

| Borrowed Bits | Binary | Decimal | Masks |

| –––––––|–––––| ––––| ––––––––––| | 0 | 00000000 | 0 | /0, /8, /16, /24 | | 1 | 10000000 | 128 | /1, /9, /17, /25 | | 2 | 11000000 | 192 | /2, /10, /18, /26 | | 3 | 11100000 | 224 | /3, /11, /19, /27 | | 4 | 11110000 | 240 | /4, /12, /20, /28 | | 5 | 11111000 | 248 | /5, /13, /21, /29 | | 6 | 11111100 | 252 | /6, /14, /22, /30 | | 7 | 11111110 | 254 | /7, /15, /23, /31 | | 8 | 11111111 | 255 | /8, /16, /24, /32 |

As powers of 2

Used to find subnets or the number of IPs.

For hosts use: \( 2^x - 2 \)

The default route, i.e. the whole Internet is /0.

Config

A terrible and ancient way to load balance showing how subnet masks work.

!
! Use two /1s, impress your friends
!
! Useful to override your default network, I see wireguard uses this occasionally.
!
ip route 0.0.0.0   128.0.0.0 10.0.0.1
ip route 128.0.0.0 128.0.0.0 10.0.0.2

Subnetting with Fingers

I just memorize these sequences, ungainly, but works.

Decimal masks - 128, 192, 224, 240, 248, 252, 254, 255

Wildcard masks - 127, 63, 31, 15, 7, 3, 1, 0

Subnet Sizes (going up) - 256, 512, 1024, 2048

Subnet Sizes (going down) - 128, 64, 32, 16, 8, 4, 2, 1

Subnet examples

/24 is 256 IPs. Most gear complains if you use .0 or .256, so we say 254 usable hosts.

/30 is the 1990s way of addressing a point-to-point link, which wastes two IPs.

/31 is exactly two IPs. This is the best subnet for point-to-point links.

/32 is a single address. We call these host routes. 8.8.8.8/32 is Google’s DNS.

References

Using 31-Bit Prefixes on IPv4 Point-to-Point Links

Subnetting with the Box Method

For visual learners.

For a /24 network, thats 256 addresses, or … this box.

Start with A /24

Write the first IP in the top corner, the last IP in the bottom corner.

Cut in half (1)

Write the last number in the bottom corner, next number in the top corner.

Two /25 networks.

Cut in half (2)

• One /25
• Two /26 networks.

Cut in half (3)

• One /25
• Four /27

Cut in half (4)

• One /25
• Two /27
• Four /28

Subnetting with Binary

• A bit is on or off
• A nibble is four bits 0000
• A byte is eight bits 0000 0000

Powers of 2: 1, 2, 4, 8, 16, 32, 64, 128, 256, 512, 1024, 2048

Examples

Start with a /24, eight free h bits, or values from 0 to 256.

0000 0000

100 hosts

A subnet needs 100 hosts, to represent that in binary, we’d need at least 7 bits, or 128.

A /24 would be too large, 256 IPs, remove a bit, and we get a /25.

10.0.0.0/25.

20 more hosts

We already spent the first portion of the address space, from 0 to 127.

0000 0000 is 0.

1000 0000 is 128.

How many bits to represent 20 hosts? at least 5, or \( 2^5 = 32-2 = 30 \)

So we start with 128, and use at least five bits.

1000 1000

References

Cisco - Binary Game

Classful Networking

Never speak out loud “This is a class C address” unless you are in a 1990s movie.

Early Internet addressing (1980s) the IP itself indicated the subnet mask, by using the High Order bits. There were only three network sizes.

Internet Address starts with 0-127? You must have one network with 16 million hosts.

Internet Address starts with 128-191? You must have one network with 65 thousand hosts.

Internet Address starts with 192-223? You must have one network with 254 hosts.

We know now this was a terrible idea, but parts of it stay with us today.

• /24 (from class C) is a very popular prefix. This is said out loud as “A twenty four”.
• /16 (from class B) is a very popular prefix. This is said out loud as “A sixteen”.
• All v4 multicast addresses still start with 1110

If someone gives you an IP, you can never guess its network without also being told the mask.

My personal favorite is 8.8.8.8/32 Google DNS it works by Anycast!

Classful networking, the RFC

RFC 791

Internet Protocol
Specification

  Addressing

    To provide for flexibility in assigning address to networks and
    allow for the  large number of small to intermediate sized networks
    the interpretation of the address field is coded to specify a small
    number of networks with a large number of host, a moderate number of
    networks with a moderate number of hosts, and a large number of
    networks with a small number of hosts.  In addition there is an
    escape code for extended addressing mode.

    Address Formats:

      High Order Bits   Format                           Class
      ---------------   -------------------------------  -----
            0            7 bits of net, 24 bits of host    a
            10          14 bits of net, 16 bits of host    b
            110         21 bits of net,  8 bits of host    c
            111         escape to extended addressing mode

ICMP

ICMP — Internet Control Message Protocol

ICMP is frequently used in network troubleshooting.

References

RFC 792: Internet Control Message Protocol | RFC Editor

IANA: Internet Control Message Protocol (ICMP) Parameters

Ping

ICMP — Internet Control Message Protocol

Ping tests IPv4 reachability and measures round-trip time using ICMP Echo messages.

Operation

┌──────┐    10.0.0.0/24    ┌──────┐
│  R1  ├───────────────────┤  R2  │
└──────┘.1               .2└──────┘

Verification

Sending the ping.

R1# ping 10.0.0.2
Sending 5, 100-byte ICMP Echos to 10.0.0.2, timeout is 2 seconds:
!!!!!
Success rate is 100 percent (5/5), round-trip min/avg/max = 1/1/2 ms

Counters on R1.

R1# show ip traffic | s ICMP      
ICMP statistics:
  Rcvd: 0 format errors, 0 checksum errors, 0 redirects, 0 unreachable
        0 echo, 5 echo reply, 0 mask requests, 0 mask replies, 0 quench
        0 parameter, 0 timestamp, 0 timestamp replies, 0 info request, 0 other
        0 irdp solicitations, 0 irdp advertisements
        0 time exceeded, 0 info replies
  Sent: 0 redirects, 0 unreachable, 5 echo, 0 echo reply
        0 mask requests, 0 mask replies, 0 quench, 0 timestamp, 0 timestamp replies
        0 info reply, 0 time exceeded, 0 parameter problem
        0 irdp solicitations, 0 irdp advertisements

Counters on R2.

R2# show ip traffic | s ICMP statistics
ICMP statistics:
  Rcvd: 0 format errors, 0 checksum errors, 0 redirects, 0 unreachable
        5 echo, 0 echo reply, 0 mask requests, 0 mask replies, 0 quench
        0 parameter, 0 timestamp, 0 timestamp replies, 0 info request, 0 other
        0 irdp solicitations, 0 irdp advertisements
        0 time exceeded, 0 info replies
  Sent: 0 redirects, 0 unreachable, 0 echo, 5 echo reply
        0 mask requests, 0 mask replies, 0 quench, 0 timestamp, 0 timestamp replies
        0 info reply, 0 time exceeded, 0 parameter problem
        0 irdp solicitations, 0 irdp advertisements

As Hex

These can be pasted into an online packet decoder.

Echo request

52 54 00 40 e2 17 52 54 00 ff e7 79 08 00 45 00
00 64 00 00 00 00 ff 01 a7 96 0a 00 00 01 0a 00
00 02 08 00 95 aa 00 00 00 00 00 00 00 00 00 03
e8 9c ab cd ab cd ab cd ab cd ab cd ab cd ab cd
ab cd ab cd ab cd ab cd ab cd ab cd ab cd ab cd
ab cd ab cd ab cd ab cd ab cd ab cd ab cd ab cd
ab cd ab cd ab cd ab cd ab cd ab cd ab cd ab cd
ab cd                                          

Echo Reply

52 54 00 ff e7 79 52 54 00 40 e2 17 08 00 45 00
00 64 00 00 00 00 ff 01 a7 96 0a 00 00 02 0a 00
00 01 00 00 9d aa 00 00 00 00 00 00 00 00 00 03
e8 9c ab cd ab cd ab cd ab cd ab cd ab cd ab cd
ab cd ab cd ab cd ab cd ab cd ab cd ab cd ab cd
ab cd ab cd ab cd ab cd ab cd ab cd ab cd ab cd
ab cd ab cd ab cd ab cd ab cd ab cd ab cd ab cd
ab cd                                          

Packet Capture

ping-r1-to-r2.pcap

References

RFC 792: Internet Control Message Protocol | RFC Editor

RFC 1122: Requirements for Internet Hosts | RFC Editor

ICMP Type 8 Echo Request

ICMP — Internet Control Message Protocol

The message preceding an ICMP Type 0 Echo Reply.

Every host MUST implement an ICMP Echo server function that receives Echo Requests and sends corresponding Echo Replies.

Source RFC 1122.

Header

 0                   1                   2                   3
 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1
┌───────────────┬───────────────┬───────────────────────────────┐
│   Type = 8    │   Code = 0    │          Checksum             │
├───────────────┴───────────────┼───────────────────────────────┤
│          Identifier           │        Sequence Number        │
├───────────────────────────────┴───────────────────────────────┤
│                             Data                              │
└───────────────────────────────────────────────────────────────┘

Fields

Ping sends Echo Requests to test IP reachability and measure round-trip time. A missing reply does not prove the destination is down; ICMP may be filtered or rate-limited.

References

RFC 792: Internet Control Message Protocol | RFC Editor

RFC 1122: Requirements for Internet Hosts | RFC Editor

IANA: Internet Control Message Protocol (ICMP) Parameters

ICMP Type 0 Echo Reply

ICMP — Internet Control Message Protocol

The response to an ICMP Type 8 Echo Request.

The data received in the echo message must be returned in the echo reply message.

Source RFC 792.

Responses will have

• Same identifier
• Same sequence number
• Same data

Header

 0                   1                   2                   3
 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1
┌───────────────┬───────────────┬───────────────────────────────┐
│   Type = 0    │   Code = 0    │          Checksum             │
├───────────────┴───────────────┼───────────────────────────────┤
│          Identifier           │        Sequence Number        │
├───────────────────────────────┴───────────────────────────────┤
│                             Data                              │
└───────────────────────────────────────────────────────────────┘

References

RFC 792: Internet Control Message Protocol | RFC Editor

RFC 1122: Requirements for Internet Hosts | RFC Editor

IANA: Internet Control Message Protocol (ICMP) Parameters

ICMP Type 3 Destination Unreachable

ICMP — Internet Control Message Protocol

An ICMP Type 3 Destination Unreachable message explains why an IPv4 packet could not reach a network, host, protocol, or port.

If, in the destination host, the IP module cannot deliver the datagram because the indicated protocol module or process port is not active, the destination host may send a destination unreachable message to the source host.

Source RFC 792.

Header

 0                   1                   2                   3
 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1
┌───────────────┬───────────────┬───────────────────────────────┐
│     Type      │     Code      │          Checksum             │
├───────────────┴───────────────┴───────────────────────────────┤
│                             unused                            │
├───────────────────────────────────────────────────────────────┤
│      Internet Header + 64 bits of Original Data Datagram      │
└───────────────────────────────────────────────────────────────┘

Codes

References

RFC 792: Internet Control Message Protocol | RFC Editor

ICMP Type 11 Time Exceeded

ICMP — Internet Control Message Protocol

An ICMP Type 11 Time Exceeded message reports that a packet’s TTL expired in transit or that IPv4 fragment reassembly timed out.

If the gateway processing a datagram finds the time to live field is zero it must discard the datagram. The gateway may also notify the source host via the time exceeded message.

Source RFC 792.

Header

 0                   1                   2                   3
 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1
┌───────────────┬───────────────┬───────────────────────────────┐
│     Type      │     Code      │          Checksum             │
├───────────────┴───────────────┴───────────────────────────────┤
│                             unused                            │
├───────────────────────────────────────────────────────────────┤
│  Internet Header + at least 64 bits of Original Datagram Data │
└───────────────────────────────────────────────────────────────┘

Codes

Traceroute

Traceroute sends probes with increasing TTL values.

Example

A sent Time-to-live exceeded message.

Received

Ethernet II, Src: 52:54:00:81:12:6f (52:54:00:81:12:6f), Dst: 52:54:00:d8:d7:63 (52:54:00:d8:d7:63)
Internet Protocol Version 4, Src: 10.1.12.2 (10.1.12.2), Dst: 10.0.0.1 (10.0.0.1)
Internet Control Message Protocol
    Type: Time-to-live exceeded (11)
    Code: 0 (Time to live exceeded in transit)
    Internet Protocol Version 4, Src: 10.0.0.1 (10.0.0.1), Dst: 10.0.0.5 (10.0.0.5)
    User Datagram Protocol, Src Port: 49435 (49435), Dst Port: mtrace (33435)

References

RFC 792: Internet Control Message Protocol | RFC Editor

RFC 1122: Requirements for Internet Hosts | RFC Editor

IANA: Internet Control Message Protocol (ICMP) Parameters

IPv6

Fundamentals

• 128 bits
• Global Scoped address tend to be Globally Unique
  • Less need for NAT
• Fixed header length
• Optional, Option headers
• SLAAC
  • Stateless host addressing
  • Router advertising the subnet prefix
• Flow Labeling (for QoS)
• Routers cannot fragment packets
• Hosts perform MTU path discovery
• Hosts can have multiple addresses in multiple subnets
• Mobile IPv6 lets mobile nodes remain reachable
• No broadcast traffic
• No ARP

Bits

IPv6 address takes this form:

FFFF:FFFF:FFFF:FFFF:FFFF:FFFF:FFFF:FFFF

The groups are called hextets, as they are made with hex digits.

• FFFF
  • 16 bits
• FF
  • 8 bits
  • One byte
• F
  • 4 bits
  • A nibble

Header format

RFCs really like groups of 32.

IPv6 addresses are 4 groups of 32.

RFC 2460 Section 3

                     1                   2                   3
 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 
┌───────┬───────────────┬───────────────────────────────────────┐
│Version│ Traffic Class │           Flow Label                  │
├───────┴───────────────┴───────┬───────────────┬───────────────┤
│         Payload Length        │  Next Header  │   Hop Limit   │
├───────────────────────────────┴───────────────┴───────────────┤
│                                                               │
│                         Source Address                        │
│                                                               │
│                                                               │
├───────────────────────────────────────────────────────────────┤
│                                                               │
│                        Destination Address                    │
│                                                               │
│                                                               │
└───────────────────────────────────────────────────────────────┘

Address Blocks

IPv6 special address blocks

WKP — Well Known Prefix

This idea was from RFC 6052.

The idea was, you could look at a v6 address with 64 in the front and understand it had been translated.

These aren’t popular anymore

IPv4-Compatible IPv6 address

deprecated

RFC 4291 Section 2.5.5.1 provided this:

::10.0.0.1

These are only API addresses to represent to a IPv6 App, “this is actually a IPv4 host”

They don’t go anywhere.

IPv4-Mapped IPv6 address

RFC 4291 Section 2.5.5.2 provides this:

These are used in RFC 4038.

::FFFF:10.0.0.1

Intended for this flow where an IPv6 only app can work with IPv4 addresses.

┌──────────────────────────────────────────────┐
│ ┌──────────────────────────────────────────┐ │
│ │                                          │ │
│ │          IPv6-only applications          │ │
│ │                                          │ │
│ └────────────────────┬─────────────────────┘ │
│                      │                       │
│ ┌────────────────────┴─────────────────────┐ │
│ │                                          │ │
│ │   TCP / UDP / others (SCTP, DCCP, etc.)  │ │
│ │                                          │ │
│ └─────────────────┬────────┬───────────────┘ │
│    IPv4-mapped    │        │    IPv6         │
│  IPv6 addresses   │        │   addresses     │
│ ┌─────────────────┴──┐ ┌───┴───────────────┐ │
│ │        IPv4        │ │       IPv6        │ │
│ └────────────┬───────┘ └───────┬───────────┘ │
│   IPv4       │                 │             │
│   addresses  │                 │             │
└───────────── │ ─────────────── │ ────────────┘
               │                 │              
          IPv4 packets      IPv6 packets        

References

RFC 2460 - Internet Protocol, Version 6 (IPv6)

IPv6 address - Wikipedia

Cisco Live - IPv6:: It’s Happening - Nathan Sherrard - BRKIPV-2191

Cisco Live - Deploying IPv6 Routing Protocols - Peter Palúch - BRKIPV-2418

RFC 9099: Operational Security Considerations for IPv6 Networks

IPv6 Subnetting

Not really used by individual sites, but if you get a block from a RIR, this is how to subnet it.

In the US ARIN handles requests for v6 space.

Could be useful to work with NPTv6

Blessed subnets

The easy v6 networks to subnet fall on hex digit boundaries of /4.

For these, see this chart.

/40

You’re given 3fff::/20, make some /40 networks.

A /40 is two hextets, and two hex digits worth of bits.

• 3fff:0:0000::/40
• 3fff:0:0100::/40
• 3fff:0:0200::/40
• 3fff:0:0300::/40
• 3fff:0:0400::/40
• 3fff:0:0500::/40

which becomes

• 3fff:0::/40
• 3fff:0:100::/40
• 3fff:0:200::/40
• 3fff:0:300::/40
• 3fff:0:400::/40
• 3fff:0:500::/40

/44

You’re given 3fff::/20, make some /44 networks.

A /44 is two hextets, and three hex digits worth of bits.

• 3fff:0:0000::/44
• 3fff:0:0010::/44
• 3fff:0:0020::/44
• 3fff:0:0030::/44
• 3fff:0:0040::/44
• 3fff:0:0050::/44

which becomes

• 3fff:0::/44
• 3fff:0:10::/44
• 3fff:0:20::/44
• 3fff:0:30::/44
• 3fff:0:40::/44
• 3fff:0:50::/44

/48

You’re given 3fff::/20, make some /48 networks.

A /48 is three hextets, worth of bits.

• 3fff:0:0::/48
• 3fff:0:1::/48
• 3fff:0:2::/48
• 3fff:0:3::/48
• 3fff:0:4::/48

Cursed subnets

What if instead, we try and subnet, inside a hex digit?

First, we’d have to know what every hex digit is in binary.

Then we’d want to figure out where the boundaries are for bits borrowed:

/49

You’re given 3fff::/20, make some /49 networks.

A /49 is three hextets, plus one binary bit.

• 3fff:0:0:0::/49
• 3fff:0:0:8000::/49
• 3fff:0:1:0::/49
• 3fff:0:1:8000::/49
• 3fff:0:2:0000::/49

/50

You’re given 3fff::/20, make some /50 networks.

A /50 is three hextets, plus two binary bits.

• 3fff:0:0:0::/50
• 3fff:0:0:4000::/50
• 3fff:0:0:8000::/50
• 3fff:0:0:C000::/50
• 3fff:0:1:0000::/50

/51

You’re given 3fff::/20, make some /51 networks.

A /51 is three hextets, plus three binary bits.

• 3fff:0:0:0::/51
• 3fff:0:0:2000::/51
• 3fff:0:0:4000::/51
• 3fff:0:0:6000::/51
• 3fff:0:0:8000::/51

References

IPv6 address - Wikipedia

IPv6 Hextet Boundaries

For any of these, to subnet them, just change the final hex digit.

Groups of 64

/64   FFFF:FFFF:FFFF:FFFF::
/128  FFFF:FFFF:FFFF:FFFF:FFFF:FFFF:FFFF:FFFF

Groups of 32

/32   FFFF:FFFF::
/64   FFFF:FFFF:FFFF:FFFF::
/96   FFFF:FFFF:FFFF:FFFF:FFFF:FFFF::
/128  FFFF:FFFF:FFFF:FFFF:FFFF:FFFF:FFFF:FFFF

As groups of 16

/16   FFFF::
/32   FFFF:FFFF::
/48   FFFF:FFFF:FFFF::
/64   FFFF:FFFF:FFFF:FFFF::
/80   FFFF:FFFF:FFFF:FFFF:FFFF::
/96   FFFF:FFFF:FFFF:FFFF:FFFF:FFFF::
/112  FFFF:FFFF:FFFF:FFFF:FFFF:FFFF:FFFF::
/128  FFFF:FFFF:FFFF:FFFF:FFFF:FFFF:FFFF:FFFF

Groups of 8

/8    FF::
/16   FFFF::
/24   FFFF:FF::
/32   FFFF:FFFF::
/40   FFFF:FFFF:FF::
/48   FFFF:FFFF:FFFF::
/56   FFFF:FFFF:FFFF:FF::
/64   FFFF:FFFF:FFFF:FFFF::
/72   FFFF:FFFF:FFFF:FFFF:FF::
/80   FFFF:FFFF:FFFF:FFFF:FFFF::
/88   FFFF:FFFF:FFFF:FFFF:FFFF:FF::
/96   FFFF:FFFF:FFFF:FFFF:FFFF:FFFF::
/104  FFFF:FFFF:FFFF:FFFF:FFFF:FFFF:FF::
/112  FFFF:FFFF:FFFF:FFFF:FFFF:FFFF:FFFF::
/120  FFFF:FFFF:FFFF:FFFF:FFFF:FFFF:FFFF:FF::
/128  FFFF:FFFF:FFFF:FFFF:FFFF:FFFF:FFFF:FFFF

Groups of 4

/4    F::
/8    FF::
/12   FFF::
/16   FFFF::
/20   FFFF:F::
/24   FFFF:FF::
/28   FFFF:FFF::
/32   FFFF:FFFF::
/36   FFFF:FFFF:F::
/40   FFFF:FFFF:FF::
/44   FFFF:FFFF:FFF::
/48   FFFF:FFFF:FFFF::
/52   FFFF:FFFF:FFFF:F::
/56   FFFF:FFFF:FFFF:FF::
/60   FFFF:FFFF:FFFF:FFF::
/64   FFFF:FFFF:FFFF:FFFF::
/68   FFFF:FFFF:FFFF:FFFF:F::
/72   FFFF:FFFF:FFFF:FFFF:FF::
/76   FFFF:FFFF:FFFF:FFFF:FFF::
/80   FFFF:FFFF:FFFF:FFFF:FFFF::
/84   FFFF:FFFF:FFFF:FFFF:FFFF:F::
/88   FFFF:FFFF:FFFF:FFFF:FFFF:FF::
/92   FFFF:FFFF:FFFF:FFFF:FFFF:FFF::
/96   FFFF:FFFF:FFFF:FFFF:FFFF:FFFF::
/100  FFFF:FFFF:FFFF:FFFF:FFFF:FFFF:F::
/104  FFFF:FFFF:FFFF:FFFF:FFFF:FFFF:FF::
/108  FFFF:FFFF:FFFF:FFFF:FFFF:FFFF:FFF::
/112  FFFF:FFFF:FFFF:FFFF:FFFF:FFFF:FFFF::
/116  FFFF:FFFF:FFFF:FFFF:FFFF:FFFF:FFFF:F::
/120  FFFF:FFFF:FFFF:FFFF:FFFF:FFFF:FFFF:FF::
/124  FFFF:FFFF:FFFF:FFFF:FFFF:FFFF:FFFF:FFF::
/128  FFFF:FFFF:FFFF:FFFF:FFFF:FFFF:FFFF:FFFF

IPv6 Network Address Planning

Terms

PIN — Place In Network

• Use v6 nibbles to tell you what the subnet does

Per the RFC

RFC 3587 contains the current recommendations for v6 site design.

“Get a /48 from your RIR.”

│ 3 │     45 bits         │  16 bits  │       64 bits              │
├───┼─────────────────────┼───────────┼────────────────────────────┤
│001│global routing prefix│ subnet ID │       interface ID         │
└───┴─────────────────────┴───────────┴────────────────────────────┘

Cisco’s Guidance

PIN

Use the hex nibbles to encode things.

Desktop subnet 200   2001:0db8:729c:4200::/52
Desktop subnet 300   2001:0db8:729c:4300::/52
Lab     subnet 004   2001:0db8:729c:A004::/52
Lab     subnet 006   2001:0db8:729c:A006::/52

BGP

Common subnets found in v6 BGP global table.

Found in the APNIC BGP in 2025 pdf.

If a /24 is the minimum accepted route prefix size in IPv4, what is the comparable size in IPv6?

There appears to be no common consensus position here. The default action many for network operators appears to have no minimum size filter at all. In theory, that would imply that a /128 route object would be accepted across the entire IPv6 DFZ. A more pragmatic observation is that a /32 would be assuredly accepted by all networks, and it appears that many network operators believe that a /48 is also generally accepted. Given that a /48 is the most common prefix size in today’s IPv6 network, this view appears to be widespread. We also see prefixes smaller in size than a /48 in the routing table with /49, /52, /56 and /64 prefixes present in the IPv6 BGP routing table. 0.7% all advertised prefixes are more specific than a /48.

Site planning

Since each site gets its own /48, there are 65k networks to work with.

References

BGP in 2025 | APNIC Blog

RFC 3587: IPv6 Global Unicast Address Format | RFC Editor

How to Request IPv6 Address Space from ARIN

IPv6 Link Local Addressing

These are on-link

Always takes the form fe80::/64

My computer: fe80::2215:3789:d6f8:a04f

References

RFC 4862: IPv6 Stateless Address Autoconfiguration | RFC Editor

Cisco - Understand the IPv6 Link-Local Address

IPv6 Unique Local Addressing

Sort of but not quite equivalent to RFC 1918.

• Any site can generate a ULA and start using it
• Not meant to go over the global v6 Internet

Terms

ULA — Unique Local Address

• Globally unique via randomness
• Take the form, fdxx:xxxx:xxxx::/48
• Five bytes or 40 bits of entropy

Site prefixes

\( 2^{40} = 1{,}099{,}511{,}627{,}776 \)

Approximately 1 trillion unique site prefixes.

Site prefix generation

Each site should generate a unique ULA using a RNG.

Random.org - Five Random Bytes

fdxx:xxxx:xxxx::/48

Examples

Don’t copy my examples … ☹️

• fd1a:b22c:c764::/48
• fd39:42fb:f7a7::/48
• fdb1:bf6c:f270::/48

References

RFC 4193: Unique Local IPv6 Unicast Addresses | RFC Editor

ICMPv6

Required for IPv6 to work correctly.

Header

 0                   1                   2                   3   
 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 
┌───────────────┬───────────────┬───────────────────────────────┐
│     Type      │     Code      │          Checksum             │
├───────────────┴───────────────┴───────────────────────────────┤
│                                                               │
│                         Message Body                          │
│                                                               │

Error Messages

Informational Messages

References

RFC 4443 - Internet Control Message Protocol (ICMPv6) for IPv6

ICMPv6 - Wikipedia

IPv6 Neighbor Discovery

RA messages are sent with ICMPv6. These can carry options, so they’ve been extended to carry RDNSS information.

The router can say “Here is the DNS info”.

Terms

ND — Neighbor Discoery

RA — Router Advertisement

In v6, routers can just advertise the prefix of the attached subnet and options like RDNSS.

RDNSS — Recursive DNS Server

DAD — Duplicate Address Detection

Functions

• SLAAC
• DAD
• Prefix Discoery
• Parameter Discovery
• Address Resolution
• Router Discovery
• Next-Hop Determination
• Neighbor Unreachability Detection
• Redirects

RDNS server option

RFC 8106: IPv6 Router Advertisement Options for DNS Configuration

 0                   1                   2                   3   
 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 
┌───────────────┬───────────────┬───────────────────────────────┐
│     Type      │     Length    │           Reserved            │
├───────────────┴───────────────┴───────────────────────────────┤
│                           Lifetime                            │
├───────────────────────────────────────────────────────────────┤
│                      Addresses of IPv6                        │
│                    Recursive DNS Servers                      │
└───────────────────────────────────────────────────────────────┘

References

RFC 4861: Neighbor Discovery for IP version 6 (IPv6) | RFC Editor

RFC 8106: IPv6 Router Advertisement Options for DNS Configuration | RFC Editor

IPv6 ND Router Advertisements

Header

RFC 4861: Neighbor Discovery for IPv6

 0                   1                   2                   3   
 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 
┌───────────────┬───────────────┬───────────────────────────────┐
│     Type      │     Code      │          Checksum             │
├───────────────┼─┬─┬───────────┼───────────────────────────────┤
│ Cur Hop Limit │M│O│  Reserved │       Router Lifetime         │
├───────────────┴─┴─┴───────────┴───────────────────────────────┤
│                         Reachable Time                        │
├───────────────────────────────────────────────────────────────┤
│                          Retrans Timer                        │
├───────────────────────────────────────────────────────────────┘
│   Options ...                                                  
└───────────────────────                                         

RDNS server option

RFC 8106: IPv6 Router Advertisement Options for DNS Configuration

 0                   1                   2                   3   
 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 
┌───────────────┬───────────────┬───────────────────────────────┐
│     Type      │     Length    │           Reserved            │
├───────────────┴───────────────┴───────────────────────────────┤
│                           Lifetime                            │
├───────────────────────────────────────────────────────────────┤
│                      Addresses of IPv6                        │
│                    Recursive DNS Servers                      │
└───────────────────────────────────────────────────────────────┘

Packet capture

Taken from my home router 13-July-2026, I’ve modified the IPs.

Frame 46: Packet, 174 bytes on wire (1392 bits), 174 bytes captured (1392 bits) on interface <removed>
Ethernet II, Src: Routerboardc_ef:69:14 (48:a9:8a:ef:69:14), Dst: IPv6mcast_01 (33:33:00:00:00:01)
Internet Protocol Version 6, Src: fe80::4aa9:8aff:feef:6914, Dst: ff02::1
Internet Control Message Protocol v6
    Type: Router Advertisement (134)
    Code: 0
    Checksum: 0xa34c [correct]
    [Checksum Status: Good]
    Cur hop limit: 0
    Flags: 0x00, Prf (Default Router Preference): Medium
    Router lifetime (s): 1800
    Reachable time (ms): 0
    Retrans timer (ms): 0
    ICMPv6 Option (Source link-layer address : 48:a9:8a:ef:69:14)
    ICMPv6 Option (Recursive DNS Server 2001:db8:12:15::6 2001:db8:12:15::7)
        Type: Recursive DNS Server (25)
        Length: 5 (40 bytes)
        Reserved
        Lifetime: 1800 (30 minutes)
        Recursive DNS Servers: 2001:db8:12:15::6
        Recursive DNS Servers: 2001:db8:12:15::7
    ICMPv6 Option (Recursive DNS Server fe80::4aa9:8aff:feef:6914)
        Type: Recursive DNS Server (25)
        Length: 3 (24 bytes)
        Reserved
        Lifetime: RDNSS address MUST no longer be used (0) (0 seconds)
        Recursive DNS Servers: fe80::4aa9:8aff:feef:6914
    ICMPv6 Option (Prefix information : 2001:db8:12:15::/64)
        Type: Prefix information (3)
        Length: 4 (32 bytes)
        Prefix Length: 64
        Flag: 0xc0, On-link Flag (L), Autonomous Address Configuration Flag (A)
            1... .... = On-link Flag (L): Set
            .1.. .... = Autonomous Address Configuration Flag (A): Set
            ..0. .... = Router Address Flag (R): Not set
            ...0 .... = DHCPv6-PD Preferred Flag (P): Not set
            .... 0000 = Reserved: 0
        Valid Lifetime: 2592000 (30 days)
        Preferred Lifetime: 604800 (7 days)
        Reserved
        Prefix: 2001:db8:12:15::

Wireshark

BPF:ip6 protochain 58

References

RFC 4861: Neighbor Discovery for IP version 6 (IPv6) | RFC Editor

RFC 8106: IPv6 Router Advertisement Options for DNS Configuration | RFC Editor

IPv4 to IPV6 Migration Strategies

Dual-stack

• AKA, Native Mode
• AKA, Ships in the Night
• Both at the same time
• Twice the administrative overhead

Tunneling

• Encapsulate the IPv6 packets
• Into IPv4
• Send

Translation

• Make the IPv6 packets back into IPv4 packets.

ISATAP — Intra-Site Automatic Tunneling Addressing

Hybrid Model

• Uses ISATAP

Service Blocks

• Combing tunnels with dual-stack.

Native mode

Both stacks are running on clients. The clients request A and AAAA records near simultaneously.

Whichever DNS records return first decides if the next set of flows are v4 or v6.

This strategy is called “Happy Eyeballs” after the RFC. Do not punish users by “preferring” one protocol or another.

Do whatever is fastest.

Tunneling

• Reduces MTU by 20 bytes.
• The preferred tunnel is GRE.

Automatic tunnels

6to4

• Uses 2002::/16

6RD

• The ISP uses its own addressing.

ISATAP — Intra-site Automatic Tunnel Addressing Protocol

• Uses these 32 bits to denotes ISATAP 0000:5EFE

Translation

• Much like NAT

DNS64 and NAT64

DNS64

• DNS creates synthetic AAAA records, so a v4 client can connect to a v6 service.

NAT64

• A router translates IPv6 traffic to IPv4 traffic.

References

Happy Eyeballs Version 2: Better Connectivity Using Concurrency

Connection of IPv6 Domains via IPv4 Clouds

Framework for IPv4/IPv6 Translation

ISATAP

• Deprecated
• Packet-in-packet technology
• Mostly historical
• Used within one site, not an Internet Technology
• “v6 islands within a v4 ocean”
• Treats the v4 network as NBMA
• ISATAP devices must be dual stacked
• Adds 20-bytes of overhead
• Works as long as v4 routing works
• Stateless

Theory

Use v4 connectivity to emulate a NBMA network. Wrap the v6 packet inside v4.

┌───────────┬───────────┐
│ v6-packet │ v4-header │ ───────►
└───────────┴───────────┘

Router ISATAP

         v6 only             v4 only             v6 only         
         ┌─────┐       ┌─────────────────┐       ┌─────┐         
         ▼     ▼       ▼                 ▼       ▼     ▼         
┌────────┐     ┌───────┐     ┌─────┐     ┌───────┐     ┌────────┐
│client-1│     │   R1  │     │ R2  │     │   R3  │     │client-2│
│   v6   ├─────┤ v4/v6 ├─────┤  v4 ├─────┤ v4/v6 ├─────┤   v6   │
└────────┘     └───────┘     └─────┘     └───────┘     └────────┘
                     ISATAP           ISATAP
                      Interface        Interface

Client ISATAP

                           v4 only                           
               ┌─────────────────────────────┐               
               ▼                             ▼               
┌────────┐     ┌─────┐     ┌─────┐     ┌─────┐     ┌────────┐
│client-1│     │ R1  │     │ R2  │     │ R3  │     │client-2│
│ v4/v6  ├─────┤  v4 ├─────┤  v4 ├─────┤  v4 ├─────┤  v4/v6 │
└────────┘     └─────┘     └─────┘     └─────┘     └────────┘
      ISATAP                                     ISATAP      
       Interface                                  Interface  

ISATAP interface in v6

All ISATAP interfaces look like this. The giveaway is 0000:5EFE in the host portion.

If a router sees a v6 destination that looks like this, it’s an ISATAP packet.

           64 bits                32 bits        32 bits     
┌─────────────────────────────┬──────────────┬──────────────┐
│       Global Unicast        │  0000:5EFE   │ IPv4 Address │
│           Prefix            │              │on ISATAP Link│
└─────────────────────────────┴──────────────┴──────────────┘

Terms

PRL — Potential Router List

• Routers that could perform ISATAP

ISATAP Interface

• Dual-stack
  • Has the v6 IP

ISATAP v4 Address

• NBMA address
• How to reach this device via ISATAP

ISATAP v6 Address

• End-to-end v6 address
• Must embed the v4 address

References

ISATAP - Wikipedia

RFC 5214: Intra-Site Automatic Tunnel Addressing Protocol (ISATAP)

ISATAP.org - Wayback Machine

IP Routing Configuration Guide - ISATAP Tunnel Support for IPv6 Support - Cisco

6RD

6to4

• The base standard
• Requires a specific prefix

6RD

• The current standard
• Allows any global prefix

RG — Router Gateway

• The CPE the ISP provides to connect to their network

BNG — Broadband Network Gateway

• This is the Customer aggregation node
• Aggregate thousands of connections

NMS — Network Management System

• Provides a place to see and configure the network

AAA — Authentication, Authorization, Accounting

• This is how the ISP controls who can access the network
• Controls QoS Deployments

BR

• Terminates the 6RD tunnels
• AKA, Border Relay

What does 6RD solve?

“We can’t deploy v6, because our BNG does (put-feature-here) and we need that!

This way, the customer gets dual stack without the ISP needing to upgrade a legacy deployment.

The four things required to setup

• IPv4 Common Prefix: - The high order bits every CE has in common for their v4 deployments.

If every customer was given an IP on the 10.0.0.0/24 network, the bits they would have in common is /24.

• 6RD Prefix: The v6 bits for the 6rd domain.
• 6RD PrefixLen: The length of the prefix for the 6rd domain.
• 6RD BR v4 Address: The Access Network v4 address for the BR.

(there is a neat way to derive the 6rd prefix, refer to Cisco’s IOS-XE manual, that’s closest to how it’s done.)

Sample config (from Cisco’s PDF)

!
! This BR is reachable via the loopback.
!
interface Loopback0
ip address 10.1.1.1 255.255.255.0
!
interface Tunnel0
 tunnel source Loopback0
 tunnel mode ipv6ip 6rd
 tunnel 6rd ipv4 prefix-len 8
 tunnel 6rd prefix 2011:1001:100:/40
 ipv6 address 2011:1001:101:101::/128 anycast
!
ipv6 route 2011:1001:100::/40 Tunnel0
ipv6 route 2011:1001:101:101::/64 Null0
!
! Simple and Easy to setup!
!
#show tunnel 6rd
Interface Tunnel0:
 Tunnel Source: 10.1.1.1
 6RD: Operational, V6 Prefix: 2011:1001:100:/40
 V4 Prefix, Length: 8, Value: 10.0.0.0
 V4 Suffix, Length: 0, Value: 0.0.0.0
 General Prefix: 2011:1001:1

References

RFC 3056: Connection of IPv6 Domains via IPv4 Clouds | RFC Editor

RFC 5969: IPv6 Rapid Deployment on IPv4 Infrastructures (6rd) – Protocol Specification | RFC Editor

APNIC Conference | 6RD Enabling IPv6 Customers on an IPv4-only Network

Interface and Hardware Component Configuration Guide, Cisco IOS XE Fuji 16.9.x - IPv6 Rapid Deployment Cisco ASR 1000 Series Aggregation Services Routers - Cisco

NAT64

Allows v6 only clients access to v4 only services.

Requires:

• DNS64 Server to translate an A record to a synthetic AAAA record
• NAT64 Router to keep state
• Upstream DNS to respond with an A record, to the DNS64 server
• Stateful Prefix

sequenceDiagram
    participant PC as Host<br/>v6 only
    participant DNS64 as Server<br/>DNS64
    participant DNS4 as Server<br/>DNS4
    participant R1 as Router<br/>NAT64
    Participant Web as Web Server<br/>v4 only

    PC    ->>   DNS64: Request AAAA
    DNS64 ->>   DNS4: Request A
    DNS4  -->>  DNS64: Answer with A
    DNS64 -->>  PC: Translate to Synthetic AAAA
    PC    ->>   R1: Send with v6 prefix
    R1    ->>   Web: Translate to v4
    Web   -->>  R1: Respond with v4
    R1    -->>  PC: Check Translations<br/>Reply with v6

Config

ipv6 unicast-routing
!
! Define NAT64 Prefix
!
nat64 prefix stateful 2001:DB8:0:64::/96
!
! Create NAT64 pool
!
nat64 v4 pool DHCP_POOL64 10.0.0.100 10.0.0.200
!
! Create ACL for NAT64 Clients
!
ipv6 access-list ACL_NAT64_CLIENTS
  permit ipv6 2001:db8:1::/48 any
!
! Match v6 clients to the v4 pool
!
nat64 v6v4 list ACL_NAT64_CLIENTS pool DHCP_POOL64 overload
!
! Enable on v6 network
!
interface GigabitEthernet3
 ipv6 address 2001:db8:1::1/64
 nat64 enable
!
! Enable on v4 network
!
interface GigabitEthernet1
 ip address 10.0.0.1 255.255.255.0
 nat64 enable

DNS64

apk update apk add unbound apk add bind-tools rc-update add unbound default

DNS64:~# cat /etc/unbound/unbound.conf 
server:
    interface: 0.0.0.0
    interface: ::0
    access-control: 0.0.0.0/0 allow
    access-control: ::/0 allow
    do-ip4: yes
    do-ip6: yes
    do-udp: yes
    do-tcp: yes
    module-config: "dns64 iterator"
    # Do not use the Well-Known-Prefix, it breaks with private v4 addresses.
    # See https://datatracker.ietf.org/doc/html/rfc6052#section-3.4
    dns64-prefix: 2001:db8:0:64::/96

forward-zone:
        name: "."
        # A different upstream server, that returns only an A record.
        forward-addr:  2001:db8:1::6

DNS4

Installing Apps

apk update
apk add unbound
apk add bind-tools
rc-update add unbound default

DNS4:~$ cat /etc/unbound/unbound.conf
server:
    interface: 0.0.0.0
    interface: ::0
    access-control: 0.0.0.0/0 allow
    access-control: ::/0 allow
    do-ip4: yes
    do-ip6: yes
    do-udp: yes
    do-tcp: yes
    local-data: "64lab.example.com. A 10.0.0.20"

Verification

R1# show nat64 translations 

Proto  Original IPv4         Translated IPv4
       Translated IPv6       Original IPv6 
----------------------------------------------------------------------------

icmp   10.0.0.20:2           [2001:db8:0:64::a00:14]:2                       
       10.0.0.100:2          [2001:db8:1:0:5054:ff:fe11:d8ec]:2              

Total number of translations: 1

The WKP (Well Known Prefix) will not work toward RFC1918 addresses. You must configure a different /96.

References

RFC 6052: IPv6 Addressing of IPv4/IPv6 Translators | RFC Editor

IP Addressing Configuration Guide, Cisco IOS XE 17.x - Stateful Network Address Translation 64 Cisco IOS XE 17 - Cisco

RFC 4291: IP Version 6 Addressing Architecture | RFC Editor

RFC 2464: Transmission of IPv6 Packets over Ethernet Networks

Stateless NAT64

Uses one v4 address for each v6 address.

Prefix Embedding

From RFC 6052

    ┌──────────────────────────────────────────────────────────────────┐
 PL │0-----------------32--40--48--56--64--72--80--88--96--104-112-120-│
    ├──────────────────┬───────────────┬───┬───────────────────────────┤
/32 │      prefix      │         v4(32)│ u │                 suffix    │
    ├──────────────────┴───┬───────────┼───┼───┬───────────────────────┤
/40 │      prefix          │     v4(24)│ u │(8)│             suffix    │
    ├──────────────────────┴───┬───────┼───┼───┴───┬───────────────────┤
/48 │      prefix              │ v4(16)│ u │ v4(16)│         suffix    │
    ├──────────────────────────┴───┬───┼───┼───────┴───┬───────────────┤
/56 │      prefix                  │(8)│ u │ v4(24)    │     suffix    │
    ├──────────────────────────────┴───┼───┼───────────┴───┬───────────┤
/64 │      prefix                      │ u │ v4(32)        │ suffix    │
    ├──────────────────────────────────┴───┴───────────┬───┴───────────┤
/96 │      prefix                                      │    v4(32)     │
    └──────────────────────────────────────────────────┴───────────────┘

Supported Prefixes

• /32
• /40
• /48
• /56
• /64
• /96

References

RFC 6052: IPv6 Addressing of IPv4/IPv6 Translators | RFC Editor

RFC 7915: IP/ICMP Translation Algorithm | RFC Editor

Cisco - IP Addressing Configuration Guide, IOS-XE 17.x - Stateless Network Address Translation 64

Ethernet CSMA/CD

Is used when the link is half-duplex. For full-duplex links CDMA/CD isn’t used.

This standard provides for two modes of operation of the MAC sublayer:

a) In half duplex mode, stations contend for the use of the physical medium, using the CSMA/CD algorithms specified.

Bidirectional communication is accomplished by rapid exchange of frames, rather than full duplex operation.

Half duplex operation is possible on all supported media; it is required on those media that are incapable of supporting simultaneous transmission and reception without interference, for example, 10BASE2 and 100BASE-T4.

b) The full duplex mode of operation can be used when all of the following are true:

1. The physical medium is capable of supporting simultaneous transmission and reception without interference (e.g., 10BASE-T, 10BASE-FL, and 100BASE-TX/FX).

2. There are exactly two stations on the LAN. This allows the physical medium to be treated as a full duplex point-to-point link between the stations. Since there is no contention for use of a shared medium, the multiple access (i.e., CSMA/CD) algorithms are unnecessary.

3. Both stations on the LAN are capable of and have been configured to use full duplex operation.

Source - IEEE 802.3-2022 - Section 4.1.1 Media Access Control, Functional model of the MAC method, Overview

References

802.3-2022 - IEEE Standard for Ethernet

10G Ethernet

Terms

WAN PHY

These can operate with SONET/SDH at STS-192c.

GPON

Gigabit Passive Optical Network. Passive because none of the intermediate gear is powered.

References

10 Gigabit Ethernet - Wikipedia

Testing PAM4 Signaling - 10GBASE-T1 Automotive

Cisco 10GBASE SFP+ Modules Data Sheet - Cisco

Mastering External Modulation in Lasers

802.1Q

A 32-bit tag added to a Ethernet frame to multiplex VLANs

 0                   1                   2                   3
 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1
┌───────────────────────────────┬─────┬─┬───────────────────────┐
│    Tag Protocol Identifier    │ PCP │D│       VLAN ID         │
└───────────────────────────────┴─────┴─┴───────────────────────┘

Terms

Tag Protocol Identifier

• 0x8100
• Same byte position as EtherType
• Tells the equipment “expect an 802.1q header”

PCP — Priority Code Point

• AKA, 802.1P priority bits
• AKA, Ethernet Class of Service
• AKA, CoS
• Normally zeros

DEI — Drop Eligible Indicator

• Bit 19
  • On or off
  • Normally off

VLAN ID

• 12 bits
• 0xFFF (4095) is reserved

VLAN ID ranges

ARP

Captured on-wire.

packet #1 - who has 10.0.0.10? Tell 10.0.0.20
packet #2 - 10.0.0.10 is at ce:b1:5f:58:1d:8a

ARP request

> Ethernet II

    Destination: Broadcast (ff:ff:ff:ff:ff:ff)
    Source: 1a:20:4e:9e:fb:9c (1a:20:4e:9e:fb:9c)
    Type: ARP (0x0806)

> Address Resolution Protocol (request)

    Hardware type: Ethernet (1)
    Protocol type: IPv4 (0x0800)
    Hardware size: 6
    Protocol size: 4
    Opcode: request (1)
    Sender MAC address: 1a:20:4e:9e:fb:9c (1a:20:4e:9e:fb:9c)
    Sender IP address: 10.0.0.20
    Target MAC address: 00:00:00_00:00:00 (00:00:00:00:00:00)
    Target IP address: 10.0.0.10

ARP reply

> Ethernet II

    Destination: 1a:20:4e:9e:fb:9c (1a:20:4e:9e:fb:9c)
    Source: ce:b1:5f:58:1d:8a (ce:b1:5f:58:1d:8a)
    Type: ARP (0x0806)
    Padding: ... lots of zeros

> Address Resolution Protocol (reply)

    Hardware type: Ethernet (1)
    Protocol type: IPv4 (0x0800)
    Hardware size: 6
    Protocol size: 4
    Opcode: reply (2)
    Sender MAC address: ce:b1:5f:58:1d:8a (ce:b1:5f:58:1d:8a)
    Sender IP address: 10.0.0.10
    Target MAC address: 1a:20:4e:9e:fb:9c (1a:20:4e:9e:fb:9c)
    Target IP address: 10.0.0.20

ARP Attacks

ARP Spoofing and ARP Poisoning are related concepts.

The attacker spoofs an IP on the network, usually the default gateway, with the MAC address of their machine, to intercept all user traffic.

This leads to ARP poisoning where device IP-to-MAC tables now contain false bindings.

Successful ARP attacks lead to traffic hijacking, traffic denial, or man-in-the-middle attacks.

References

ARP spoofing - Wikipedia

DAI

Minimum config

ip dhcp snooping vlan 10
ip arp inspection vlan 10
ip arp inspection validate src-mac dst-mac ip 
!
! Ports
!
interface GigabitEthernet0/1
 description towards DHCP server
 ip arp inspection trust
 ip dhcp snooping trust

Validation

access-1# show ip dhcp snooping binding 
MacAddress          IpAddress        Lease(sec)  Type           VLAN  Interface
------------------  ---------------  ----------  -------------  ----  --------------------
52:54:00:0D:65:73   10.10.10.102     80574       dhcp-snooping   10    GigabitEthernet0/0
Total number of bindings: 1

access-1# show ip arp inspection 

Source Mac Validation      : Enabled
Destination Mac Validation : Enabled
IP Address Validation      : Enabled

 Vlan     Configuration    Operation   ACL Match          Static ACL
 ----     -------------    ---------   ---------          ----------
   10     Enabled          Active                         

 Vlan     ACL Logging      DHCP Logging      Probe Logging
 ----     -----------      ------------      -------------
   10     Deny             Deny              Off          

 Vlan      Forwarded        Dropped     DHCP Drops      ACL Drops
 ----      ---------        -------     ----------      ---------
   10            134              0              0              0

 Vlan   DHCP Permits    ACL Permits  Probe Permits   Source MAC Failures
 ----   ------------    -----------  -------------   -------------------
   10             48              0              0                     0

 Vlan   Dest MAC Failures   IP Validation Failures   Invalid Protocol Data
 ----   -----------------   ----------------------   ---------------------
          
 Vlan   Dest MAC Failures   IP Validation Failures   Invalid Protocol Data
 ----   -----------------   ----------------------   ---------------------
   10                   0                        0                       0

Reference

Cisco - Dynamic ARP Inspection

Practical Networking - Gratuitous ARP

Proxy ARP

There are lots of requirements to get this to work.

• Router needs to see the host on-link
• Host is configured for a much larger subnet then what is actually present
• The ARP target the host is requesting is in the routing table
• The ARP target cannot be out the same interface the ARP was heard on

Lab topology

Evidences

This happens if the host is off-link for the router. The router will see the arp, but also filter it.

*May  4 20:39:31.619: IP ARP req filtered src 192.168.100.100 5254.00b7.f21b, dst 192.168.23.2 0000.0000.0000 wrong cable, interface GigabitEthernet1 tableid 0
*May  4 20:39:32.675: IP ARP req filtered src 192.168.100.100 5254.00b7.f21b, dst 192.168.23.2 0000.0000.0000 wrong cable, interface GigabitEthernet1 tableid 0

Verification

R1# show ip traffic | i proxy
        0 proxy name requests, 0 where-is requests, 0 other
  Sent: 0 address requests, 0 address replies (0 proxy)
        0 proxy name replies, 0 where-is replies
  Sent: 1 requests, 8 replies (2 proxy), 0 reverse

Captures

ARP-proxy-arp.pcap

References

Cisco - Proxy ARP

Cisco - Proxy ARP Problem

VLAN Access Control Lists

Can be IPs and/or MACs.

These work on routed or switched traffic.

Routed flow

block
columns 1
    id1(("Ingress\nPort"))
    block:one
      id2("Port\nACL")
      id3("VLAN\nACL")
      id4("Routed\nACL")
    end
    id5(["SVI"])
    id6("Routing\nOperation")
    id7(["SVI"])
    block:two
      id8("Routed\nACL")
      id9("VLAN\nACL")
      id10("Port\nACL")
    end
    id11(("Egress\nPort"))
    id1 space:2 id2
    id1  --- id2
    id2  --- id3
    id3  --- id4
    id4  --- id5
    id5  --- id6
    id6  --- id7
    id7  --- id8
    id8  --- id9
    id9  --- id10
    id10 --- id11
    style id6 fill:#1A1,stroke:#333,stroke-width:2px

Switched flow

The VLAN ACL is only processed once, on switching operation.

block
columns 1
    id1(("Ingress\nPort"))
    id2("Port\nACL")
    block:switching
        id3("VLAN\nACL")
        id6("Switching\nOperation")
    end
    id10("Port\nACL")
    id11(("Egress\nPort"))
    id1  --- id2
    id2  --- id3
    id3  --- id6
    id6  --- id10
    id10 --- id11
    style id6 fill:#1A1,stroke:#333,stroke-width:2px
    style switching fill:#1A1,stroke:#333,stroke-width:2px

Config

Copied from the TAC notes.

ip access-list extended TEST
 10 permit ip host 10.1.1.1 any
 20 permit ip any host 10.1.1.1
!
ip access-list extended ELSE
 10 permit ip any any
!
vlan access-map VACL 10
 match ip address TEST
 action forward
vlan access-map VACL 20
 match ip address ELSE
 action drop
!
vlan filter VACL vlan-list 10

References

Cisco Catalyst IR8340 Rugged Series Router Software Configuration Guide, Cisco IOS XE Release 17.15.x - VLAN Access Control Lists Cisco Catalyst IR8300 Rugged Series Router - Cisco

Validate Security ACLs on Catalyst 9000 Switches - Cisco

Solved: PACL and VACL processing order - Cisco Community

QoS

… is planned unfairness. To make some traffic faster, other traffic must wait.

Terms

BE — Best Effort

• Default traffic class

FIFO — First in, First out

• Default behavior

Differentiated Services

• AKA, DiffServ
• Giving packet flows different levels of network service
• Based on classification

Integrated Services

• AKA, IntServ
• Packet flows explicitly reserve bandwidth along a path
• Uses admission control

RSVP — Resource Reservation Protocol

• An IntServ Implementation

Marking

• Changing the DSCP bits in the IP header field

DSCP — Differentiated Services Code Point

• The markings of an IP packet that allows DiffServ

PHB — Per Hop Behavior

• What a node should or shouldn’t do with marked traffic.

Queuing

• Hold a packet in memory
  • Delays transmission
  • Buffers have limits
  • Expensive, because memory is expensive
  • Enables RED

LLQ — Low Latency Queuing

• Describes queue behavior for the EF PHB
  • Always service this queue first
  • No RED
  • Must be policed or it will starve other queues

EF — Expedited Forwarding

• The highest PHB for network data

Control Traffic

• Routing Traffic
  • OSPF, EIGRP, IS-IS, BGP, etc

CAR — Committed Access Rate

• Contractual data rate a traffic source will flow at
• SLA oriented

SLA — Service Level Agreement

• A Business agreements about data servicing requirements

WFQ — Weighted Fair Queuing

• Legacy default on serial interfaces at E1 speeds and below ~2.048 Mbps
• Sorts traffic into high bw and low bw classes.

CBWFQ — Class Based Weighted Fair Queuing

• AKA, Modular QoS

• Multiple queues

• Bandwidth limits

• Different kinds of queues, like LLQ

MQC

• Modular QoS CLI

PQ — Priority Queue

• A queue that is served first, even if other queues have been waiting longer
• An LLQ is an example of a PQ

Type of service

How these 8 bits work has changed over the years.

                 0 1 2 3 4 5 6 7
                ┌─────┬─────┬─┬─┐
 RFC 791 (1981) │IP Pr│ ToS │0│0│
                └─────┴─────┴─┴─┘

                 0 1 2 3 4 5 6 7
                ┌─────┬───────┬─┐
RFC 1349 (1992) │IP Pr│  TOS  │0│
                └─────┴───────┴─┘

                 0 1 2 3 4 5 6 7
                ┌───────────┬─┬─┐
RFC 2474 (1998) │    DSCP   │0│0│
                └───────────┴─┴─┘

                 0 1 2 3 4 5 6 7
                ┌───────────┬───┐
RFC 3168 (2001) │    DSCP   │ECN│
                └───────────┴───┘

PHB - per hop behaviors

Assured forwarding

Assured Forwarding PHB Group

AF uses the first 6 bits to create 4 traffic classes, 4 is best.

Within those classes, there is a drop precedence, or … at what point of queue congestion should this traffic be dropped.

Used for RED, or WRED.

Four AF classes, each should get its own resources.

Binary

Drop
 Precedence      Class 1        Class 2        Class 3       Class 4

            ┌───────────────┬───────────────┬───────────────┬──────────────┐
   Low    │ │ AF11  001 010 │ AF21  010 010 │ AF31  011 010 │ AF41 100 010 │
   Medium │ │ AF12  001 100 │ AF22  010 100 │ AF32  011 100 │ AF42 100 100 │
   High   ▼ │ AF13  001 110 │ AF23  010 110 │ AF33  011 110 │ AF43 100 110 │
            └───────────────┴───────────────┴───────────────┴──────────────┘
             ────────────► Importance to Organization ───────────►

DSCP

Drop
 Precedence      Class 1        Class 2        Class 3       Class 4

            ┌───────────────┬───────────────┬───────────────┬──────────────┐
   Low    │ │ AF11  DSCP 10 │ AF21  DSCP 18 │ AF31  DSCP 26 │ AF41 DSCP 34 │
   Medium │ │ AF12  DSCP 12 │ AF22  DSCP 20 │ AF32  DSCP 28 │ AF42 DSCP 36 │
   High   ▼ │ AF13  DSCP 14 │ AF23  DSCP 22 │ AF33  DSCP 30 │ AF43 DSCP 38 │
            └───────────────┴───────────────┴───────────────┴──────────────┘
             ────────────► Importance to Organization ───────────►

Yields the following formula.

DSCP = 8 (class) + 2 (drop)

QoS consequences

LAN QoS with voice (buffer management)

• One voice packet, no voice, but modem will retrain
• Two voice packets, audio clip, fax call disconnection
• VoIP QoS cannot be fixed by adding bandwidth
  • It must be treated as LLQ

QoS commands

RFC 4594 — DiffServ service classes

Source: RFC 4594 (Aug 2006), updated by RFC 5865 and RFC 8622. AF drop precedence: x1=low, x2=medium, x3=high drop probability.

References

An Architecture for Differentiated Services

QoS Trust Boundaries

• CoS is 3 bits, or 7 values
• DSCP is 6 bits, or 64 values

Config

Trust if its a phone

!
! if a phone detected with CDP
!    then trust dscp
!
mls qos trust device cisco-phone
mls qos trust dscp

Absolute Trust

mls qos trust dscp

DSCP transparency

Is off by default.

This means the CoS bits take priority and re-write the DSCP portion of the L3 header.

To turn this feature on, (to let DSCP move transparently through the switches) use no mls qos rewrite ip dscp

IOS-XE C9300 trust and queuing behavior

An example for one platform to demonstrate how complex QoS trust can become.

Maps

Default CoS-to-DSCP Map

Default DSCP-to-CoS Map

References

Cisco - QoS Configuration Guide, Cisco IOS-XE - 26.x.x - C9300

802.1P

Layer 2 QoS.

The 3-bit PCP portion of the 802.1q tag.

 0                   1                   2                   3
 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1
┌───────────────────────────────┬─────┬─┬───────────────────────┐
│    Tag Protocol Identifier    │ PCP │D│       VLAN ID         │
└───────────────────────────────┴─────┴─┴───────────────────────┘

Terms

Tag Protocol Identifier

• Same as 802.1q
• 0x8100
• Same byte position as EtherType
• Tells the equipment “expect an 802.1q header”

PCP — Priority Code Point

• AKA, 802.1P
• AKA, Priority Bits
• AKA, Priority Tagging
• AKA, Ethernet Class of Service
• AKA, Trafic Class Expediting
• AKA, CoS
• Not zeros in this case

VLAN ID

Can be anything from 1 to 4094, but zero is reserved.

Vlan 0

• “This frame doesn’t have a vlan”
• “I wanted to use the L2 QoS features”

Seen in ISP networks.

DEI — Drop Eligible Indicator

• Bit 19
  • On or off
  • Still normally off

References

Cisco - VLAN 0 Priority Tagging Support

Congestion

Terms

Incipient Congestion

• Happens from multiplexing
• Statistical burst of traffic
• Sometimes packets, buffer or drop
• Irrespective of a long-term load

Persistent Congestion

• Over-Subscription
• Packet loss

CC — Congestion Control

AQM — Active Queue Management

• Prevents the queue from filling even faster
• Drops packets selectively

RED — Random Early Detection

• Based on Queue Depth
  • Queue is 100 packets
    • 30 packets, drop rate of a new packet is 20%
    • 50 packets, drop rate is 50%
    • 100 packet, drop rate is 100%

Congestion Collapse

• Badly designed protocols react to congestion by trying to send faster or more often.

TCP’s implementation of CC is responsible for the Internet not suffering from Congestion Collapse.

TCP

• ECN is marked
• Dropped packets cause streams to slow down.

Global Synchronization

TCP on its own, will settle into this pattern, when the queues fill, multiple flows all speeding up and slowing down around the same period.

References

RFC 7567: IETF Recommendations Regarding Active Queue Management | RFC Editor

RFC 7141: Byte and Packet Congestion Notification | RFC Editor

Random Early Detection Gateways for Congestion Avoidance - 1993 IEEE

Cisco IOS QoS Commands

See All Matched QoS

show mls qos interface statistics

See the DSCP to CoS Map

show mls qos map cos-dscp

See QoS on a single Interface

show mls qos interface Fa0/1

Trust All QoS Markings

mls qos trust dscp

Trust QoS if it comes from a Cisco Phone

mls qos trust device cisco-phone

XR QoS Commands

Display all service-policies configured in a direction

show running interface * service-policy input
show running interface * service-policy output

Display QoS Statistics

show policy-map interface <int> input
show policy-map interface <int> output

Display details of a service-policy programmed in hardware

show qos interface <int> input
show qos interface <int> output

Display the interfaces a policy-map is attached to

show policy-map targets [pmap-name <name>] [location <loc>]

Shape outgoing to 100Mbps and police incoming to 100Mbps

interface GigabitEthernet0/0/0/0.100 l2transport
 service-policy input incoming-limit-to-100Mb
 service-policy output outgoing-limit-to-100Mb
!
policy-map incoming-limit-to-100Mb
 class class-default
  police rate 100 mbps
   exceed-action drop
  !
!
end-policy-map
!
policy-map outgoing-limit-to-100Mb
 class class-default
  shape average 100 mbps
 !
end-policy-map

Verification

show policy-map interface g0/0/0/0.100 input
show policy-map interface g0/0/0/0.100 output

References

Modular QoS Configuration Guide for Cisco ASR 9000 Series Routers, IOS XR Release 7.9.x - Configuring Hierarchical Modular QoS Cisco ASR 9000 Series Aggregation Services Routers - Cisco

Spanning Trees

Algorhyme

I think that I shall never see
a graph more lovely than a tree.

A tree whose crucial property
is loop-free connectivity.

A tree which must be sure to span
so packets can reach every LAN.

First the root must be selected.
By ID it is elected.

Least cost paths from root are traced.
In the tree these paths are placed.

A mesh is made by folks like me,
then bridges find a spanning tree.

– Radia Perlman

L2 networks were getting larger, and a common mistake was creating a loop, where frame would go around infinitely.

Spanning Tree Protocol STP disables ports to break L2 loops.

STP

Terms

STP — Spanning Tree Protocol

• Frequently cited as 802.1D

Bridge

• A device that participates in the spanning tree algorithm

Root Bridge

• The bridge that wins the STP election

Bridge ID

• Three fields, next to each other
• Bridge Priority, Extension ID (the VLAN), MAC Address

BPDU — Bridge Protocol Data Unit

• The frame used in 802.1D STP

802.1D

• An IEEE standard. The oldest Ethernet STP

Root ID

• The bridge that has won and is winning the elections

Designated ports

• Sends BPDUs downstream
• AKA DP

Root Port

• Receives BPDUs, from upstream switch.
• Each bridge can have only one RP. RP is picked by port-selection-algo
• AKA, RP
• AKA, Upstream

TCN — Topology change notification

• This is its own message
• Sent by the bridge that sees a STP change, upstream via it’s RP
• One of the only upstream messages

TCA Bit — Topology Change Acknowledge

• Sent by the upstream bridge
• Lets the downstream reporting bridge know the TCN was relayed upstream
• Inside a config BPDU

TC Bit — Topology Change

• The root bridge sets the TC Bit
• Downstream bridges shorten their MAC aging timer to Forward Delay (default 15 seconds)

How STP makes a loop free topology

STP elects root and designated ports, aka RP, and DPs. It also moves STP ports into Blocking.

• A bridge can only have one RP.
• All ports on the root are DPs.
• Ports on the root bridge never enter blocking.
• Blocked ports must keep receiving BPDUs to stay blocked (the election must continue, forever)
• if two would-be DPs send and receive BPDUs.
  • There is a loop.
  • The port that has the inferior BPDU will block.

1. All bridges turn on send BPDUs on all STP ports, themselves as root.
2. STP ports (bridges) compare BPDUs.
3. Bridge with lowest Bridge ID is root, (Lowest priority, if priority is default, Lowest mac, usually the oldest switch)
4. All ports on root bridge are DP, and BPDU cost field is set to zero.
5. Root sends BPDUs.
6. DPs send configuration BPDUs.
7. RPs receive configuration BPDUs.
8. Root bridge sends BPDU, cost is 0, with port identifiers set.
9. A non-root bridge can only have one RP.
10. Non-root bridge gets BPDUs. It uses the port selection Algo to pick one RP.
11. Non-root bridge starts STP elections on all other ports, by sending BPDUs. It takes the cost inside the received BPDU, and adds it’s port cost.
12. If a DP gets a BPDU, STP blocks the port if the received BPDU is better.

Port selection algo

• All choices are made based on the received BPDU.
• Modifications are made on the upstream switch.

1. Lowest cost to root.
2. Lowest system priority of advertising switch.
3. Lowest MAC of advertising switch.
4. Port Identifier Byte of advertising switch (port priority + port number)

Timers

• Hello Time is usually 2 seconds between BPDUs.

• Forward Delay is typically 15 seconds. It’s between off -> listening -> learning.

Device priority

4 bits, goes in geometric sequence starting from 0 to 61440.

switch(config)# spanning-tree vlan 60 priority ?
% Bridge Priority must be in increments of 4096.
% Allowed values are: 
  0     4096  8192  12288 16384 20480 24576 28672
  32768 36864 40960 45056 49152 53248 57344 61440

Root bridges election in spanning tree

Best practice is to set the root to 0 and the secondary to 4096.

Two bridges send each other BPDUs, they compare bridge IDs to see who will keep sending BPDUs

The bridge with the lower ID (priority + mac address) wins. The non-root-bridge copies this bridge ID into it’s BPDU, and sends that downstream.

The default for priority is 32768 or 0x80 on the wire. Because the 802.1D committee exists, the priority is this, plus the vlan ID.

Always configure a root bridge, or the oldest device with probably the lowest mac address wins the root bridge election.

Path cost

The root bridge BPDU gets stuff tack’d onto it. The root bridge advertises itself as 0 cost.

Cost is the value of the link, towards the root bridge.

 ┌───────┐                                                                   
 │  SW1  │                                                                   
 └───┬───┘                                                                   
     │                                                                       
     │  Cost in BPDU from SW1 is 0                                           
     │                                                                       
Eth0 │ ◄──── Interface is Assigned a cost of 100 by SW2 based on link Speed
 ┌───┴───┐                                                                   
 │  SW2  │                                                                   
 └───┬───┘                                                                   
Eth1 │                                                                       
     │   Cost in BPDU on-the-wire is now 100, SW2 Eth0 Cost                  
Eth0 │                                                                       
 ┌───┴───┐                                                                   
 │  SW2  │                                                                   
 └───────┘                                                                   

Port types

Designated ports

• Send BPDUs downstream

Root Ports

• The selected port towards the root bridge

  • Lowest total cost

    • OR Lowest advertised priority
      • OR Lowest advertised port ID (interface number)

Root path cost

Root Path Cost

What the interfaces costs + the advertised cost to the root. The root sends a cost of 0.

STP path calculations

spanning-tree pathcost method long

802.1D - spanning tree

The 802.1D committee wanted two learning states¹, one with and one without learning station addresses. This is why it’s more complicated.

┌─────────────┐                                                     
│     off     │                                                     
└──────┬──────┘                                                     
       │                                                            
       │  Turn on interface                                         
       ▼                                                            
┌─────────────┐                                                     
│  Listening  │ Receive + Send BPDUs                                
└──────┬──────┘                                                     
       │                                                            
       │  forward delay (default 15s)                               
       ▼                                                            
┌─────────────┐                                                     
│  Learning   │ Receive + Send BPDUs + Program CAM                  
└──────┬──────┘                                                     
       │                                                            
       │  forward delay (default 15s)                               
       ▼                                                            
┌─────────────┐                                                     
│  Forwarding │ Receive + Send BPDUs + Program CAM + Forward Frames 
└─────────────┘                                                     

BPDU frame format

Wireshark

This is what the BPDU looks like on-the-wire.

Spanning Tree Protocol
    Protocol Identifier: Spanning Tree Protocol (0x0000)
    Protocol Version Identifier: Spanning Tree (0)
    BPDU Type: Configuration (0x00)
    BPDU flags: 0x00
        0... .... = Topology Change Acknowledgment: No
        .... ...0 = Topology Change: No
    Root Identifier: 0 / 1 / 52:54:00:c4:f3:e7
        Root Bridge Priority: 0
        Root Bridge System ID Extension: 1
        Root Bridge System ID: 52:54:00:c4:f3:e7 (52:54:00:c4:f3:e7)
    Root Path Cost: 0
    Bridge Identifier: 0 / 1 / 52:54:00:c4:f3:e7
        Bridge Priority: 0
        Bridge System ID Extension: 1
        Bridge System ID: 52:54:00:c4:f3:e7 (52:54:00:c4:f3:e7)
    Port identifier: 0x8001 < --------- first byte is Port Priority
    Message Age: 0                        second byte is Port Number
    Max Age: 20
    Hello Time: 2
    Forward Delay: 15

STP header

Times in in 256th of a second.

 0                   1                   2                   3   
 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 
┌───────────────────────────────┬───────────────┬───────────────┐
│          Protocol ID          │    Version    │   BPDU Type   │
└───────────────────────────────┴───────────────┴───────────────┘
┌───────────────┬────────────────────────────────────────────────
│     Flag      │                    Root ID                     
└───────────────┴────────────────────────────────────────────────
─────────────────────────────────────────────────────────────────
                             Root ID                             
─────────────────────────────────────────────────────────────────
────────────────┬────────────────────────────────────────────────
   Root ID      │              Root Path Cost                    
────────────────┴────────────────────────────────────────────────
────────────────┬────────────────────────────────────────────────
  Root Path Cost│                Bridge ID                       
────────────────┴────────────────────────────────────────────────
                                                                 
 0                   1                   2                   3   
 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 
─────────────────────────────────────────────────────────────────
                          Bridge ID                              
─────────────────────────────────────────────────────────────────
────────────────┬───────────────────────────────┬────────────────
   Bridge ID    │          Port ID              │ Message age    
────────────────┴───────────────────────────────┴────────────────
────────────────┬───────────────────────────────┬────────────────
  Message Age   │          Max Age              │ Hello Time     
────────────────┴───────────────────────────────┴────────────────
────────────────┬───────────────────────────────┬───────────────┐
    Hello Time  │        Forward Delay          │ V1 Length     │
────────────────┴───────────────────────────────┴───────────────┘
┌───────────────────────────────┐                                
│           V3 Length           │                                
└───────────────────────────────┘                                

Port elections

Bridge Priority, Vlan, Bridge MAC, Port Priority, Port Number

Default settings

Who is the root?

Both bridges temporarily send BPDUs with themselves both set as root.

┌─────┐                                                                     ┌─────┐
│    1├─ 32768 1 52:54:00:4b:99:08 8001 ─── 32768 1 52:54:00:e8:3a:ff 8001 ─┤1    │
│SW1 2├─ 32768 1 52:54:00:4b:99:08 8002 ─── 32768 1 52:54:00:e8:3a:ff 8002 ─┤2 SW2│
│    3├─ 32768 1 52:54:00:4b:99:08 8003 ─── 32768 1 52:54:00:e8:3a:ff 8003 ─┤3    │
└─────┘                                                                     └─────┘

SW1 wins with 4b. SW1 has the lower MAC address.

32768 / 1 / 52:54:00:4b:99:08 / 8001 < 32768 / 1 / 52:54:00:e8:3a:ff

Setting bridge priority to zero

Who is the root?

Both bridges temporarily send BPDUs with themselves both set as root.

┌─────┐                                                                   ┌─────┐
│    1├─ 32768 1 52:54:00:4b:99:08 8001 ───── 0 1 52:54:00:e8:3a:ff 8001 ─┤1    │
│SW1 2├─ 32768 1 52:54:00:4b:99:08 8002 ───── 0 1 52:54:00:e8:3a:ff 8002 ─┤2 SW2│
│    3├─ 32768 1 52:54:00:4b:99:08 8003 ────- 0 1 52:54:00:e8:3a:ff 8003 ─┤3    │
└─────┘                                                                   └─────┘

SW2 wins with 0. SW2 has the lower bridge priority.

32768 / 1 / 52:54:00:4b:99:08 / 8001 > 0 / 1 / 52:54:00:e8:3a:ff

!
! SW2
!
spanning-tree vlan 1 priority 0

Port blocking, port default

Which ports block?

┌────────┐                                                           ┌──────────┐
│    DP 1├── 32768 1 52:54:00:4b:99:08 8001 ─────────────────────────┤1 RP      │
│SW1 DP 2├── 32768 1 52:54:00:4b:99:08 8002 ─────────────────────────┤2 BLK  SW2│
│    DP 3├── 32768 1 52:54:00:4b:99:08 8003 ─────────────────────────┤3 BLK     │
└────────┘                                                           └──────────┘

• All ports on root bridge are DP.
• SW2 gets three BPDUs, the best BPDU is on port 1, it has the lowest port number.
• SW2 sets the other two ports to BLK.

Port blocking, port priority

Which ports block?

┌────────┐                                                          ┌──────────┐
│    DP 1├── 32768 1 52:54:00:4b:99:08 8001 ────────────────────────┤1 BLK     │
│SW1 DP 2├── 32768 1 52:54:00:4b:99:08 8002 ────────────────────────┤2 BLK  SW2│
│    DP 3├── 32768 1 52:54:00:4b:99:08 0003 ────────────────────────┤3 RP      │
└────────┘                                                          └──────────┘

• All ports on root bridge are DP.
• SW2 gets three BPDUs, the best BPDU is on port 3, it has the lowest priority. 00
• SW2 sets the other two ports to BLK.

!
! SW1
!
interface 3
 spanning-tree vlan 1 port-priority 0

Port blocking, cost

Which ports block?

┌────────┐                                                          ┌─────────┐
│    DP 1├── 32768 1 52:54:00:4b:99:08 8001 ────────────── Cost 4 ──┤1 BLK    │
│SW1 DP 2├── 32768 1 52:54:00:4b:99:08 8002 ────────────── Cost 1 ──┤2 RP  SW2│
│    DP 3├── 32768 1 52:54:00:4b:99:08 8003 ────────────── Cost 4 ──┤3 BLK    │
└────────┘                                                          └─────────┘

• All ports on root bridge are DP.
• SW2 gets three BPDUs, the best BPDU is on port 2, The local switch marked the received cost on that port as 1
• SW2 sets the other two ports to BLK.

!
! SW2
!
interface 2
 spanning-tree vlan 1 cost 1

Topology change notifications (TCNs)

• A TCN is a kind of BPDU message
• There is no root ID or bridge ID
• The TCN is sent out the RP

Spanning Tree Protocol
    Protocol Identifier: Spanning Tree Protocol (0x0000)
    Protocol Version Identifier: Spanning Tree (0)
    BPDU Type: Topology Change Notification (0x80)

1. Bridge sees change in STP topology, sends TCN to upstream bridge.
2. Upstream sees TCN, sends a regular BPDU back with TCA bit set.
3. Upstream bridge sends TCN upstream, this continues until TCN reaches the root.
4. Root Bridge sees the TCN, marks BPDUs with TC bit set.
5. All bridges see TC, and shorten their MAC aging timer to Forward Delay (default 15 seconds).
6. Root bridge stops sending TCs.

The max-age default for Cisco is 300 seconds (5 minutes).

Receiving a TCN shortens the aging timer to forward delay usually 15 seconds. This means any server that is not actively sending, will have it’s traffic flooded onto that VLAN.

switch# show mac address-table aging-time 
Global Aging Time:  300

Captures

STP-stable-state.pcap

STP-TCN-topology-change-notification.pcap

References

R. Perlman, Interconnections: Bridges, Routers, Switches, and Internetworking Protocols, 2nd ed. Boston, MA: Addison-Wesley, 1999.

Understand and Tune Spanning Tree Protocol Timers - Cisco

Layer 2 Configuration Guide, Cisco IOS-XE 17.16.X

1. Interconnections - Radia Perlman, page 67. ↩

Finding TCNs

switch# show spanning-tree vlan 20 detail | s Spanning
 VLAN0020 is executing the rstp compatible Spanning Tree protocol
  Bridge Identifier has priority 32768, sysid 20, address aabb.cc00.0100
  Configured hello time 2, max age 20, forward delay 15, transmit hold-count 6
  Current root has priority 8212, address aabb.cc00.0200
  Root port is 7 (Ethernet1/2), cost of root path is 200
  Topology change flag not set, detected flag not set
  Number of topology changes 8 last change occurred 01:07:20 ago   < ----
          from Ethernet1/2                                         < ----
  Times:  hold 1, topology change 35, notification 2
          hello 2, max age 20, forward delay 15 
  Timers: hello 0, topology change 0, notification 0, aging 300

On the device

switch# show spanning-tree vlan 20 detail | i VLAN|transitions 
 VLAN0020 is executing the rstp compatible Spanning Tree protocol
 Port 2 (Ethernet0/1) of VLAN0020 is designated forwarding 
   Number of transitions to forwarding state: 2
 Port 4 (Ethernet0/3) of VLAN0020 is alternate blocking 
   Number of transitions to forwarding state: 1
 Port 7 (Ethernet1/2) of VLAN0020 is root forwarding 
   Number of transitions to forwarding state: 2
 Port 8 (Ethernet1/3) of VLAN0020 is alternate blocking 
   Number of transitions to forwarding state: 0
 Port 12 (Ethernet2/3) of VLAN0020 is designated forwarding 
   Number of transitions to forwarding state: 2

In the logs

switch# show logging | i %LINK
*Jul  8 04:22:24.660: %LINK-3-UPDOWN: Interface Ethernet0/0, changed state to up
*Jul  8 04:22:24.702: %LINK-3-UPDOWN: Interface Ethernet0/1, changed state to up
*Jul  8 04:22:24.715: %LINK-3-UPDOWN: Interface Ethernet0/2, changed state to up
*Jul  8 04:22:24.740: %LINK-3-UPDOWN: Interface Ethernet0/3, changed state to up
*Jul  8 04:22:24.769: %LINK-3-UPDOWN: Interface Ethernet1/0, changed state to up
*Jul  8 04:22:24.794: %LINK-3-UPDOWN: Interface Ethernet1/1, changed state to up
*Jul  8 04:22:24.819: %LINK-3-UPDOWN: Interface Ethernet1/2, changed state to up
*Jul  8 04:22:24.858: %LINK-3-UPDOWN: Interface Ethernet1/3, changed state to up
*Jul  8 04:22:24.888: %LINK-3-UPDOWN: Interface Ethernet2/0, changed state to up
*Jul  8 04:22:24.903: %LINK-3-UPDOWN: Interface Ethernet2/1, changed state to up
*Jul  8 04:22:24.927: %LINK-3-UPDOWN: Interface Ethernet2/2, changed state to up
*Jul  8 04:22:24.942: %LINK-3-UPDOWN: Interface Ethernet2/3, changed state to up
*Jul  8 04:22:24.965: %LINK-3-UPDOWN: Interface Ethernet3/0, changed state to up
*Jul  8 04:22:24.989: %LINK-3-UPDOWN: Interface Ethernet3/1, changed state to up
*Jul  8 04:22:25.013: %LINK-3-UPDOWN: Interface Ethernet3/2, changed state to up
*Jul  8 04:22:25.033: %LINK-3-UPDOWN: Interface Ethernet3/3, changed state to up
*Jul  8 04:22:26.685: %LINK-5-CHANGED: Interface Vlan1, changed state to administratively down
*Jul  8 04:24:58.575: %LINK-5-CHANGED: Interface Ethernet0/0, changed state to administratively down
*Jul  8 04:25:06.138: %LINK-3-UPDOWN: Interface Ethernet0/0, changed state to up
*Jul  8 04:26:59.260: %LINK-5-CHANGED: Interface Ethernet0/0, changed state to administratively down
*Jul  8 04:27:11.982: %LINK-3-UPDOWN: Interface Ethernet0/0, changed state to up
*Jul  8 04:28:43.205: %LINK-5-CHANGED: Interface Ethernet0/0, changed state to administratively down
*Jul  8 04:31:09.988: %LINK-3-UPDOWN: Interface Ethernet0/0, changed state to up
*Jul  8 04:33:53.881: %LINK-5-CHANGED: Interface Ethernet0/0, changed state to administratively down
*Jul  8 04:34:02.140: %LINK-3-UPDOWN: Interface Ethernet0/0, changed state to up
*Jul  8 05:00:52.111: %LINK-5-CHANGED: Interface Ethernet0/0, changed state to administratively down
*Jul  8 05:00:59.749: %LINK-3-UPDOWN: Interface Ethernet0/0, changed state to up
*Jul  8 05:03:48.728: %LINK-5-CHANGED: Interface Ethernet0/0, changed state to administratively down
*Jul  8 05:03:54.050: %LINK-3-UPDOWN: Interface Ethernet0/0, changed state to up
*Jul  8 05:07:04.113: %LINK-5-CHANGED: Interface Ethernet0/0, changed state to administratively down
*Jul  8 05:07:06.713: %LINK-3-UPDOWN: Interface Ethernet0/0, changed state to up
*Jul  8 05:07:31.603: %LINK-5-CHANGED: Interface Ethernet0/0, changed state to administratively down
*Jul  8 05:07:36.280: %LINK-3-UPDOWN: Interface Ethernet0/0, changed state to up
*Jul  8 05:11:32.247: %LINK-3-UPDOWN: Interface Vlan10, changed state to up
*Jul  8 06:35:29.308: %LINK-5-CHANGED: Interface Ethernet0/0, changed state to administratively down
*Jul  8 06:35:43.756: %LINK-3-UPDOWN: Interface Ethernet0/0, changed state to up

RSTP

Discarding

• On
• Blocking data traffic
• Might be receiving BPDUs
  • STP states: Disabled, Blocking and Listening

Learning

• On
• Building the CAM table
• Only forwarding BPDUs

Forwarding

• On
• Forwarding BPDUs and Data traffic

Root port

• AKA RP
• A switch has one RP

Designated port

• AKA DP
• Each link can have only one DP

Backup port

• Typically connected to a hub
• Used on shared segments
• Provides link redundancy

Edge ports

• Edge of the STP topology, meant for hosts
• Edge ports have portfast turned on

Non-Edge ports

• Have received a BPDUs
  • Could be attached to switches

Point-to-point ports

• Connect to other RSTP switches with full duplex

Propose bit

• Added in 802.1w

• Switches compare BPDUs

  • Switch with Superior BPDU
    • Sets the Propose and DP bits

Agree Bit

• Sent in response to a Propose Bit
  • “Yeah, I agree”
  • Transition immediately to forwarding

Packet headers

Spanning Tree Protocol
    Protocol Identifier: Spanning Tree Protocol (0x0000)
    Protocol Version Identifier: Rapid Spanning Tree (2)
    BPDU Type: Rapid/Multiple Spanning Tree (0x02)
    BPDU flags: 0x0e, Port Role: Designated, Proposal
        0... .... = Topology Change Acknowledgment: No
        .0.. .... = Agreement: No
        ..0. .... = Forwarding: No
        ...0 .... = Learning: No
        .... 11.. = Port Role: Designated (3)
        .... ..1. = Proposal: Yes
        .... ...0 = Topology Change: No
    Root Identifier: 0 / 1 / 52:54:00:eb:eb:96
    Root Path Cost: 4
    Bridge Identifier: 32768 / 1 / 52:54:00:a9:d4:07
    Port identifier: 0x8001
    Message Age: 1
    Max Age: 20
    Hello Time: 2
    Forward Delay: 15
    Version 1 Length: 0

Proposal

Like 802.1D, we have BPDUs. To speed things up some logic has been added.

Is this port full-duplex? If so, it’s point-to-point, and if so, there is a RP and DP.

No BPDUs being received? Wait for the forward delay, transition port to forwarding.

Fastest scenario, the Would-Be DP is already transmitting

Captures

RSTP-initial-bringup-fastest.pcap

References

Cisco - Spanning Tree Protocol

MST

Cisco switches provide three kinds of spanning tree modes.

switch(config)# spanning-tree mode ?
  mst         Multiple spanning tree mode
  pvst        Per-Vlan spanning tree mode
  rapid-pvst  Per-Vlan rapid spanning tree mode

The Industry has three kinds of interop.

Industry liked what Cisco was doing with “per vlan” so MST merges that feature into 802.1s.

Terms

CST — Common Spanning Tree

• For interoperability we fall back to 802.1D, with one STP

CST Root

• The one root bridge for the entire CST

MST — Multiple Spanning Trees

MSTI — MST Instance

• A group of vlans on a common MSTI

MST Region

• A group of switches with the same high-level config

MST Region Boundary

• Where a MST region sends and receives BPDUs with a different switching instance (could be STP, RSTP, MST but a different region)

MST Region Root

• MST can propagate multiple MSTIs. Each MSTI can have its own root

IST — Internal Spanning Tree

• Instance 0, the first instance

IST Root

• The IST root is the CST root

CIST — Common and Internal Spanning Tree

• MST can derive what STP would do, for interop
• If a switch in a MST region is connected to a much older switch, it will present it with a CST

MST Region Boundary

• Any port that connects to a 802.1D or 802.1W device

PVST Simulation

• If a MST switch is root for the whole switch topology
  • Map the IST (instance 0) onto the CST, by sending BPDUs for all the VLANS it sees on the neighbor

PVST Simulation Check

• If a MST device receives a superior BPDU
  • Shut down the port

Packet

From wireshark

Spanning Tree Protocol                                                                                  
    Protocol Identifier: Spanning Tree Protocol (0x0000)                                                
    Protocol Version Identifier: Multiple Spanning Tree (3)                                             
    BPDU Type: Rapid/Multiple Spanning Tree (0x02)                         ───────┐                     
    BPDU flags: 0x7c, Agreement, Forwarding, Learning, Port Role: Designated      │                     
    Root Identifier: 0 / 0 / 52:54:00:5f:ff:79                                    │ Inter-op Data       
    Root Path Cost: 20000                                                         │   The CST MST       
    Bridge Identifier: 4096 / 0 / 52:54:00:82:c0:7f                               │     presents outside
    Port identifier: 0x800e                                                       │       its region    
    Message Age: 1                                                                │                     
    Max Age: 20                                                                   │                     
    Hello Time: 2                                                                 │                     
    Forward Delay: 15                                                      ───────┘                     
    Version 1 Length: 0                                                                                 
    Version 3 Length: 80                                                                                
    MST Extension                                                                                       
        MST Config ID format selector: 0                       ───────┐                                 
        MST Config name: green                                        │   What MST                      
        MST Config revision: 3                                        │    shows                        
        MST Config digest: 059b580e0d7ab80bcf83df54c634d006           │      other                      
        CIST Internal Root Path Cost: 20000                           │        devices                  
        CIST Bridge Identifier: 32768 / 0 / 52:54:00:04:67:92         │          in                     
        CIST Remaining hops: 19                                       │           its                   
        MSTID 1, Regional Root Identifier 0 / 52:54:00:82:c0:7f       │            Region               
                                                               ───────┘                                 

MST keeps track of a few things:

• Root Identifier, the interop bridge for the whole topology

• Bridge Identifier, the interop field that makes a MST region appear as one bridge outside of it

• CIST Bridge, the bridge that originated the BPDU. Not visible outside of MST

• Regional Root, the bridge that is the root for green. Not visible outside of MST

Basic config

spanning-tree mode mst

More involved config

default spanning-tree mst configuration
spanning-tree mst configuration
 name blue
 revision 3
 instance 1 vlan 10, 20, 30, 40

Config validation

S21# show spanning-tree mst configuration 
Name      [red]
Revision  3     Instances configured 2

Instance  Vlans mapped
--------  ---------------------------------------------------------------------
0         1-19,21-39,41-4094
1         20,40
-------------------------------------------------------------------------------

Outputs

• A switch ff79 in a different MST region is the root for the CST.
• Our regional root c07f is one hop away.

S32# show spanning-tree mst 

##### MST0    vlans mapped:   1-9,11-19,21-29,31-39,41-4094
Bridge        address 5254.0004.6792  priority      32768 (32768 sysid 0)
Root          address 5254.005f.ff79  priority      0     (0 sysid 0)
              port    Gi3/2           path cost     20000    
Regional Root address 5254.0082.c07f  priority      4096  (4096 sysid 0)
                                      internal cost 20000     rem hops 19
Operational   hello time 2 , forward delay 15, max age 20, txholdcount 6 
Configured    hello time 2 , forward delay 15, max age 20, max hops    20

Interface        Role Sts Cost      Prio.Nbr Type
---------------- ---- --- --------- -------- --------------------------------
Gi3/1            Desg FWD 20000     128.14   P2p 
Gi3/2            Root FWD 20000     128.15   P2p 

##### MST1    vlans mapped:   10,20,30,40
Bridge        address 5254.0004.6792  priority      32769 (32768 sysid 1)
Root          address 5254.0082.c07f  priority      1     (0 sysid 1)
              port    Gi3/2           cost          20000     rem hops 19

Interface        Role Sts Cost      Prio.Nbr Type
---------------- ---- --- --------- -------- --------------------------------
Gi3/1            Desg FWD 20000     128.14   P2p 
Gi3/2            Root FWD 20000     128.15   P2p 

Captures

MST-region-frame.pcap

References

Cisco - Understand the Multiple Spanning Tree Protocol (802.1s)

Layer 23 - MSTP Protocol Explained: Multiple Spanning Tree in Depth

Defend Spanning Tree

STP on its own is kind of brittle, so there are lots of features to enable to make it more resilient.

DTP

DTP — Dynamic Trunking Protocol.

DTP is a Cisco proprietary point-to-point protocol, for full-duplex switchlinks.

An older feature intended to automate parts of network setup, you could set one switch to dynamic desireable and it will form trunks automatically.

The best practice is to disable this feature on trunks links with switchport nonegotiate

DTP is normally every 30 seconds.

Modes

switchport mode dynamic auto

• Send DTP
• Usually Default
• Become a trunk if the neighbor is a trunk
• Become a trunk if the neighbor is set to desireable

switchport mode access

• Probably doesn’t¹ send DTP frames
• If it does, it asks the neighbor to become an access port

switchport mode trunk

• Send DTP
• Asks the neighbor to become a trunk port

switchport mode dynamic desirable

• Become a trunk only if the neighbor can be convinced to become a trunk
• Works with trunk, desirable or auto

switchport nonegotiate

• Disable DTP
• Only works with access or trunk

Commands

show dtp
debug dtp packets

Verification

Performed on a C3560CX, running 15.2(7r)E.

No DTP with switchport mode access

switch# show dtp interface tenGigabitEthernet 1/0/7 | i Enabled
  Enabled:                                  no

References

Dynamic Trunking Protocol - Wikipedia

Configuring VLAN Trunks Cisco Catalyst 9500 Series Switches - Cisco IOS XE 26.x.x

Solved: Switchport Mode Access question - Cisco Community

1. The IOS-XE guide says “negotiates” but in lab, I don’t see DTP frames on these ports. I checked on IOSv, IOL, and a C3560CX running 15.2(7r)E.
   Consensus online is this turns off DTP. ↩

Portfast

The best practice is to configure this on every port connected to hosts.

• Hosts and end devices
• Transition to forwarding faster
• Do not send TCN BPDUs

Note

This stops TCNs, which cause the max-age timer to reduce to the duration of the forward delay on receipt of a TCN.

Hosts wich require portfast

• PXE
• DHCP

The minimum time for 802.1D is 30 seconds, (2x the forwarding delay).

This means some computers will miss DHCP, and sometimes PXE fails.

Config

Enable on all access ports

spanning-tree portfast default

Enable on trunks

Some trunk ports connect to servers. Portfast can be enabled on those too.

spanning-tree portfast trunk

BPDU Guard

• Only works if the attached device sends a BPDU. Cannot prevent a switch from being attached to a port. 802.1x helps with that.

Detects A BPDU, and Err-Disables A port

The global command only affects ports that have portfast already turned on, i.e. this is an edge feature.

switch(config)# spanning-tree portfast bpduguard default

… should be set so access ports go errdisable when a rogue switch is connected and require an operator to correct.

Seeing err-disabled status

switch# show int status

Port      Name               Status       Vlan       Duplex  Speed Type 
[output omitted]
Et2/3                        err-disabled 1            auto   auto unknown
Et3/0                        connected    trunk        auto   auto unknown
Et3/1                        connected    1            auto   auto unknown

Turning on automated recovery

switch(config)# errdisable recovery cause bpduguard

Verify

switch# show errdisable recovery 
ErrDisable Reason            Timer Status
-----------------            --------------
arp-inspection               Disabled
bpduguard                    Enabled

[output omitted]
          
Interface       Errdisable reason       Time left(sec)
---------       -----------------       --------------
unicast-flood                Disabled
vmps                         Disabled
psp                          Disabled
dual-active-recovery         Disabled
evc-lite input mapping fa    Disabled
Recovery command: "clear     Disabled

Timer interval: 300 seconds

Interfaces that will be enabled at the next timeout:

Interface       Errdisable reason       Time left(sec)
---------       -----------------       --------------
Et2/3                  bpduguard          296

BPDU Filter

Maybe you don’t want a port to send BPDUs. This effectively turns off spanning-tree.

Warning

This feature melts networks. It disables spanning tree.

Caution

This is not deterministic. It works differently per port, vs globally.

From the IOS-XE config guide

The BPDU filtering feature can be globally enabled on the switch or can be enabled per interface, but the feature operates with some differences.

Enabling BPDU filtering on PortFast enabled interfaces at the global level keeps those interfaces that are in a PortFast operational state from sending or receiving BPDUs.

The interfaces still send a few BPDUs at link-up before the switch begins to filter outbound BPDUs. You should globally enable BPDU filtering on a switch so that hosts that are connected to these interfaces do not receive BPDUs. If a BPDU is received on a PortFast enabled interface, the interface loses its PortFast operational status, and BPDU filtering is disabled.

Enabling BPDU filtering on an interface without also enabling the PortFast feature keeps the interface from sending or receiving BPDUs.

Globally

If a port, is running portfast, this feature will transmit a few BPDUs when the port first turns on.

spanning-tree portfast bpdufilter default

Per interface

This port will never send BPDUs.

interface 1
  spanning-tree bpdufilter enable

References

Layer 2 Configuration Guide, Cisco IOS XE 26.x.x (Catalyst 9300 Switches) - Configuring Optional Spanning-Tree Features Cisco IOS XE 26 - Cisco

Root Guard

For ports which must participate in STP, but never accept a superior BPDU.

• Downlinks

Unlike BPDU Guard, the port still participates in spanning tree.

Example

Normally SW1 is the root bridge

┌──────┐                                                      ┌──────┐
│SW1 DP├─ 0/1/52:54:00:e8:3a:ff ─── 4096/1/52:54:00:4b:99:08 ─┤RP SW2│
└──────┘                                                      └──────┘

Someone configures SW2 to be the root by making the switch priority 0.

┌──────┐                                                      ┌──────┐
│SW1 RP├─ 0/1/52:54:00:e8:3a:ff ────── 0/1/52:54:00:4b:99:08 ─┤DP SW2│
└──────┘                                                      └──────┘

Config

This can be prevented with this config. Root guard goes onto DPs.

!
! SW1
!
interface 1
  spanning tree guard root

Verification

sw1# show spanning-tree 

VLAN0001
  Spanning tree enabled protocol ieee
  Root ID    Priority    4097
             Address     5254.004f.110e
             This bridge is the root
             Hello Time   2 sec  Max Age 20 sec  Forward Delay 15 sec

  Bridge ID  Priority    4097   (priority 4096 sys-id-ext 1)
             Address     5254.004f.110e
             Hello Time   2 sec  Max Age 20 sec  Forward Delay 15 sec
             Aging Time  300 sec

Interface           Role Sts Cost      Prio.Nbr Type
------------------- ---- --- --------- -------- --------------------------------
Gi0/0               Desg BKN*4         128.1    P2p *ROOT_Inc 
Gi0/2               Desg BKN*4         128.3    P2p *ROOT_Inc 

Logs

*May  3 20:14:45.169: %SPANTREE-2-ROOTGUARD_BLOCK: Root guard blocking port GigabitEthernet0/0 on VLAN0001.  

Picture Example

Image courtesy of BRKENS-2031.

References

Cisco Live - Enterprise Campus Design - Marcin Hamróz, and Jarosław Gawron - BRKENS-2031

Loop Guard

A unidirectional failure on a root or alternate port will cause spanning tree to loop.

Loopguard enforces a simple rule.

If a port was receiving BPBUs and suddenly it stops, don’t change the STP.

• This is one of the unidirectional preventatives
• This is only for switch-to-switch ports

… if a port doesn’t get a BPDU, it enters STP loop-inconsistent disabling the port.

Terms

Unidirectional Link

A failure where one side of a fiber pair is broken.

A unidirectional failure will always result in one side not getting information.

• SW1 sends BPDUs
• SW2 never gets BPDUs

Topology

• SW1 is a root bridge
• SW2 is experiencing a UL failure
• SW2 will transition Port 1 to a RP

                                                      No BPDU Received
┌────────────────────┐                            ┌────────────────────┐
│     ┌─────────────┐│                            │ ┌────────────┐     │
│     │ Port ┌────┐ ││ BPDU ────►XXXX             │ │┌────┐ Port │     │
│     │  1   │ TX ├─││────────────── Fiber Cut! ──│─│┤ RX │  1   │     │
│ SW1 │      └────┘ ││                            │ │└────┘      │ SW2 │
│     │  RP  ┌────┐ ││                            │ │┌────┐  DP  │     │
│     │      │ RX ├─││────────────────────────────│─│┤ TX │      │     │
│     │      └────┘ ││                ◄───── BPDU │ │└────┘      │     │
│     └─────────────┘│                            │ └────────────┘     │
└────────────────────┘                            └────────────────────┘

Config

Default

spanning-tree loopguard default

Per port

interface 1
  spanning-tree guard loop

References

Understand STP Loop Guard and UDLD Features - Cisco

UplinkFast

An older feature, doesn’t work with rapid-pvst, or MST.

• Bypasses listening and learning states.
• Blasts fake traffic out the UplinkFast Port to program the upstream CAM tables.
• Only recommended on the access layer.

Config

spanning-tree uplinkfast 

Validation

Port costs are raised by 3000.

access# show spanning-tree 

VLAN0001
  Spanning tree enabled protocol ieee
  Root ID    Priority    24577
             Address     5254.00bf.ea62
             Cost        3004
             Port        1 (GigabitEthernet0/0)
             Hello Time   2 sec  Max Age 20 sec  Forward Delay 15 sec

  Bridge ID  Priority    49153  (priority 49152 sys-id-ext 1)
             Address     5254.007a.bd39
             Hello Time   2 sec  Max Age 20 sec  Forward Delay 15 sec
             Aging Time  15  sec
  Uplinkfast enabled

Interface           Role Sts Cost      Prio.Nbr Type
------------------- ---- --- --------- -------- --------------------------------
Gi0/0               Root FWD 3004      128.1    P2p 
Gi0/1               Altn BLK 3004      128.2    P2p 

!
! After shutting down the port in CML
!
*Jun  4 02:41:34.557: %SPANTREE_FAST-7-PORT_FWD_UPLINK: VLAN0001 GigabitEthernet0/1 moved to Forwarding (UplinkFast).

References

Layer 2 Configuration Guide, Cisco IOS XE 17.13.x (Catalyst 9300 Switches) - Configuring Optional Spanning-Tree Features Cisco Catalyst 9300 Series Switches - Cisco

Port Security

The default settings for port security will not age out learned mac addresses. To get aging back to the mac address table default of 5 minutes, set this feature to 5.

Config

interface GigabitEthernet0/0
 switchport access vlan 10
 switchport mode access
 switchport port-security maximum 2
 switchport port-security aging time 5
 switchport port-security
 negotiation auto
 spanning-tree portfast edge

Validation

This is the primary table for this feature. This table is used to populate the mac address table.

switch# show port-security address
               Secure Mac Address Table
-----------------------------------------------------------------------------
Vlan    Mac Address       Type                          Ports   Remaining Age
                                                                   (mins)    
----    -----------       ----                          -----   -------------
  10    5254.000d.6573    SecureDynamic                 Gi0/0       10
-----------------------------------------------------------------------------

Scraping the mac-address table for things programmed in by the port security feature.

switch# show mac address-table secure 
          Mac Address Table
-------------------------------------------

Vlan    Mac Address       Type        Ports
----    -----------       --------    -----
  10    5254.000d.6573    STATIC      Gi0/0

Asking port security how many ports are currently controlled by the feature.

switch #show port-security 
Secure Port  MaxSecureAddr  CurrentAddr  SecurityViolation  Security Action
                (Count)       (Count)          (Count)
---------------------------------------------------------------------------
      Gi0/0              2            1                  0         Shutdown
---------------------------------------------------------------------------
Total Addresses in System (excluding one mac per port)     : 0
Max Addresses limit in System (excluding one mac per port) : 4096

References

Cisco - Port Security - IOS-XE 17.14 on the C9300

UDLD

This is the more common way to prevent ULD Failures, enabling ULD.

• ULD: Unidirectional Link Detection.

This feature has two modes:

• Normal: Detect a fiber port that’s miss-cabled. Maybe port 1 and port 2 (four fibers total) got mixed up.

• Aggressive: Detect one way traffic on both fiber, and twisted pair.

From Cisco:

• On fiber-optic or twisted-pair links, one of the ports cannot send or receive traffic.
• On fiber-optic or twisted-pair links, one of the ports is down while the other is up.
• One of the fiber strands in the cable is disconnected.

Requirements

• Both devices need this feature turned on.
• Both sides need the same mode (normal or aggressive)

Global

udld enable

Per interface

interface 1
  udld enable

Verification

sw1# show udld neighbors 
Port     Device Name   Device ID     Port ID    Neighbor State
----     -----------   ---------     -------    --------------
Gi0/0    91YBLF6S1KI     1            Gi0/0      Bidirectional

Capture

UDLD.pcap

Reference

Layer 2 Configuration Guide, Cisco IOS XE 17.16.x (Catalyst 9500 Switches)

VTP

Caution

VTP Melts Networks.

See VTP Bomb.

VTP — Vlan Trunk Protocol

• Cisco proprietary
• Runs on trunk ports
• Comes in three versions
  • Defaults to v1

Design guidance

transparent or off.

See BRKENS-2031.

Modes

Server

• Create, Modify, Delete
• Default
• Other Servers can modify based on Revision

Client

• Gets VLANs from a server

Transparent

• Don’t program the control plane of this device with any recieved VTP packets
• Forward any received VTP packets recieved to other trunk links
• A version 2 thing

Off

• Don’t program the control plane
• Do not forward VTP

Version 3

• Saves VLANS to vlan.dat
• Primary and secondary servers
  • Default is secondary
• Private VLANs
• Extended VLANs
  • 1006 to 4094
• Propigating MST info

References

VLANs VTP - Cisco

Understand VTP - Cisco

Configure VTP - Cisco

Configure VTP - VLAN Configuration Guide - IOS XE 17.18.x - Cisco

Cisco Live - Enterprise Campus Design - Marcin Hamróz, and Jarosław Gawron - BRKENS-2031

VTP Bomb

Caused when one VTP switch overwrites the VLANS of other VTP switches.

See VTP Design Guidance.

Terms

VBS — VTP Bomb Switch

• Usually sits around for a bit until it’s plugged back in

VTP isn’t really client-server

It operates more like peer-to-peer.

VTPv1 and VTPv2 allow multiple switches to be servers.

Any server switch can make VLAN changes.

VTP ties the desired VLAN state to configuration revision number

Setup one - repurposed decom

• VTP revision number: 50
• VTP mode: server
• Switch is removed
• VLANs deleted
• VTP revision number: 58
• VBS Created
  • Stored
  • Pulled out of storage
  • Plugged into trunk port
  • Has higher revision number
  • Deletes VLANs

Setup two - site move

• VTP revision number: 21
• VTP mode: server
• Multisite network
  • Templated config
  • Same VTP domain everywhere
• Biggest site
• VBS Created
  • Stored
• Site-B has revision: 15
• Site-B needs a new switch
• VBS is used
  • Pulled out of storage
  • Goes to Site-B
  • Plugged into trunk port
  • Has higher revision number
  • Deletes VLANs

Setup three - lab work

• VTP revision number: 38
• VTP mode: server
• Junior engineer
• Doesn’t fear VTP
• Disconnects a switch from production
• Takes to desk
• Practices with making vlans
• VTP revision number: 45
• VBS Created
• A switch is needed for an expansion
• VBS is used
  • Taken off desk
  • Plugged into trunk port
  • Has higher revision number
  • Deletes VLANs

Stories

JumpyMud7539

I’ve just recovered from over writing a hospitals vlan database when collecting a new looking switch from the store room that was unboxed and installing it in the production network without really checking the running config.

VTP obviously had it’s own agenda. I am a network engineer who boasts no qualifications just experience on the job and the ability to blog my way through.

I must say when things go catastrophically wrong is when I learn best. I’m surrounded by good engineers who say that what they teach you in certifications gives you good theory but it doesn’t always relate to production environments.

As soon as it happened, I knew the problem and how to fix it regardless, I became too complacent. I wont lose my job, but i felt really bad. I feel I’ve learned a good lesson, I’ve learnt more about vtp, what we need to do going forward.

The organisation has also agreed to put me on a ccna course to enable me to progress.

Source

FLATLANDRIDER

Setting up a new switch at a remote site. Cisco switch with VTP deployed.

I set up the basics and was getting VTP set up. The device was in client mode but I needed to change the VTP version.

Well you can only change the version in server mode. So I changed it to server mode so I could change the version.

As soon as I did that I lost connection to the remote site. The switch had pushed its empty vlan database to all other switches in the site, deleting all vlans. Even the true VTP server was overwritten.

I took the entire remote site down, which was in a different country. I ended up having to remote into someone’s computer using their hotspot so I could gain access to the core switch (VTP server) and recreate the vlan database.

Source

Win_Sys

I think we have all done this at one point or another.

I took down an entire school once when I didn’t check VTP. Wiped the DC switch of most of its VLANs.

Source

[deleted]

VTP mess-up (yes, I know)

I am a student employee, and I have been put in charge of re-speccing our VTP so we avoid the typical vulnerability of VTP where you plugin a switch from a different location and wipe out the whole HO. (Very grateful for the responsibility, but it makes my nuts sweat, tbh).

So in my department we are connected to switch A.

• In server-mode
• Pruning enabled
• Password enabled
• Version 2

So I think “aight, let’s go”, and change it to the new domain. 5 seconds later 30 employees are DC. I panick (not too hard anyway).

My superior doesn’t know what happened, tries to pick up the pieces, but my console window breaks down.

We connect to the switch via console.

Turn it back to the old domain

VOILA, it works!

I don’t dare touch VTP again.

Dear /networking, what happened?!

Source

gypsy_endurance

Back in 2001’ish, was working Tier 2 networking shop at a Fin Srv company.

I was adding two 1RU switches to a very large Cisco campus network for a particular business unit (The Bond Pit).

Anyhoo, prestaged everything at my desk and waited for the change window. Plugged the switches in and was looking through some operational commands when a guy walks in and says “are you doing something with the network?”

I walk out of the room I’m in following him and about 30+ bond traders are standing up at their desks with their hands in the air complaining about the network being down.

The engineer that did the configs I was implementing, looked at the configs on the 6500s that we were uplinked to and all the vlans were gone.

Long story, shortened…Cisco’s client/server VTP feature didn’t actually act like a client/server.

The configuration revision in the VTP database was higher, or lower (don’t remember anymore) on the 1RU Cisco “clients” and overwrote the vlan database on the “server” 6500s.

The Cisco TAC case answer was “that’s how it’s supposed to work.” Talking about fearing for your job for the next 12 hrs. LOL!

Source

Joe Morris

Plugged a previous used switch into a switched network. All Cisco. Running VTP

The newly connected switch won the vtp war and wiped the vlan db from all other switches.

Source

Metro Ethernet

General caveats

• Shut down interfaces before configuring (best practice on ME3400)
• No DTP — dynamic trunk negotiation does not exist
• No VTP — VLAN Trunking Protocol does not exist
• Default STP mode is PVST
• Typically deployed as the last hop (service provider edge / CPE)
• Port model is based on port-types and UNI VLANs rather than standard IOS switchport modes

Port types (MEF terminology)

UNI — user network interface

• Connects to customer end devices (phones, computers, routers at the customer site)
• All ports except uplinks default to UNI
• Can only forward traffic toward NNI ports
• No STP
• No CDP
• No Link Aggregation

ENI — enhanced network interface

• Connects to routers or switches (slightly trusted devices)
• Supports STP
• Supports CDP
• Supports Link Aggregation

NNI — network node interface

• Network-to-Network — connects to other provider nodes/uplinks
• No Layer 2 protocol filtering — passes BPDUs, CDP, etc. transparently

Port type comparison

Private VLANs

Default port type on a Catalyst ME3400 is UNI (User Network Interface)

UNI ports can only send traffic to NNI ports in the same vlan. The default UNI mode is isolated.

show vlan uni-vlan type

show port-type

You can set the uni-vlan type with this command:

vlan 100
  uni-vlan community

This example uses a ME3400.

Config example

vlan 100
private-vlan primary
private-vlan association 1000,2000,3000
!
vlan 1000
private-vlan isolated
!
vlan 2000
private-vlan community
!
vlan 3000
private-vlan community
!
!
interface FastEthernet0/2
!
! Tell it its a private-vlan host
! Tell it which private VLANs its in
!
 switchport private-vlan host-association 100 1000
 switchport mode private-vlan host
 duplex full
end
!
interface GigabitEthernet0/1
 port-type nni
 switchport private-vlan mapping 100 1000,2000,3000
 switchport mode private-vlan promiscuous
 speed nonegotiate
end

vlan 100
 private-vlan association add ... this doesn't work at all!!

Verification

switch #show vlan private-vlan 

Primary Secondary Type              Ports
------- --------- ----------------- ------------------------------------------
100     1000      isolated          Fa0/1, Fa0/2, Gi0/1
100     2000      community         Fa0/3, Fa0/4, Gi0/1
100     3000      community         Gi0/1

SW1#show vlan private-vlan type 

Vlan Type
---- -----------------
100  primary          
1000 isolated         
2000 community        
3000 community  

MPLS

MPLS requires CEF

Frame format

  RFC 3032 - MPLS Label Stack Encoding

  0                   1                   2                   3
   0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1
  +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ Label
  |                Label                  | Exp |S|       TTL     | Stack
  +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ Entry

                      Label:  Label Value, 20 bits
                      Exp:    Experimental Use, 3 bits
                      S:      Bottom of Stack, 1 bit
                      TTL:    Time to Live, 8 bits

MPLS control plane

Local Label - This is what the LSP tells others it wants to recieve.

Outgoing label - What the LSP does next.

Null labels

[3] Implicit Pop the label. Implicit because the label is missing.

[0] Explicit Keep the label, but the destination must pop it. Used for MPLS QoS.

Steps to build the LFIB

1. Find the Next-Hop for the destination prefix

show ip route 3.3.3.4
 * 10.1.2.1, from 10.1.2.1 via Eth0/0

2. Confirm the LDP neighbor behind that Next-Hop

show mpls ldp neighbor 10.1.2.1
 * peer LDP ident 3.3.3.3; Local LDP ident 2.2.2.2

3. Check what label that peer advertised for the prefix

show mpls ldp binding 3.3.3.4 255.255.255.255
 * remote binding: lsr: 3.3.3.3, label: imp-null

4. Verify the resulting LFIB entry

show mpls forwarding 3.3.3.4
- Local 20, Outgoing Pop, Prefix 3.3.3.3/32, Outgoing Int E0/0, Nexthop 10.1.2.1

Testing pseudowires

You can test MPLS pseduowire labels by doing labeled pings.

ping mpls pseudowire 3.3.3.3 11000

L3VPN

Customer Exchanges routes with the provider via a routing protocol

VPN just means isolation. No encryption or confidentiality

PE-CE links are their own island

VRF = VPN Routing Forwarding Instance

You can use loopbacks to route between VRFs

L3VPNs rely on Extended Comunnities.

• Basically just arbitrary TLVs attached to BGP prefixes

VPNV4 = PE to PE Label Information ipv4 unicast vrf = BGP within vrfs for PE to CE connectivity

Example

router bgp 100
 neighbor 3.3.3.3 remote-as 100  ! note the neighbor is the same AS, this is the PE
!
address-family vpnv4
 neighbor 3.3.3.3 activate ! this is a global neighbor, part of our network
 neighbor 3.3.3.3 send-community extended
!
address-family ipv4 unicast vrf red
 neighbor 4.4.4.4 remote-as 400 ! Customer neighbor, in their own VRF instance 
 neighbor 4.4.4.4 activate ! this is if you speak BGP to them. It could be any VRF aware IGP. You need to redistribute from the IGP into BGP

Route distinguishers

• ALL CE routes from ALL VRFs are placed into the same VPNv4 table. What makes them unique to vrfs are the RDs.
  • Transparent to the customer, and only lives on the PE router
  • 100:100:192.168.10/24

Route targets

• This is the BGP extended community
• route-target export adds the community to the outbound update
• route-target import defines what routes to bring into the VRF

One RD per customer site.

show ip bgp vpnv4 all sum VPNv4 show ip bgp sum IPv4

LDP label gets from PE to PE VPN label identifies VRF to remote PE

MPLS DiffServ

• Network is a single DiffServ Domain

• ip2mpls Copy DSCP to EXP

• mpls2mpls Copy EXP Upward and Downward

• mpls2ip Copy EXP into the DSCP

• Uniform changes the markings.

Short-Pipe

• Customer network is a different DiffServ Domain
• ip2mpls # Maybe copy
• Copy top EXP into lower labels.
• Use the exposed DSCP when there is no label.
• When mpls2ip is done, act on the ip information not the prior EXP bits.

Pipe mode (multiple DiffServ domains, default, no change to CE marking)

• ip2mpls # Maybe copy DSCP to EXP.
• SP MPLS exp may be remarked in transit
• IPv4 is not remarked at egress
• Customer marking is unchanged.
• mpls2ip operation is based on what the prior MPLS EXP tag was.
• Must shut off PHP

References

RFC 3270: Multi-Protocol Label Switching (MPLS) Support of Differentiated Services | RFC Editor

IRB on the ASR9K

10.0.0.2/24

 ┌───────┐
 │Host A │
 └───┬───┘
     │  VLAN                                                                  ASR 9k
     │   10
     │          ┌─────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────┐
     │          │            Sub-interfaces                                Bridiging Domain                       Routing Domain          │
     │          │                                          ┌────────────────────────────────────────────────────────────────────────────┐ │
     │          │ ┌──────────────────────────────────────┐ │                                                 |                          │ │
     │          │ │interface g0/0/0/2.10 l2transport     │ │                                                 |                          │ │
     │          │ │  no ip address                       │ │ ┌─────────────────────────────────────────────┐ |                          │ │
     │ g0/0/0/1 │ │  encapsulation dot1q 10 exact        │ │ │ L2vpn                                       │ |                          │ │
     └──────────┼─┤  rewrite ingress tag pop 1 symmetric ├─┼─┤   bridge group BG_test                      │ |                          │ │
                │ └──────────────────────────────────────┘ │ │     bridge-domain BD_1                      │ |         BVI              │ │
                │                                          │ │       interface Gigabit Ethernet 0/0/0/1.10 ├─────────────               │ │
       g0/0/0/2 │ ┌──────────────────────────────────────┐ │ │       interface Gigabit Ethernet 0/0/0/2.10 │ |                          │ │
     ┌──────────┼─┤interface-g0/0/0/2.10-l2transport     ├─┼─┤     routed interface BVI 10                 │ | interface BVI 10         │ │
     │          │ │  no ip address                       │ │ └─────────────────────────────────────────────┘ |   ipv4 address 10.0.0.1  │ │
     │          │ │  encapsulation dot1q 10 exact        │ │                                                 |                          │ │
     │          │ │  rewrite ingress tag pop 1 symmetric │ │                                                 |                          │ │
     │          │ └──────────────────────────────────────┘ │                                                 |                          │ │
     │          │                                          │                                                 |                          │ │
     │ VLAN     │                                          └────────────────────────────────────────────────────────────────────────────┘ │
     │  10      └─────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────┘
 ┌───┴───┐
 │Host B │
 └───────┘
10.0.0.3/24

Simple BBA on the ASR 1k

BBA — Broadband Aggregation

Pulled from very old notes :)

BBA

!
! ASR1k
!
bba-group pppoe PPPOE_USERS
  virtual template 1
!
interface Virtual-Template1
 ip address 10.10.10.1 255.255.255.0
 peer default ip address dhcp
 ppp authentication pap 

Client

2800 Side

Current configuration : 76 bytes
!
interface Dialer1
 ip address dhcp
 encapsulation ppp
 dialer pool 1
!
interface FastEthernet0/0
 no ip address
 duplex auto
 speed aut
 pppoe enable
 pppoe-client dial-pool-number 1
end

PBB

PBB 802.1ah PBB or MAC-in-MAC)

graph LR
    CE1([CE]) ---|UNI .1ad| PEB1[PEB]
    CE2([CE]) ---|UNI .1ad| PEB2[PEB]

    subgraph AL["Access Network · 802.1ad"]
        PEB1 --- PBa1[PB]
        PEB2 --- PBa1
        PBa1 --- PBa3[PB]
        PBa1 --- PBa2
        PBa2[PB] --- PBa3[PB]
    end

    PBa3 ---|UNI .1ah| BEB1
    PBa3 ---|UNI .1ah| BEB2

    subgraph CORE["Core Network · 802.1ah"]
        BEB1[BEB] --- BCB1[BCB]
        BEB2[BEB] --- BCB1
        BCB1 --- BCB2[BCB]
        BCB2 --- BCB3
        BCB1 --- BCB3[BCB]
        BCB3 --- BEB3[BEB]

    end

    BEB3 ---|UNI .1ah| PBb1

    subgraph AR["Access Network · 802.1ad"]
        PBb1[PB] --- PBb3[PB]
        PBb1 --- PBb2
        PBb2[PB] --- PBb3
        PBb3 --- PEB3[PEB]
    end

    PEB3 ---|UNI .1ad| CE3([CE])

Terms

• PB: Provider Bridge.

• PEB: Provider Edge Bridge.

• B-Component: A bridging component contained in a BEB that bridges in provider space (Backbone MAC addresses, B-VLAN).

• BCB: Backbone Core Bridge, An S-VLAN bridge used within the core of a PBBN.

• BEB: Backbone Edge Bridge, A backbone edge bridge positioned at the edge of PBBN that encapsulates customer frames for transmission across a PBBN.

• B-MAC: Backbone MAC Address. An individual MAC address associated with a Provider Instance Port (PIP) and used in creating the MAC header of I-tagged frames transmitted across a PBBN.

• Backbone Service Instance: An instance of the MAC service in a PBBN provided between two or more Virtual Instance Ports (VIPs) in BEBs.

• I-SID: Backbone Service Instance Identifier. A 24-bit field of the backbone service instance tag (I-TAG) that identifies the backbone service instance of a frame. The I-SID defines the service instance that the frame should be “mapped to”.

• I-TAG Backbone Service Instance Tag. A tag with a Ethertype value allocated for IEEE 802.1q backbone service instance tag type.

• B-TAG: Backbone VLAN Tag. A field defined in IEEE 802.1ah provider MAC encapsulation header that conveys the backbone VLAN ID information. The format of the B-TAG is same as that of an IEEE 802.1ad S-TAG field.

• CBP: Customer Backbone Port. A BEB port that can receive and transmit I-tagged frames for multiple customers,

and can assign B-VIDs (backbone VLAN IDs) and translate I-SID on the basis of the received I-SID.

• I-Component: A bridging component contained in a BEB that bridges in the Customer space (Customer MAC addresses, S-VLAN).
• PIP: Provider Instance Port. The set of Virtual Instance Ports (VIPs) that are supported by a single instance.
• Service frame: A frame exchanged between a provider and a customer.
• VIP: Virtual Instance Port. A bridge port on an I-Component in BEB that provides access to a single backbone service instance.

References

L2VPN and Ethernet Services Configuration Guide for Cisco ASR 9000 Series Routers, IOS XR Release 24.1.x, 24.2.x, 24.3.x, 24.4.x - Implementing IEEE 802.1ah Provider Backbone Bridge Cisco ASR 9000 Series Aggregation Services Routers - Cisco

SPAN, RSPAN, ERSPAN

Local

monitor session 1 source interface GigabitEthernet1/0/1 both
monitor session 1 destination interface GigabitEthernet1/0/2

RSPAN

• VLAN Encapsulated
• Does not support layer 2 protocols
  • No CDP, BPDUs, LLDP, etc
• If the source is a trunk port, you can use the filter keyword to select specific vlans

Source switch

vlan 3000
 remote-span
monitor session 1 source interface GigabitEthernet1/0/1 both
monitor session 1 destination remote vlan 3000

Destination switch

vlan 3000
 remote-span
monitor session 1 source remote vlan 3000
monitor session 1 destination interface GigabitEthernet1/0/2

ERSPAN

GRE Encapsulated.

These will encapsulate BPDUs and other Layer 2 protocols.

These need ip routing turned on.

These do not support QoS.

Source switch

monitor session 1 type erspan-source
 !
 ! Could also put a vlan here
 !
 source interface Gi2
 destination
  erspan-id 100
  ip address 10.0.12.2
  origin ip address 10.0.12.1
 no shutdown

Destination switch

monitor session 1 type erspan-destination
 destination interface Gi2
 source
  erspan-id 100
  !
  ! An outside address on this box, not a loopback.
  ! this is the de-encapsulation interface.
  !
  ip address 10.0.12.2
 no shutdown

References

Cisco - Configuring SPAN and RSPAN

Cisco - Configuring ERSPAN

Cisco - ERSPAN Restrictions (IOS XR)

Multicast

Invented by Steve Deering, in his 1991 PhD thesis work Multicast Routing in a Datagram Internetwork.

Multicast is always to a group, a destination, or a set of destinations.

Terms

Multicast

• A one-to-many service using UDP packets destined to group IP address
• Hosts subscribe to the group, routers and switches replicate for the group

IGMP — Internet Group Management Protocol

• Receivers send IGMP to LHR to request multicast streams
• Switches see IGMP (for snooping), and the LHR uses this to build the MDT

PIM — Protocol Independent Multicast

• Multicast capable routers communicate to each over via PIM

IIF — Incoming Interface

• AKA, the RPL interface
• Part of the MDT

OIL — Outgoing Interface List

• Part of the MDT

MDT — Multicast Distribution Tree

• The full set of links participating in multicast, via PIM, IGMP, including IILs, and OILs

RP — Rendezvous Point

• A router designated as the root of a shared tree

(*, G)

• AKA, a shared tree
• Pronounced as “Star comma Gee”
• Require a RP
• Called Star comma Gee, because typing “show ip mroute” … this is what shows up

(S, G)

• AKA, a source tree
• These do not require a RP
• Pronounced as “Ess comma Gee”

Source Tree

• AKA, SPT
• Shortest path tree
• SPT is best tree

RPT — Rendezvous Point Tree

• *,G that points towards the RP

ASM — Any Source Multicast

• The host only knows the group it wants to receive (239.10.10.10)

SSM — Source Specific Multicast

• The host already knows the source, and group address (10.0.0.1, 232.10.10.10)

Upstream

• Towards the source

Downstream

• Towards group members

FHR — First hop router

• This router receives a multicast stream

LHR — Last Hop Router

• Receives IGMP messages from receivers, which are translated into PIM join messages

MRIB — Multicast Routing Information Base

• Shows RPTs, SPTs, RPFIs, OILs, and IILs

MFIB — Multicast Forwarding Information Base

• AKA MFIB
• Used the program the ASICS

RIB — Routing Information Base

DF — Designated Forwarder

• Used in PIDIR-PIM

RPF - reverse path forwarding

PIM is protocol independent, in the sense, that if a stream turns on, it must have a source, so it takes the form (0.0.0.1, 239.1.1.1), a (S,G).

If we do show ip route 10.0.0.1, we’ll see the interface the router intends to send any traffic towards that source address. This is the upstream interface.

As multicast traffic flows from 10.0.0.1, it should flow into the upstream interface, and out of any downstream interfaces the OIL.

Tracing the traffic back to the source this way is called reverse path forwarding and the interface along this path is the RPF.

The PIM neighbor on the RPF is called the RPF neighbor.

Any multi-cast traffic from any given source, not received on the RPF is discarded. This prevents loops.

Shared trees

(*,G) entries in the mroute table require fewer resources, since multiple sources can use the same tree.

(*,G) entries in the mroute table represent a security risk, because any source can send to this shared tree.

PIM

PIM forms adjacencies in only one direction.

The multicast source is the root of the tree. Packets flow downstream from the source. Control plane traffic like PIM joins flow upstream to the RP, or to the receiver.

PIM Modes

PIM message types

Shared-Tree (*,G)

• Shared trees are essential for multiple senders to the same group

• A single tree is built for each group, regardless of source

  • 3 sources, 1 tree

• Selects a router as the root of the tree

• If a receiver is on the same subnet as the sending host, it will need to revert to PIM Dense for that segment

• This isn’t always better. Shared trees will typically take suboptimal paths through a network

• Source trees are better distributed, hence they are more robust

• RP Selection is a hassle

Source based multicast (S,G)

• PIM dense uses a separate tree for each multicast source and destination group
• Groups do not share trees
  • 3 Sources 3 trees

Commands

show mrib route
show ip mroute
!
! PIM
!
show pim rpf hash
show pim range-list
show pim topology
!
! What interface should I receive this host traffic from?
!
show ip rpf 10.0.0.0
show ip mfib
!
! See if multicast even works
!
show ip pim stats
!
! PIM traffic
!
show ip pim interface detail
!
! DF election
!
show ip pim interface df

FLAGS
 A - Accepting. This interface is accepting data
 F - Forwarding. Where to send multicast traffic

Nexus 7K

show forwarding multicast route group <>

Lab stuff

BPF - Capture all PIM, but not PIM hello messages.

ip proto 103 and not ether[34] == 0x20

References

IP Multicast - Cisco

PIM Dense

Based on RFC 3973 Protocol Independent Multicast Dense Mode (PIM-DM)

• Push Model

  • Good for when every subnet probably wants this traffic

• No PIM DR

  • All FHR forward multicast traffic
    • Multicast traffic is flooded out every interface that isn’t the RPF.

• Eventually builds a SPT after prunes

• IGMP joins turn into graft messages

• Prunes last 3 minutes

  • Flood and Prune
  • Routers with no Receivers or duplicate S,G traffic prune.
  • 224.0.0.13 to find neighbors
  • Receivers prune back
  • Router attached to LAN listens for multicast control plane.
    • Receives source traffic
      • Insert (*,G) and (S,G) into mrib
      • Incoming traffic is attached to IIL
      • OIL is all other interfaces
      • Flood to OIL
      • PIM dense always uses SPT.

• Prune occurs

  • Traffic flows stop, but (S,G) remains in table
  • Multicast fails RPF
  • No downstream neighbor or reciever
  • Downstream sent prune
  • LAN Prune override exception

• After pruning

  • Flood again, prune back, flood again, prune back

PIM Sparse

Based on RFC4601 - Protocol Independent Multicast Sparse Mode (PIM-SM)

• Explicit joins everywhere. No flooding.
• LHR, sends a PIM-Join towards the RP, building a (*,G).
• Phased *
  1. The RPT tree
  • Receivers sending their (*,G) messages towards the RP.
  • FHR encapsulates the multicast traffic directly towards the RP.
  • PIM-Register
  • RP de-encapsulates the traffic, sending it down the RPT.
  • 2. Register Stop
    • The RP sends a (S,G) towards the source.
    • When multicast packets start showing up, without encapsulation, the RP sends a Register-Stop.
  • 3. SPT tree
    • LHR requests a (S,G) entry towards it’s upstream, until it’s joined to the (S,G) tree.
    • When the LHR starts getting two copies of the traffic, it sends a (S,G,rpt) prune message, towards the RP. (A prune specific to the RPT)
• If two LHRs exist, and duplicate traffic is detected a PIM elections happens.
  • These Asserts are every 3 minutes.
  • RPTbit, 0 is preferred and means “has (S,G) tree”
    • Metric Preference (Administrative Distance)
      • Metric
        • IP address of subnet interface.
• Specify the tunnel, for the pim-register messages on Cisco via ip pim register-source loopback 0
• The tunnel interface encapsulates the entire multicast packet, which adds 28 bytes of overhead. Packets close to the MTU will be silently dropped on IOS-XE.

a DR is elected by highest priority, or highest IP in the subnet.

• DR sends the PIM join upstream.

The RP always gets the stream, even if it has no receivers to forward it to.

Captures

PIM-SM-register-register-stop-prune.pcap

BIDIR-PIM

• Superset of PIM-SM
• No (S,G) entries
• Traffic can flow up and down the same tree.
• Still needs RPs
  • RP must be dedicated to BIDIR-PIM.
• Each bidirectional link has a DF election.
  • Ingress packets on any PIM interface can be forwarded downstream onto DF links.
    • No DF links, no forwarding.
  • Ingress packets to a DF can be forwarded upstream via the RPF towards the RPA.

References

RFC 5015: Bidirectional Protocol Independent Multicast (BIDIR-PIM) | RFC Editor

Auto-RP

Cisco devices can announce their willingness to be an RP, via cisco-rp-announce

A different service, a mapping agent, will read these messages, pick a winner, then advertise that out via cisco-rp-discovery

• 5.5.5.5, Candidate RP.
• 4.4.4.4, mapping agent.

R4# show ip pim autorp

AutoRP Information:

  AutoRP is enabled.
  RP Discovery packet MTU is 1500.
  224.0.1.40 is joined on Loopback0.
  AutoRP groups over sparse mode interface is enabled

PIM AutoRP Statistics: Sent/Received
  RP Announce: 0/16, RP Discovery: 64/42

These packets are slow.

R4# debug ip pim auto-rp

PIM Auto-RP debugging is on
!
! Sent to cisco-rp-discovery
!
*Apr 25 19:57:08.940: Auto-RP(0): Build RP-Discovery packet
*Apr 25 19:57:08.941: Auto-RP(0):  Build mapping (224.0.0.0/4, RP:5.5.5.5), PIMv2 v1,
*Apr 25 19:57:08.942: Auto-RP(0): Send RP-discovery packet of length 48 on GigabitEthernet0/3 (1 RP entries)
*Apr 25 19:57:08.943: Auto-RP(0): Send RP-discovery packet of length 48 on GigabitEthernet0/4 (1 RP entries)
*Apr 25 19:57:08.945: Auto-RP(0): Send RP-discovery packet of length 48 on GigabitEthernet0/0 (1 RP entries)
*Apr 25 19:57:08.948: Auto-RP(0): Send RP-discovery packet of length 48 on Loopback0(*) (1 RP entries)
*Apr 25 19:57:12.008: Auto-RP(0): Received RP-discovery packet of length 48, from 10.0.45.5, ignored
!
! Received by cisco-rp-announce
!
*Apr 25 19:58:30.159: Auto-RP(0): Received RP-announce packet of length 48, from 5.5.5.5, RP_cnt 1, ht 181
*Apr 25 19:58:30.159: (0): pim_add_prm:: 224.0.0.0/240.0.0.0, rp=5.5.5.5, repl = 0, ver =3, is_neg =0, bidir = 0, crp = 0
*Apr 25 19:58:30.160: Auto-RP(0): Update
*Apr 25 19:58:30.160:  prm_rp->bidir_mode = 0 vs bidir = 0 (224.0.0.0/4, RP:5.5.5.5), PIMv2 v1
R4# undebug all
All possible debugging has been turned off

References

Cisco - Configuring a Rendezvous Point

MSDP

• Link multicast domains via RPs
• Provide Redundant RPs via Anycast RP

A source turns on, and when the first RP finds out about it, it notifies other RPs.

Details

• RPs register to each other, in different multicast domains
• RP sends a SA (source active) message
• Still needs PIM running for the S,G
• TCP port 639
• Has keepalives

Startup

1. Multicast source starts up.

2. FHR router sends a register message to the closest RP.

3. RP registers this as a SA (Source Active).

4. Sends Source Active Messages to other RPs

(*,G) means there is an interested receiver.

Config

!
! RP1
!
int lo0
description "Only used for Anycast RP"
ip address 10.0.0.1 255.255.255.255
!
int lo1
ip add 10.1.1.1 255.255.255.255
!
ip msdp peer 10.1.1.2 connect-source loopback 1
ip msdp originator-id loopback 1
ip pim rp-address 10.0.0.1

!
! RP2
!
int lo 0
  description "Only used for Anycast RP"
  ip address 10.0.0.1 255.255.255.255
!
int lo1
  ip address 10.1.1.2 255.255.255.255
!
ip msdp peer 10.1.1.1 connect-source lo1
ip msdp originator-id loopback 1
ip pim rp-address 10.0.0.1

Commands

show ip msdp peer
show ip msdp sa-cache

References

RFC 4611 - Multicast Source Discovery Protocol (MSDP) Deployment Scenarios

Anycast RP - Cisco Whitepaper

IPv6 Multicast Addresses

Terms

NVO — Network Virtualization Overlay

• Stuff like VXLAN

BUM – Broadcast, Unknown Unicast, Multicast

• Traffic that is one-to-many

Interface local scope

• Never goes onto the wire
• Popular for neighbor discovery

Link local scope

• Never routed

References

IPv6 Multicast Address Space - IANA

Multicast L2 Addressing

An L2 multicast address is 6 bytes, or 48 bits.

The address is built by combining these three things.

Multicast Prefix + 0 + low-order-23-bits-from-v4

Multicast prefix

The Multicast Prefix is 01:00:5E 3 bytes or 24 bits.

25th bit

A binary zero 0

Last 23 bits

Using 232.10.10.10

Copy the low order 23 bits directly from the v4 address.

232.10.10.10/8

1110 1000 . 0000 1010 . 0000 1010 . 0000 1010
             \______________________________/

000 1010 0000 1010 0000 1010

Building the address

Using 232.10.10.10

          1 :         0 :        5E :        0A :       0A  :        0A
  0000 0001 . 0000 0000 . 0101 1110 . 0000 1010 . 0000 1010 . 0000 1010
  \__________________________________/|\______________________________/
         Multicast Prefix             |   The 23 bits from the v4 IP
           Always 01:00:5E            |
                                      └─ The required zero bit for multicast

becomes 01:00:5E:0A:0A:0A

Multicast overlap

To express a v4 multicast address uniquely in L2 we need 28 bits, since the first 4 bits of every v4 multicast address are 1110

L2 in multicast only gets 23 bits, which means 5 bits of overlap.

2^5 is 32, so there are 32 v4 addresses that share the same L2 address.

All 32 IPv4 addresses mapping to 01:00:5E:0A:0A:0A

══════════════════════════════════════════════════════════════════════════════
Address           Octet 1    Octet 2    Octet 3    Octet 4
──────────────────────────────────────────────────────────────────────────────
224. 10.10.10     1110 0000  0000 1010  0000 1010  0000 1010
224.138.10.10     1110 0000  1000 1010  0000 1010  0000 1010
225. 10.10.10     1110 0001  0000 1010  0000 1010  0000 1010
225.138.10.10     1110 0001  1000 1010  0000 1010  0000 1010
226 .10.10.10     1110 0010  0000 1010  0000 1010  0000 1010
226.138.10.10     1110 0010  1000 1010  0000 1010  0000 1010
227 .10.10.10     1110 0011  0000 1010  0000 1010  0000 1010
227.138.10.10     1110 0011  1000 1010  0000 1010  0000 1010
228 .10.10.10     1110 0100  0000 1010  0000 1010  0000 1010
228.138.10.10     1110 0100  1000 1010  0000 1010  0000 1010
229 .10.10.10     1110 0101  0000 1010  0000 1010  0000 1010
229.138.10.10     1110 0101  1000 1010  0000 1010  0000 1010
230 .10.10.10     1110 0110  0000 1010  0000 1010  0000 1010
230.138.10.10     1110 0110  1000 1010  0000 1010  0000 1010
231 .10.10.10     1110 0111  0000 1010  0000 1010  0000 1010
231.138.10.10     1110 0111  1000 1010  0000 1010  0000 1010
232 .10.10.10     1110 1000  0000 1010  0000 1010  0000 1010  < --- This is our SSM address
232.138.10.10     1110 1000  1000 1010  0000 1010  0000 1010
233 .10.10.10     1110 1001  0000 1010  0000 1010  0000 1010  < --- An address in the GLOP block
233.138.10.10     1110 1001  1000 1010  0000 1010  0000 1010
234 .10.10.10     1110 1010  0000 1010  0000 1010  0000 1010
234.138.10.10     1110 1010  1000 1010  0000 1010  0000 1010
235 .10.10.10     1110 1011  0000 1010  0000 1010  0000 1010
235.138.10.10     1110 1011  1000 1010  0000 1010  0000 1010
236 .10.10.10     1110 1100  0000 1010  0000 1010  0000 1010
236.138.10.10     1110 1100  1000 1010  0000 1010  0000 1010
237 .10.10.10     1110 1101  0000 1010  0000 1010  0000 1010
237.138.10.10     1110 1101  1000 1010  0000 1010  0000 1010
238 .10.10.10     1110 1110  0000 1010  0000 1010  0000 1010
238.138.10.10     1110 1110  1000 1010  0000 1010  0000 1010
239 .10.10.10     1110 1111  0000 1010  0000 1010  0000 1010
239.138.10.10     1110 1111  1000 1010  0000 1010  0000 1010  < --- an Organizational scope address
══════════════════════════════════════════════════════════════════════════════
                       ^^^^  ^
                       ||||  |
                       └└└└──└─ I incremented these five bits to show the pattern.

References

RFC 1112: Host extensions for IP multicasting | RFC Editor

Multicast L3 Addressing

Bits

• IPv4 L3 multicast address always start with 1110 or 224.0.0.0/4

• IPv6 L3 multicast address always start with 1111 1111 or ff00::/8

L3 Ranges

IPv4

IPv4 combines scope and purpose in its address ranges.

IANA Assignments

IPv6

IPv6 separates scope cleanly by last nibble in the first hextet.

L3 addresses Combines IPv4 and IPv6

L3

IL — Interface-Local

• Node only
• Not transmitted
  • Never on the wire

LL — Link-Local

• TTL of 1
• Transmitted
  • On the wire

SL — Site-Local

• Routeable
• Not Global Internet

L2 and L3

• IPv4 is always 01:00:5e
• IPv6 is always 33:33:00

IGMPv3

IGMP — Internet Group Membership Protocol

Hosts show interest in IPv4 multicast groups with IGMPv3.

Multiple group records can fit into a single report.

 0                   1                   2                   3    
 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1  
┌───────────────┬───────────────┬───────────────────────────────┐ 
│  Type = 0x22  │    Reserved   │           Checksum            │ 
├───────────────┴───────────────┼───────────────────────────────┤ 
│             Flags             │  Number of Group Records (M)  │ 
├───────────────────────────────┴───────────────────────────────┤ 
│                                                               │ 
│                                                               │ 
│                        Group Record [1]                       │ 
│                                                               │ 
│                                                               │ 
├──────────────────────────────────────────────────────────────-│ 
│                                                               │ 
│                                                               │ 
│                        Group Record [2]                       │ 
│                                                               │ 
│                                                               │ 
└───────────────────────────────────────────────────────────────┘ 

References

RFC 9776: Internet Group Management Protocol, Version 3 | RFC Editor

Static Routing

Static routes cannot respond to things like bandwidth, delay, load, or reliability.

These routes are a choice where there is no alternative path.

Config

Normal Route

ip route 0.0.0.0 0.0.0.0 192.168.0.1

Floating Static

Maybe the network already has a default route from an IGP like OSPFv2.

Since the OSPFv2 route is already present this won’t be installed.

We call that floating.

!
! 225 is a higher administrative distance than EIGRP, OSPF, ISIS, iBGP, and eBGP.
!
ip route 0.0.0.0 0.0.0.0 192.168.0.1 225

References

Static Routing - IP Routing Configuration Guide, IOS-XE 17.x - Cisco

OSPFv2

OSPFv2 is protocol 89.

Terms

LS — Link State

LSA — Link State Advertisement

LSDB — Link-State Database

RID — Router ID

• Unique 32-bit number to identify the router in a graph
• Doesn’t have to be an on-the-box IP, but is usually a loopback
• Should be easy to identify from a Router-ID subnet

Process ID

• Where the databases lives
• Not transmitted
• Allows multiple OSPFv2 processes

DR — Designated Router

• Network vertex for a broadcast or NBMA network
• Used to reduce the number of FULL adjacencies

Advertising Router

• Router that created the LSA
• Value in this field is the RID

The Update Rule

• Routers can only modify LSAs they originate
• Its RID is inside the “Advertising Router” field

LS Sequence

• The first sequence number in any LSA is 0x80000001
• Higher sequence numbers are newer LSAs

LS Checksum

• Used to ensure the LSA was transmitted without corruption
• Everything is checked except LS Age

LS Age

• LSAs time out in an hour
• Refreshed every 30 minutes
• LSA Age increments traveling through routers

IFF — If and only if

Packet types

Hello packets

These things must match for an adjacency to form

• Hello time
• Dead time
• Subnet
• Subnet mask
• Interface MTU
• Area
• Area flags (NSSA, Stub)
• Is DR/BDR enabled
• Authentication

These must not match

• Router ID

Check with debug ip ospf event

Identical databases

Each router can perform its own SPT via Dijkstra’s algorithm.

LSAs are flooded throughout an area, all routers in the same area should have the same LSAs and same database.

R1# show ip ospf database database-summary  | s Area 0
Area 0 database summary
  LSA Type      Count    Delete   Maxage
  Router        5        0        0       
  Network       5        0        0       
  Summary Net   8        0        0       
  Summary ASBR  2        0        0       
  Type-7 Ext    0        0        0       
    Prefixes redistributed in Type-7  0
  Opaque Link   0        0        0       
  Opaque Area   0        0        0       
  Subtotal      20       0        0

R2# show ip ospf database database-summary | s Area 0
Area 0 database summary
  LSA Type      Count    Delete   Maxage
  Router        5        0        0       
  Network       5        0        0       
  Summary Net   8        0        0       
  Summary ASBR  2        0        0       
  Type-7 Ext    0        0        0       
    Prefixes redistributed in Type-7  0
  Opaque Link   0        0        0       
  Opaque Area   0        0        0       
  Subtotal      20       0        0

Can also check with checksums

show ip ospf | i Checksum

Adjacency state machine

Routing hierarchy

OSPFv2 has four levels of routing hierarchy.

The bit E is what makes E1 and E2 routes. The bit being set is an E2 route, which is considered less preferred.

Default route

OSPFv2 has two ways of originating a default route.

default-information originate if a default route is present.

default-information originate always do it anyway.

Cost

Should be set manually on each node.

The defaults make 100Mbps and above all the same cost.

auto-cost reference-bandwidth 40,000

Area summary

These will show up as an O IA route in OSPFv2, and a route-to-null on the ABR.

• Requires a route present in the RIB

v4

router ospf 1
 router-id 2.2.2.2
 area 1 range 10.0.0.0 255.255.224.0

v6

router ospfv3 1
 !
 address-family ipv6 unicast
  area 1 range 2001:DB8::/56

Route-Filtering

You can use the same command to tell the router to … exclude these routes from the backbone, via the not-advertise keyword.

Using range

The area command is now a route-filter.

Range v4

router ospf 1
 router-id 2.2.2.2
 area 1 range 10.0.0.0 255.255.224.0 not-advertise

Range v6

router ospfv3 1
 !
 address-family ipv6 unicast
  area 1 range 2001:DB8::/56 not-advertise
 exit-address-family

Using Filter-Lists

These are a bit harder to use, in and out are inbound and outbound to the area.

For this topology

             Area 0                               Area 1            
                                                               
                                 |           10.0.10.0/24           
                                 |         2001:db8:0:10/64         
                                 |                            +----+
                              +----+       +------------------+ R3 |
+----+                        |    +-------+                  +----+
| R1 +------------------------+ R2 |                        
+----+                        |    +------+     
             10.0.0.0/24      +----+      |                   +----+
           2001:db8:0:0/64       |        +-------------------+ R4 |
                                 |           10.0.20.0/24     +----+
                                 |         2001:db8:0:20/64         

v4

ip prefix-list PREFIX_LIST_LOOPBACK_v4 seq 10 deny 1.1.1.1/32
ip prefix-list PREFIX_LIST_LOOPBACK_v4 seq 20 deny 2.2.2.2/32
ip prefix-list PREFIX_LIST_LOOPBACK_v4 seq 30 deny 3.3.3.3/32
!
router ospf 1
 area 0 filter-list prefix PREFIX_LIST_LOOPBACK_v4 in
 area 1 filter-list prefix PREFIX_LIST_LOOPBACK_v4 in

v6

!
ipv6 prefix-list PREFIX_LIST_v6 seq 10 deny FD::1/128
ipv6 prefix-list PREFIX_LIST_v6 seq 20 deny FD::3/128
ipv6 prefix-list PREFIX_LIST_v6 seq 30 deny FD::4/128
!
router ospfv3 1
 !
 address-family ipv6 unicast
  area 0 filter-list prefix PREFIX_LIST_v6 in
  area 1 filter-list prefix PREFIX_LIST_v6 in

OSPF graceful restart

• Restart OSPF without dropping adjacencies
• Keeps the OSPF table in memory while OSPF rebuilds.

Area types

I made a chart.

References

RFC 2328 - OSPF Version 2

J. T. Moy, OSPF: Anatomy of an Internet Routing Protocol. Reading, MA: Addison-Wesley, 1998.

OSPFv2 DR Election

DR and BDR

OSPFv2 uses explicit acknowledgments (re-sending the LSAs), so as neighbors and adjacencies grow, the amount of OSPFv2 traffic on a network increases.

A network with six OSPFv2 routers forming a full-mesh requires 15 adjacencies.

To mitigate the scaling problem, on broadcast segments OSPFv2 elects a DR, and BDR, to maintain the LSDB.

The RFC calls this a “network vertex”. We can also use the term DR.

• All routers listen for hello on 224.0.0.5
• DR floods LSAs to the routers with 224.0.0.5
• DROTHER talks to the DR/BDR on 224.0.0.6

In the diagram (from the RFC), everything connects to N2, so problem solved.

                                    **FROM**
                +---+      +---+
                |RT3|      |RT4|              |RT3|RT4|RT5|RT6|N2 |
                +---+      +---+        *  ------------------------
                  |    N2    |          *  RT3|   |   |   |   | X |
            +----------------------+    T  RT4|   |   |   |   | X |
                  |          |          O  RT5|   |   |   |   | X |
                +---+      +---+        *  RT6|   |   |   |   | X |
                |RT5|      |RT6|        *   N2| X | X | X | X |   |
                +---+      +---+

                          Broadcast or NBMA networks

DR

Forms full adjacencies.

R1# show ip ospf neighbor 

Neighbor ID     Pri   State           Dead Time   Address         Interface
2.2.2.2          50   FULL/BDR        00:00:31    10.0.0.2        Ethernet0/0
3.3.3.3           1   FULL/DROTHER    00:00:37    10.0.0.3        Ethernet0/0
4.4.4.4           1   FULL/DROTHER    00:00:34    10.0.0.4        Ethernet0/0
5.5.5.5           1   FULL/DROTHER    00:00:32    10.0.0.5        Ethernet0/0
6.6.6.6           1   FULL/DROTHER    00:00:31    10.0.0.6        Ethernet0/0

• First router online on the segment is the DR.

Drother

• Only forms full adjacencies with the DR, and BDR.
• When it sends LSAs, sends them to the DR/BDR via 224.0.0.6.

R1# show ip ospf neighbor 

Neighbor ID     Pri   State           Dead Time   Address         Interface
2.2.2.2          50   FULL/BDR        00:00:31    10.0.0.2        Ethernet0/0
3.3.3.3           1   FULL/DROTHER    00:00:37    10.0.0.3        Ethernet0/0
4.4.4.4           1   FULL/DROTHER    00:00:34    10.0.0.4        Ethernet0/0
5.5.5.5           1   FULL/DROTHER    00:00:32    10.0.0.5        Ethernet0/0
6.6.6.6           1   FULL/DROTHER    00:00:31    10.0.0.6        Ethernet0/0

OSPFv2 DR LSAs

Type 1 - router

DR

R1# show ip ospf database router 1.1.1.1

            OSPF Router with ID (1.1.1.1) (Process ID 1)

                Router Link States (Area 0)

  LS age: 32
  Options: (No TOS-capability, DC)
  LS Type: Router Links
  Link State ID: 1.1.1.1
  Advertising Router: 1.1.1.1
  LS Seq Number: 8000007B
  Checksum: 0x1A77
  Length: 36
  Number of Links: 1

    Link connected to: a Transit Network
     (Link ID) Designated Router address: 10.0.0.1
     (Link Data) Router Interface address: 10.0.0.1
      Number of MTID metrics: 0
       TOS 0 Metrics: 10

DROther

R4#show ip ospf database router 4.4.4.4

            OSPF Router with ID (4.4.4.4) (Process ID 1)

                Router Link States (Area 0)

  LS age: 135
  Options: (No TOS-capability, DC)
  LS Type: Router Links
  Link State ID: 4.4.4.4
  Advertising Router: 4.4.4.4
  LS Seq Number: 8000007C
  Checksum: 0x5D18
  Length: 36
  Number of Links: 1

    Link connected to: a Transit Network
     (Link ID) Designated Router address: 10.0.0.1
     (Link Data) Router Interface address: 10.0.0.4
      Number of MTID metrics: 0
       TOS 0 Metrics: 10

Type 2 - network

R4# show ip ospf database network 

            OSPF Router with ID (4.4.4.4) (Process ID 1)

                Net Link States (Area 0)

  LS age: 183
  Options: (No TOS-capability, DC)
  LS Type: Network Links
  Link State ID: 10.0.0.1 (address of Designated Router)
  Advertising Router: 1.1.1.1
  LS Seq Number: 80000002
  Checksum: 0x4481
  Length: 48
  Network Mask: /24
        Attached Router: 1.1.1.1
        Attached Router: 2.2.2.2
        Attached Router: 3.3.3.3
        Attached Router: 4.4.4.4
        Attached Router: 5.5.5.5
        Attached Router: 6.6.6.6

OSPFv2 LSA Types

LSA type to route type

Format

The Router ID is what is used to build the SPT. It’s very important it’s both

• Correct
• Easy to identify the router

  +-------------------------+ Three fields to differentiate LSAs
  |         LS Age          |     - LS Type
  +-------------------------+     - Link State ID
  |  Options      LS Type   |     - Advertising Router
  +-------------------------+
  |     Link State ID       |  < -- Unique number from the Advertising Router for Each LSA
  +-------------------------+
  |   Advertising Router    |  < -- Router ID
  +-------------------------+
  |    LS Sequence Number   |  < -- How old the LSA is. LSAs with higher numbers are updates to older LSAs
  +-------------------------+
  |      LS Checksum        |
  +-------------------------+
  |        Length           |
  +-------------------------+

OSPFv2 Type 1 LSA - Router

This LSA describes the networks attached to a router.

Topology

                                                 ASBR                                    
                                                  OSPF                                   
                     A23  Regular               ┌────┐     ┌─────┐  EIGRP AS 33  ┌─────┐ 
               ┌────────────────────────────────┤ R3 ├─────┤ R33 ├───────────────┤ R34 │ 
               │                                └────┘     └─────┘ 10.33.34.0/24 └─────┘ 
               │     A24  Stub                  ┌────┐     ┌─────┐                       
               │ ┌──────────────────────────────┤ R4 ├─────┤ R44 │                       
               │ │                              └────┘     └─────┘                       
┌────┐ Area 0 ┌┴─┴─┐ A25  NSSA                  ┌────┐     ┌─────┐  EIGRP AS 55  ┌─────┐ 
│ R1 ├────────┤ R2 ├────────────────────────────┤ R5 ├─────┤ R55 ├───────────────┤ R56 │ 
└────┘        │ ABR│                            └────┘     └─────┘ 10.55.56.0/24 └─────┘ 
              └┬─┬─┘ A26  Totally Stubby        ┌────┐     ┌─────┐                       
               │ └──────────────────────────────┤ R6 ├─────┤ R66 │                       
               │                                └────┘     └─────┘                       
               │     A27  NSSA Totally Stubby   ┌────┐     ┌─────┐  EIGRP AS 77  ┌─────┐ 
               └────────────────────────────────┤ R7 ├─────┤ R77 ├───────────────┤ R78 │ 
                                                └────┘     └─────┘ 10.77.78.0/24 └─────┘ 

Example

R1# show ip ospf database router 

            OSPF Router with ID (10.0.0.1) (Process ID 1)

                Router Link States (Area 0)

  LS age: 325
  Options: (No TOS-capability, DC)
  LS Type: Router Links
  Link State ID: 10.0.0.1
  Advertising Router: 10.0.0.1
  LS Seq Number: 80000012
  Checksum: 0xDC6
  Length: 48
  Number of Links: 2

    Link connected to: a Stub Network
     (Link ID) Network/subnet number: 10.0.0.1
     (Link Data) Network Mask: 255.255.255.255
      Number of MTID metrics: 0
       TOS 0 Metrics: 1

    Link connected to: a Transit Network
     (Link ID) Designated Router address: 10.1.2.1
     (Link Data) Router Interface address: 10.1.2.1
      Number of MTID metrics: 0
       TOS 0 Metrics: 1


  LS age: 134
  Options: (No TOS-capability, DC)
  LS Type: Router Links
  Link State ID: 10.0.0.2
  Advertising Router: 10.0.0.2
  LS Seq Number: 80000014
  Checksum: 0x26A4
  Length: 48
  Area Border Router
  AS Boundary Router
  Number of Links: 2

    Link connected to: a Stub Network
     (Link ID) Network/subnet number: 10.0.0.2
     (Link Data) Network Mask: 255.255.255.255
      Number of MTID metrics: 0
       TOS 0 Metrics: 1

    Link connected to: a Transit Network
     (Link ID) Designated Router address: 10.1.2.1
     (Link Data) Router Interface address: 10.1.2.2
      Number of MTID metrics: 0
       TOS 0 Metrics: 1

OSPFv2 Type 2 LSA - Network

This LSA describes a network vertex, a DR for a Broadcast Segment.

Topology

                                                 ASBR                                    
                                                  OSPF                                   
                     A23  Regular               ┌────┐     ┌─────┐  EIGRP AS 33  ┌─────┐ 
               ┌────────────────────────────────┤ R3 ├─────┤ R33 ├───────────────┤ R34 │ 
               │                                └────┘     └─────┘ 10.33.34.0/24 └─────┘ 
               │     A24  Stub                  ┌────┐     ┌─────┐                       
               │ ┌──────────────────────────────┤ R4 ├─────┤ R44 │                       
               │ │                              └────┘     └─────┘                       
┌────┐ Area 0 ┌┴─┴─┐ A25  NSSA                  ┌────┐     ┌─────┐  EIGRP AS 55  ┌─────┐ 
│ R1 ├────────┤ R2 ├────────────────────────────┤ R5 ├─────┤ R55 ├───────────────┤ R56 │ 
└────┘        │ ABR│                            └────┘     └─────┘ 10.55.56.0/24 └─────┘ 
              └┬─┬─┘ A26  Totally Stubby        ┌────┐     ┌─────┐                       
               │ └──────────────────────────────┤ R6 ├─────┤ R66 │                       
               │                                └────┘     └─────┘                       
               │     A27  NSSA Totally Stubby   ┌────┐     ┌─────┐  EIGRP AS 77  ┌─────┐ 
               └────────────────────────────────┤ R7 ├─────┤ R77 ├───────────────┤ R78 │ 
                                                └────┘     └─────┘ 10.77.78.0/24 └─────┘ 

Example

R1# show ip ospf database network 

            OSPF Router with ID (10.0.0.1) (Process ID 1)

                Net Link States (Area 0)

  LS age: 508
  Options: (No TOS-capability, DC)
  LS Type: Network Links
  Link State ID: 10.1.2.1 (address of Designated Router)
  Advertising Router: 10.0.0.1
  LS Seq Number: 80000008
  Checksum: 0x5DA5
  Length: 32
  Network Mask: /24
        Attached Router: 10.0.0.1
        Attached Router: 10.0.0.2

OSPFv2 Type 3 LSA - Summary

These describe networks, reachable via an ABR.

Topology

                                                 ASBR                                    
                                                  OSPF                                   
                     A23  Regular               ┌────┐     ┌─────┐  EIGRP AS 33  ┌─────┐ 
               ┌────────────────────────────────┤ R3 ├─────┤ R33 ├───────────────┤ R34 │ 
               │                                └────┘     └─────┘ 10.33.34.0/24 └─────┘ 
               │     A24  Stub                  ┌────┐     ┌─────┐                       
               │ ┌──────────────────────────────┤ R4 ├─────┤ R44 │                       
               │ │                              └────┘     └─────┘                       
┌────┐ Area 0 ┌┴─┴─┐ A25  NSSA                  ┌────┐     ┌─────┐  EIGRP AS 55  ┌─────┐ 
│ R1 ├────────┤ R2 ├────────────────────────────┤ R5 ├─────┤ R55 ├───────────────┤ R56 │ 
└────┘        │ ABR│                            └────┘     └─────┘ 10.55.56.0/24 └─────┘ 
              └┬─┬─┘ A26  Totally Stubby        ┌────┐     ┌─────┐                       
               │ └──────────────────────────────┤ R6 ├─────┤ R66 │                       
               │                                └────┘     └─────┘                       
               │     A27  NSSA Totally Stubby   ┌────┐     ┌─────┐  EIGRP AS 77  ┌─────┐ 
               └────────────────────────────────┤ R7 ├─────┤ R77 ├───────────────┤ R78 │ 
                                                └────┘     └─────┘ 10.77.78.0/24 └─────┘ 

Example

R1# show ip ospf database summary 

            OSPF Router with ID (10.0.0.1) (Process ID 1)

                Summary Net Link States (Area 0)

  LS age: 1901
  Options: (No TOS-capability, DC, Upward)
  LS Type: Summary Links(Network)
  Link State ID: 10.0.0.3 (summary Network Number)
  Advertising Router: 10.0.0.2
  LS Seq Number: 80000009
  Checksum: 0x8693
  Length: 28
  Network Mask: /32
        MTID: 0         Metric: 2 

OSPFv2 Type 4 LSA - ASBR Summary

This LSA describes how to reach an ASBR, via an ABR.

Topology

                                                 ASBR                                    
                                                  OSPF                                   
                     A23  Regular               ┌────┐     ┌─────┐  EIGRP AS 33  ┌─────┐ 
               ┌────────────────────────────────┤ R3 ├─────┤ R33 ├───────────────┤ R34 │ 
               │                                └────┘     └─────┘ 10.33.34.0/24 └─────┘ 
               │     A24  Stub                  ┌────┐     ┌─────┐                       
               │ ┌──────────────────────────────┤ R4 ├─────┤ R44 │                       
               │ │                              └────┘     └─────┘                       
┌────┐ Area 0 ┌┴─┴─┐ A25  NSSA                  ┌────┐     ┌─────┐  EIGRP AS 55  ┌─────┐ 
│ R1 ├────────┤ R2 ├────────────────────────────┤ R5 ├─────┤ R55 ├───────────────┤ R56 │ 
└────┘        │ ABR│                            └────┘     └─────┘ 10.55.56.0/24 └─────┘ 
              └┬─┬─┘ A26  Totally Stubby        ┌────┐     ┌─────┐                       
               │ └──────────────────────────────┤ R6 ├─────┤ R66 │                       
               │                                └────┘     └─────┘                       
               │     A27  NSSA Totally Stubby   ┌────┐     ┌─────┐  EIGRP AS 77  ┌─────┐ 
               └────────────────────────────────┤ R7 ├─────┤ R77 ├───────────────┤ R78 │ 
                                                └────┘     └─────┘ 10.77.78.0/24 └─────┘ 

Example

R1# show ip ospf database asbr-summary 

            OSPF Router with ID (10.0.0.1) (Process ID 1)

                Summary ASB Link States (Area 0)

  LS age: 429
  Options: (No TOS-capability, DC, Upward)
  LS Type: Summary Links(AS Boundary Router)
  Link State ID: 10.0.0.33 (AS Boundary Router address)
  Advertising Router: 10.0.0.2
  LS Seq Number: 80000008
  Checksum: 0x4DAE
  Length: 28
  Network Mask: /0
        MTID: 0         Metric: 2 

OSPFv2 Type 5 LSA - External

This LSA describes external networks.

Topology

                                                 ASBR                                    
                                                  OSPF                                   
                     A23  Regular               ┌────┐     ┌─────┐  EIGRP AS 33  ┌─────┐ 
               ┌────────────────────────────────┤ R3 ├─────┤ R33 ├───────────────┤ R34 │ 
               │                                └────┘     └─────┘ 10.33.34.0/24 └─────┘ 
               │     A24  Stub                  ┌────┐     ┌─────┐                       
               │ ┌──────────────────────────────┤ R4 ├─────┤ R44 │                       
               │ │                              └────┘     └─────┘                       
┌────┐ Area 0 ┌┴─┴─┐ A25  NSSA                  ┌────┐     ┌─────┐  EIGRP AS 55  ┌─────┐ 
│ R1 ├────────┤ R2 ├────────────────────────────┤ R5 ├─────┤ R55 ├───────────────┤ R56 │ 
└────┘        │ ABR│                            └────┘     └─────┘ 10.55.56.0/24 └─────┘ 
              └┬─┬─┘ A26  Totally Stubby        ┌────┐     ┌─────┐                       
               │ └──────────────────────────────┤ R6 ├─────┤ R66 │                       
               │                                └────┘     └─────┘                       
               │     A27  NSSA Totally Stubby   ┌────┐     ┌─────┐  EIGRP AS 77  ┌─────┐ 
               └────────────────────────────────┤ R7 ├─────┤ R77 ├───────────────┤ R78 │ 
                                                └────┘     └─────┘ 10.77.78.0/24 └─────┘ 

Example

R1#show ip ospf database external 10.55.56.0

            OSPF Router with ID (10.0.0.1) (Process ID 1)

                Type-5 AS External Link States

  LS age: 353
  Options: (No TOS-capability, DC, Upward)
  LS Type: AS External Link
  Link State ID: 10.55.56.0 (External Network Number )
  Advertising Router: 10.0.0.2
  LS Seq Number: 80000008
  Checksum: 0x86FB
  Length: 36
  Network Mask: /24
        Metric Type: 2 (Larger than any link state path)
        MTID: 0 
        Metric: 100 
        Forward Address: 10.0.0.55
        External Route Tag: 0

OSPFv2 Network and LSA Chart

OSPFv2 LSA Default Routes

R2 is advertising a default route to these routers.

Each router is inside a different area type.

                    Regular              ┌────┐
              ┌──────────────────────────┤ R3 │ Type 5
              │                          └────┘
              │     Stub                 ┌────┐
              │ ┌────────────────────────┤ R4 │ Type 3
      Area 0  │ │                        └────┘
┌────┐       ┌┴─┴─┐ NSSA                 ┌────┐
│ R1 ├───────┤ R2 ├──────────────────────┤ R5 │ Type 7
└────┘       └┬─┬─┘                      └────┘
              │ │   Totally Stubby       ┌────┐
              │ └────────────────────────┤ R6 │ Type 3
              │                          └────┘
              │     NSSA Totally Stubby  ┌────┐
              └──────────────────────────┤ R7 │ Type 3
                                         └────┘

Type 3 default

R2# show ip ospf database summary 0.0.0.0 self-originate 

            OSPF Router with ID (2.2.2.2) (Process ID 1)

                Summary Net Link States (Area 24)

  LS age: 951
  Options: (No TOS-capability, DC, Upward)
  LS Type: Summary Links(Network)
  Link State ID: 0.0.0.0 (summary Network Number)
  Advertising Router: 2.2.2.2
  LS Seq Number: 80000002
  Checksum: 0x73C1
  Length: 28
  Network Mask: /0
        MTID: 0         Metric: 1 


                Summary Net Link States (Area 26)

  LS age: 951
  Options: (No TOS-capability, DC, Upward)
  LS Type: Summary Links(Network)
  Link State ID: 0.0.0.0 (summary Network Number)
  Advertising Router: 2.2.2.2
  LS Seq Number: 80000002
  Checksum: 0x73C1
  Length: 28
  Network Mask: /0
        MTID: 0         Metric: 1 


                Summary Net Link States (Area 27)

  LS age: 953
  Options: (No TOS-capability, DC, Upward)
  LS Type: Summary Links(Network)
  Link State ID: 0.0.0.0 (summary Network Number)
  Advertising Router: 2.2.2.2
  LS Seq Number: 80000002
  Checksum: 0xFA32
  Length: 28
  Network Mask: /0
        MTID: 0         Metric: 1

Type 5 default

R2# show ip ospf database external 0.0.0.0 self-originate 

            OSPF Router with ID (2.2.2.2) (Process ID 1)

                Type-5 AS External Link States

  LS age: 59
  Options: (No TOS-capability, DC, Upward)
  LS Type: AS External Link
  Link State ID: 0.0.0.0 (External Network Number )
  Advertising Router: 2.2.2.2
  LS Seq Number: 80000001
  Checksum: 0xFEAB
  Length: 36
  Network Mask: /0
        Metric Type: 2 (Larger than any link state path)
        MTID: 0 
        Metric: 1 
        Forward Address: 0.0.0.0
        External Route Tag: 1

Type 7 default

R2# show ip ospf database nssa-external 0.0.0.0 self-originate

            OSPF Router with ID (2.2.2.2) (Process ID 1)

                Type-7 AS External Link States (Area 25)

  LS age: 312
  Options: (No TOS-capability, No Type 7/5 translation, DC, Upward)
  LS Type: AS External Link
  Link State ID: 0.0.0.0 (External Network Number )
  Advertising Router: 2.2.2.2
  LS Seq Number: 80000001
  Checksum: 0xD0D8
  Length: 36
  Network Mask: /0
        Metric Type: 2 (Larger than any link state path)
        MTID: 0 
        Metric: 1 
        Forward Address: 0.0.0.0
        External Route Tag: 0

OSPFv2 Network Types

OSPF Representation of routers and networks

Moy                         Standards Track                    [Page 13]

RFC 2328                     OSPF Version 2                   April 1998

                                                  **FROM**

                                           *      |RT1|RT2|
                +---+Ia    +---+           *   ------------
                |RT1|------|RT2|           T   RT1|   | X |
                +---+    Ib+---+           O   RT2| X |   |
                                           *    Ia|   | X |
                                           *    Ib| X |   |

                     Physical point-to-point networks


                                                  **FROM**
                      +---+                *
                      |RT7|                *      |RT7| N3|
                      +---+                T   ------------
                        |                  O   RT7|   |   |
            +----------------------+       *    N3| X |   |
                       N3                  *

                              Stub networks

                                                  **FROM**
                +---+      +---+
                |RT3|      |RT4|              |RT3|RT4|RT5|RT6|N2 |
                +---+      +---+        *  ------------------------
                  |    N2    |          *  RT3|   |   |   |   | X |
            +----------------------+    T  RT4|   |   |   |   | X |
                  |          |          O  RT5|   |   |   |   | X |
                +---+      +---+        *  RT6|   |   |   |   | X |
                |RT5|      |RT6|        *   N2| X | X | X | X |   |
                +---+      +---+

                          Broadcast or NBMA networks

References

RFC 2328 - OSPF

OSPFv2 Point-to-Multipoint

p2mp — point-to-multipoint

• Usually Hub-and-Spoke topologies
• No DR election for this network type
• Install a /32

What is hub and spoke

This network type assumes that spoke sites cannot reach each other directly, but are in the same L3 subnet.

This network type also assumes hair-pins are OK (traffic must go in and out of the same interface on one node)

Since each router is in the same L3 subnet, but not directly L2 reachable, each router will advertise its own, in-subnet IP as a /32.

The use-case for this network type is frame-relay, older network types where traveling through the hub is required.

Multicast

If multicast works (even carried by mGRE) usepoint-to-multipoint.

If multicast does not work, use point-to-point non-broadcast and manually define the neighbors under OSPFv2.

An example problem

A network is set up on Ethernet.

… Maybe it looks like this.

       10.12.34.0/24      
                          
┌──────┐          ┌──────┐
│  R1  ├─┐      ┌─┤  R2  │
└──────┘ │      │ └──────┘
        ┌┴──────┴┐        
        │ Switch │        
        └┬──────┬┘        
┌──────┐ │      │ ┌──────┐
│  R3  ├─┘      └─┤  R4  │
└──────┘          └──────┘

This will not work.

IPs are

• 10.12.34.1
• 10.12.34.2
• 10.12.34.3
• 10.12.34.4

RIDs are

• 1.1.1.1
• 2.2.2.2
• 3.3.3.3
• 4.4.4.4

OSPFv2 Network Types, attempted.

• point-to-multipoint
• point-to-multipoint non-broadcast

R1 - doesn’t form a full mesh

4.4.4.4 is broken.

Technically it flaps, going up and down.

R1# show ip ospf neighbor 

Neighbor ID     Pri   State           Dead Time   Address         Interface
4.4.4.4           0   EXSTART/  -     00:01:59    10.12.34.4      Ethernet0/0
2.2.2.2           0   FULL/  -        00:01:59    10.12.34.2      Ethernet0/0
3.3.3.3           0   FULL/  -        00:01:45    10.12.34.3      Ethernet0/0

R1 OSPFv2 debugs show adjacency problems

• R1-R4 Hello messages seem to work fine
• DBD messages from R1 never get to R4.

debug ip ospf adjacency
debug ip ospf hello

R1# show debug
Packet Infra debugs:

Ip Address                                               Port
------------------------------------------------------|----------

OSPF:
  OSPF hello debugging is on
  OSPF adjacency debugging is on

R1#
*Jun  2 23:40:18.278: OSPF-1 HELLO Et0/0: Rcv hello from 4.4.4.4 area 0 10.12.34.4
*Jun  2 23:40:18.278: OSPF-1 ADJ   Et0/0: 2 Way Communication to 4.4.4.4, state 2WAY
*Jun  2 23:40:18.278: OSPF-1 ADJ   Et0/0: Nbr 4.4.4.4: Prepare dbase exchange
*Jun  2 23:40:18.278: OSPF-1 ADJ   Et0/0: Send DBD to 4.4.4.4 seq 0xB07 opt 0x52 flag 0x7 len 32
*Jun  2 23:40:18.397: OSPF-1 HELLO Et0/0: Rcv hello from 3.3.3.3 area 0 10.12.34.3
*Jun  2 23:40:22.784: OSPF-1 ADJ   Et0/0: Send DBD to 4.4.4.4 seq 0xB07 opt 0x52 flag 0x7 len 32
*Jun  2 23:40:22.784: OSPF-1 ADJ   Et0/0: Retransmitting DBD to 4.4.4.4 [1]
*Jun  2 23:40:25.728: OSPF-1 HELLO Et0/0: Send hello to 10.12.34.4 area 0 from 10.12.34.1
*Jun  2 23:40:25.728: OSPF-1 HELLO Et0/0: Send hello to 10.12.34.2 area 0 from 10.12.34.1
*Jun  2 23:40:25.728: OSPF-1 HELLO Et0/0: Send hello to 10.12.34.3 area 0 from 10.12.34.1
*Jun  2 23:40:27.481: OSPF-1 ADJ   Et0/0: Send DBD to 4.4.4.4 seq 0xB07 opt 0x52 flag 0x7 len 32
*Jun  2 23:40:27.481: OSPF-1 ADJ   Et0/0: Retransmitting DBD to 4.4.4.4 [2]
*Jun  2 23:40:28.018: OSPF-1 HELLO Et0/0: Rcv hello from 2.2.2.2 area 0 10.12.34.2
*Jun  2 23:40:32.481: OSPF-1 ADJ   Et0/0: Send DBD to 4.4.4.4 seq 0xB07 opt 0x52 flag 0x7 len 32
*Jun  2 23:40:32.481: OSPF-1 ADJ   Et0/0: Retransmitting DBD to 4.4.4.4 [3]
*Jun  2 23:40:37.083: OSPF-1 ADJ   Et0/0: Send DBD to 4.4.4.4 seq 0xB07 opt 0x52 flag 0x7 len 32
*Jun  2 23:40:37.083: OSPF-1 ADJ   Et0/0: Retransmitting DBD to 4.4.4.4 [4]
*Jun  2 23:40:41.603: OSPF-1 ADJ   Et0/0: Send DBD to 4.4.4.4 seq 0xB07 opt 0x52 flag 0x7 len 32
*Jun  2 23:40:41.603: OSPF-1 ADJ   Et0/0: Retransmitting DBD to 4.4.4.4 [5]
*Jun  2 23:40:45.534: OSPF-1 HELLO Et0/0: Rcv hello from 3.3.3.3 area 0 10.12.34.3
*Jun  2 23:40:46.486: OSPF-1 ADJ   Et0/0: Send DBD to 4.4.4.4 seq 0xB07 opt 0x52 flag 0x7 len 32
*Jun  2 23:40:46.486: OSPF-1 ADJ   Et0/0: Retransmitting DBD to 4.4.4.4 [6]
*Jun  2 23:40:47.070: OSPF-1 HELLO Et0/0: Rcv hello from 4.4.4.4 area 0 10.12.34.4
*Jun  2 23:40:51.106: OSPF-1 ADJ   Et0/0: Send DBD to 4.4.4.4 seq 0xB07 opt 0x52 flag 0x7 len 32
*Jun  2 23:40:51.106: OSPF-1 ADJ   Et0/0: Retransmitting DBD to 4.4.4.4 [7]
*Jun  2 23:40:53.805: OSPF-1 HELLO Et0/0: Send hello to 10.12.34.4 area 0 from 10.12.34.1
*Jun  2 23:40:53.805: OSPF-1 HELLO Et0/0: Send hello to 10.12.34.2 area 0 from 10.12.34.1
*Jun  2 23:40:53.805: OSPF-1 HELLO Et0/0: Send hello to 10.12.34.3 area 0 from 10.12.34.1
*Jun  2 23:40:55.769: OSPF-1 ADJ   Et0/0: Send DBD to 4.4.4.4 seq 0xB07 opt 0x52 flag 0x7 len 32
*Jun  2 23:40:55.769: OSPF-1 ADJ   Et0/0: Retransmitting DBD to 4.4.4.4 [8]
*Jun  2 23:40:55.909: OSPF-1 HELLO Et0/0: Rcv hello from 2.2.2.2 area 0 10.12.34.2

R1 - routing table is weird

R1 thinks the way to get to R4 is via R2.

Notice the other OSPFv2 routers are fine.

R1# show ip route

[output omitted]

      10.0.0.0/8 is variably subnetted, 5 subnets, 2 masks
C        10.12.34.0/24 is directly connected, Ethernet0/0
L        10.12.34.1/32 is directly connected, Ethernet0/0
O        10.12.34.2/32 [110/10] via 10.12.34.2, 00:35:13, Ethernet0/0
O        10.12.34.3/32 [110/10] via 10.12.34.3, 00:35:12, Ethernet0/0
O        10.12.34.4/32 [110/20] via 10.12.34.2, 00:34:19, Ethernet0/0

R1 ARP is correct.

L2 programming for a spoke site doesn’t matter for Hub-and-Spoke designs.

R1# show ip arp
Protocol  Address          Age (min)  Hardware Addr   Type   Interface
Internet  10.12.34.1              -   aabb.cc00.3d00  ARPA   Ethernet0/0
Internet  10.12.34.2            128   aabb.cc00.4d00  ARPA   Ethernet0/0
Internet  10.12.34.3            128   aabb.cc00.4e00  ARPA   Ethernet0/0
Internet  10.12.34.4            128   aabb.cc00.3e00  ARPA   Ethernet0/0

R1 - CEF is wrong

This is how we know something is very wrong, a paradigm has been broken.

CEF is using the next-hop of R2.

R1# show ip cef 10.12.34.4 detail 
10.12.34.4/32, epoch 0
  Adj source: IP adj out of Ethernet0/0, addr 10.12.34.4 7162A5807E90
    Dependent covered prefix type adjfib, cover 10.12.34.0/24
  nexthop 10.12.34.2 Ethernet0/0

Checking router LSAs

Self-Originated Router-LSA on R4

Point-to-Multipoint describes the network as a series of /32 links. What does that mean?

It means when a router goes FULL the first thing it does is advertise its own connected IP directly into OSPFv2 as a stub Network.

R4# show ip ospf database router self-originate 

            OSPF Router with ID (4.4.4.4) (Process ID 1)

                Router Link States (Area 0)

  LS age: 792
  Options: (No TOS-capability, DC)
  LS Type: Router Links
  Link State ID: 4.4.4.4
  Advertising Router: 4.4.4.4
  LS Seq Number: 80000016
  Checksum: 0x9FC5
  Length: 48
  Number of Links: 2

    Link connected to: another Router (point-to-point)
     (Link ID) Neighboring Router ID: 2.2.2.2
     (Link Data) Router Interface address: 10.12.34.4
      Number of MTID metrics: 0
       TOS 0 Metrics: 10

    Link connected to: a Stub Network
     (Link ID) Network/subnet number: 10.12.34.4
     (Link Data) Network Mask: 255.255.255.255
      Number of MTID metrics: 0
       TOS 0 Metrics: 0

R4 only has adjacency with R2.

R4# show ip ospf neighbor 

Neighbor ID     Pri   State           Dead Time   Address         Interface
N/A               0   DOWN/  -           -        10.12.34.3      Ethernet0/0
1.1.1.1           0   INIT/  -        00:01:59    10.12.34.1      Ethernet0/0
2.2.2.2           0   FULL/  -        00:01:53    10.12.34.2      Ethernet0/0

R2 has an adjacency with R1

R2# show ip ospf neighbor 

Neighbor ID     Pri   State           Dead Time   Address         Interface
1.1.1.1           0   FULL/  -        00:01:38    10.12.34.1      Ethernet0/0
3.3.3.3           0   EXSTART/  -     00:01:51    10.12.34.3      Ethernet0/0
4.4.4.4           0   FULL/  -        00:01:49    10.12.34.4      Ethernet0/0

Checking LSAs for the area.

Notes are embedded in the output.

R1# show ip ospf database router 

            OSPF Router with ID (1.1.1.1) (Process ID 1)

                Router Link States (Area 0)

  LS age: 1159
  Options: (No TOS-capability, DC)
  LS Type: Router Links
  Link State ID: 1.1.1.1
  Advertising Router: 1.1.1.1
  LS Seq Number: 80000010
  Checksum: 0xCB60
  Length: 60
  Number of Links: 3
!
! R1 is adjacent to R2
!
    Link connected to: another Router (point-to-point)
     (Link ID) Neighboring Router ID: 2.2.2.2
     (Link Data) Router Interface address: 10.12.34.1
      Number of MTID metrics: 0
       TOS 0 Metrics: 10

[output omitted]

    Link connected to: a Stub Network
     (Link ID) Network/subnet number: 10.12.34.1
     (Link Data) Network Mask: 255.255.255.255
      Number of MTID metrics: 0
       TOS 0 Metrics: 0


  LS age: 1155
  Options: (No TOS-capability, DC)
  LS Type: Router Links
  Link State ID: 2.2.2.2
  Advertising Router: 2.2.2.2
  LS Seq Number: 8000001A
  Checksum: 0xE531
  Length: 60
  Number of Links: 3

    Link connected to: another Router (point-to-point)
     (Link ID) Neighboring Router ID: 1.1.1.1
     (Link Data) Router Interface address: 10.12.34.2
      Number of MTID metrics: 0
       TOS 0 Metrics: 10
!
! R2 is adjacency to R4
!
    Link connected to: another Router (point-to-point)
     (Link ID) Neighboring Router ID: 4.4.4.4
     (Link Data) Router Interface address: 10.12.34.2
      Number of MTID metrics: 0
       TOS 0 Metrics: 10

    Link connected to: a Stub Network
     (Link ID) Network/subnet number: 10.12.34.2
     (Link Data) Network Mask: 255.255.255.255
      Number of MTID metrics: 0
       TOS 0 Metrics: 0

[output omitted]
!
! R4 advertises it's IP as a /32, via p2mp
!
  LS age: 1141
  Options: (No TOS-capability, DC)
  LS Type: Router Links
  Link State ID: 4.4.4.4
  Advertising Router: 4.4.4.4
  LS Seq Number: 80000016
  Checksum: 0x9FC5
  Length: 48
  Number of Links: 2

    Link connected to: another Router (point-to-point)
     (Link ID) Neighboring Router ID: 2.2.2.2
     (Link Data) Router Interface address: 10.12.34.4
      Number of MTID metrics: 0
       TOS 0 Metrics: 10

    Link connected to: a Stub Network
     (Link ID) Network/subnet number: 10.12.34.4 
     (Link Data) Network Mask: 255.255.255.255
      Number of MTID metrics: 0
       TOS 0 Metrics: 0

The capture

OSPFv2-p2mp-failing

R1 sends a unicast DBD packet to R4, but uses the wrong mac address.

          10.12.34.0/24         
                                
    .3d00             .4d00     
   ┌──────┐          ┌──────┐   
.1 │  R1  ├─┐      ┌─┤  R2  │ .2
   └──────┘ │      │ └──────┘   
           ┌┴──────┴┐           
           │ Switch │           
           └┬──────┬┘           
   ┌──────┐ │      │ ┌──────┐   
.3 │  R3  ├─┘      └─┤  R4  │ .4
   └──────┘          └──────┘   
    .4e00             .3e00     

The Packets in Text

!
! Send a DBD Packet
!
Frame 1: Packet, 78 bytes on wire (624 bits), 78 bytes captured (624 bits)
Ethernet II, Src: aa:bb:cc:00:3d:00 (aa:bb:cc:00:3d:00), Dst: aa:bb:cc:00:4d:00 (aa:bb:cc:00:4d:00)
Internet Protocol Version 4, Src: 10.12.34.1, Dst: 10.12.34.4
Open Shortest Path First
    OSPF Header
    OSPF DB Description
    OSPF LLS Data Block
!
! Get an ICMP TTL exceeded
!
Frame 2: Packet, 70 bytes on wire (560 bits), 70 bytes captured (560 bits)
Ethernet II, Src: aa:bb:cc:00:4d:00 (aa:bb:cc:00:4d:00), Dst: aa:bb:cc:00:3d:00 (aa:bb:cc:00:3d:00)
Internet Protocol Version 4, Src: 10.12.34.2, Dst: 10.12.34.1
Internet Control Message Protocol
    Type: Time-to-live exceeded (11)
    Code: 0 (Time to live exceeded in transit)
    Checksum: 0xf0db [correct]
    [Checksum Status: Good]
    Unused: 00000000
    Internet Protocol Version 4, Src: 10.12.34.1, Dst: 10.12.34.4
    Open Shortest Path First
        OSPF Header
            Version: 2
            Message Type: DB Description (2)
            Packet Length: 32
            Source OSPF Router: 1.1.1.1

The OSPFv2 shortest path topology.

OSPFv2 using point-to-multipoint assumes the topology looks like this.

         ┌──────┐              
         │  R2  │              
         └──┬───┘              
            │                  
            │                  
          .-~~~-.              
  .- ~ ~-(       )_ _          
 /                    ~ -.     
|                          ',  
 \                         .'  
   ~- ._ ,. ,.,.,., ,.. -~     
     │     '       '    │      
     │                  │      
     │                  │      
┌────┴─┐              ┌─┴────┐ 
│  R1  │              │  R4  │ 
└──────┘              └──────┘ 

Workarounds

You can just make the routers not advertise their /32 networks using prefix suppression. This may not work in all IOS, IOS-XE versions.

interface Ethernet0/0
 ip address 10.12.34.1 255.255.255.0
 ip ospf network point-to-multipoint non-broadcast
 ip ospf prefix-suppression
 ip ospf 1 area 0
end

References

Point-to-multipoint communication - Wikipedia

Solved: OSPF Point-to-multipoint network type - Cisco Community

RFC 2328: OSPF Version 2 | RFC Editor

Cisco - IP Routing: OSPF Configuration Guide - OSPF Mechanism to Exclude Connected IP Prefixes from LSA Advertisements Cisco - IOS 15.2E

OSPFv2 LFA

Relies on Triangles.

References

Loop-Free Alternate: OSPF Meets EIGRP « ipSpace.net blog

RFC 5286: Basic Specification for IP Fast Reroute: Loop-Free Alternates | RFC Editor

OSPFv2 Sham Links

The problem

A customer with L3VPN service via OSPFv2-BGP-VPNv4 decides to connect two sites together via OSPFv2 backdoor, a direct connection they manage themselves.

When they turn on their private OSPFv2 peering, all the traffic between these two sites now prefers the new link, vs the L3VPN cloud.

The solution