CCIE TO BE

This is how long I waited for NTP to sync today:

R2(config)#ntp server 136.10.4.4
R2(config)#^Z
Feb 1 19:26:53.915: %SYS-5-CONFIG_I: Configured from console by console

Feb 1 19:37:11.852: NTP Core(NOTICE): Clock is synchronized.

More than 10 minutes. It should be noted that the clocks were only seconds apart to begin with. Code on these routers is 12.4(22)T. I don't know if I have ever waited so long but it's unbelievably ridiculous.

Then I enable authentication:

R4(config)#ntp authentication-key 1 md5 ipexpert

R2(config)#ntp authentication-key 1 md5 ipexpert
R2(config)#ntp trusted-key 1
R2(config)#ntp authenticate
R2(config)#ntp server 136.10.4.4 key 1

Feb 1 19:45:02.628: NTP Core(INFO): key (1) added.
Feb 1 19:45:02.752: NTP Core(INFO): key (1) marked as trusted.
Feb 1 19:45:03.276: NTP Core(INFO): system event 'event_clock_reset' (0x05) status 'sync_alarm, sync_unspec, 10 events, event_peer/strat_chg' (0xC0A4)
Feb 1 19:45:03.276: NTP Core(NOTICE): Clock synchronization lost.

Peers never come up, I get this every so often (debug ntp all):

.Feb 1 19:45:47.852: NTP message sent to 136.10.4.4, from interface 'Loopback0' (136.10.2.2).
.Feb 1 19:45:47.852: NTP message received from 136.10.4.4 on interface 'Loopback0' (136.10.2.2).
.Feb 1 19:45:47.852: NTP Core(DEBUG): ntp_receive: message received
.Feb 1 19:45:47.852: NTP Core(DEBUG): ntp_receive: peer is 0x674B9DF8, next action is 1.
.Feb 1 19:45:47.852: NTP Core(NOTICE): ntp_receive: dropping message: crypto-NAK.

.Feb 1 19:50:52.852: NTP message sent to 136.10.4.4, from interface 'Loopback0' (136.10.2.2).
.Feb 1 19:50:52.852: NTP message received from 136.10.4.4 on interface 'Loopback0' (136.10.2.2).
.Feb 1 19:50:52.852: NTP Core(DEBUG): ntp_receive: message received
.Feb 1 19:50:52.852: NTP Core(DEBUG): ntp_receive: peer is 0x674B9DF8, next action is 1.
.Feb 1 19:50:52.852: NTP Core(NOTICE): ntp_receive: dropping message: crypto-NAK.

Here we are still

.Feb 1 19:58:19.851: NTP message sent to 136.10.4.4, from interface 'Loopback0' (136.10.2.2).
.Feb 1 19:58:19.851: NTP message received from 136.10.4.4 on interface 'Loopback0' (136.10.2.2).
.Feb 1 19:58:19.851: NTP Core(DEBUG): ntp_receive: message received
.Feb 1 19:58:19.851: NTP Core(DEBUG): ntp_receive: peer is 0x674B9DF8, next action is 1.
.Feb 1 19:58:19.851: NTP Core(NOTICE): ntp_receive: dropping message: crypto-NAK.

So, for kicks on the master I do this:

R4(config)#ntp authenticate
R4(config)#ntp trusted-key 1

I now get a new message on R2:

.Feb 1 20:01:20.851: NTP message sent to 136.10.4.4, from interface 'Loopback0' (136.10.2.2).
.Feb 1 20:01:20.851: NTP message received from 136.10.4.4 on interface 'Loopback0' (136.10.2.2).
.Feb 1 20:01:20.851: NTP Core(DEBUG): ntp_receive: message received
.Feb 1 20:01:20.851: NTP Core(DEBUG): ntp_receive: peer is 0x674B9DF8, next action is 1.
.Feb 1 20:01:20.851: NTP Core(DEBUG): receive: packet given to process_packet

This looks promising:

R2#
.Feb 1 20:03:30.851: NTP message sent to 136.10.4.4, from interface 'Loopback0' (136.10.2.2).
.Feb 1 20:03:30.851: NTP message received from 136.10.4.4 on interface 'Loopback0' (136.10.2.2).
.Feb 1 20:03:30.851: NTP Core(DEBUG): ntp_receive: message received
.Feb 1 20:03:30.851: NTP Core(DEBUG): ntp_receive: peer is 0x674B9DF8, next action is 1.
.Feb 1 20:03:30.851: NTP Core(DEBUG): receive: packet given to process_packet
.Feb 1 20:03:30.851: NTP Core(DEBUG): Peer becomes reachable, poll set to 6.
.Feb 1 20:03:30.851: NTP Core(INFO): peer 136.10.4.4 event 'event_reach' (0x84) status 'unreach, conf, auth, 1 event, event_reach' (0xE014)

TA-DA!

.Feb 1 20:06:43.851: NTP Core(NOTICE): Clock is synchronized.

I have never had to enable trusted-key on the master before. Watch this:

R4(config)#no ntp trusted-key 1

Back on R2:

R2#
Feb 1 20:07:47.851: NTP message sent to 136.10.4.4, from interface 'Loopback0' (136.10.2.2).
Feb 1 20:07:47.851: NTP message received from 136.10.4.4 on interface 'Loopback0' (136.10.2.2).
Feb 1 20:07:47.851: NTP Core(DEBUG): ntp_receive: message received
Feb 1 20:07:47.851: NTP Core(DEBUG): ntp_receive: peer is 0x674B9DF8, next action is 1.
Feb 1 20:07:47.851: NTP Core(INFO): system event 'event_clock_reset' (0x05) status 'sync_alarm, sync_unspec, 15 events, event_peer/strat_chg' (0xC0F4)
Feb 1 20:07:47.851: NTP Core(NOTICE): Clock synchronization lost.
.Feb 1 20:07:47.851: NTP Core(NOTICE): ntp_receive: dropping message: crypto-NAK.

Maybe something has changed in this T train but looks like we need "ntp trusted-key" on the Master now. I am not an NTP guru by any means but if you look at some of my other ntp blogs, you will see I didn't need this command. Note that I only needed "trusted-key" on the Master, not "ntp authenticate" even though I showed it above. Removing it did not cause sync loss. Something to keep in mind if you find yourself singing the NTP blues.

Oh, and while you are waiting for the sync - go configure something else in the meantime!

This is scenario 3 dealing with NTP and in this post I will set up NTP authentication. Remember that the purpose is for the client to authenticate the server. First will make sure NTP is synchronized on the client. Next, we will turn on authentication on the client only until the clocks become unsynchronized. Finally, we will configure the server with the appropriate key and make sure the client is synchronized again.

We use the same topology as the other NTP scenarios:
R4, R5 and R6 connected via full mesh frame-relay on subnet 172.12.45.0/24

Let's make sure R6, the master, is synchronized first:

R6#show clock
20:14:37.215 MDT Mon Jun 2 2008

R6#show ntp status | inc Clock is
Clock is synchronized, stratum 2, reference is 127.127.7.1

R4:

R4#show ntp status | in Clock
Clock is synchronized, stratum 3, reference is 172.12.45.6

and R5:

R5#show ntp status | inc Clock
Clock is synchronized, stratum 3, reference is 172.12.45.6

Let's configure authentication on R4:

R4(config)#ntp authenticate
R4(config)#ntp authentication-key 1 md5 cisco
R4(config)#ntp trusted-key 1
R4(config)#ntp server 172.12.45.6 key 1

R4#debug ntp validity
NTP peer validity debugging is on
R4#debug ntp authentication
NTP authentication debugging is on

In a few moments we get these debug messages:

R4(config)#
Jun 3 03:24:49.759: NTP: packet from 172.12.45.6 failed validity tests 10
Jun 3 03:24:49.763: Authentication failed
R4(config)#

What's strange is that R4 is still synchronized with R6:

R4#show ntp status
Clock is synchronized, stratum 3, reference is 172.12.45.6

So I waited about 10 minutes (it seemed that long) and now R4 is unsynchronized:

R4#show ntp status | inc Clock
Clock is unsynchronized, stratum 16, no reference clock

Now let's configure NTP authentication on R6:

R6(config)#ntp authenticate
R6(config)#ntp authentication-key 1 md5 cisco

On R4 I will set up some debugging, a lot of the output doesn't make much sense, but here it is:

R4#debug ntp packets
NTP packets debugging is on

!! HERE R4 TRANSMITS A PACKET WITH AUTHENTICATION-KEY 1

.Jun 3 03:36:10.711: NTP: xmit packet to 172.12.45.6:
.Jun 3 03:36:10.715: leap 3, mode 3, version 3, stratum 0, ppoll 64
.Jun 3 03:36:10.715: rtdel 0D4E (51.971), rtdsp 05CD (22.659), refid AC0C2D06 (172.12.45.6)
.Jun 3 03:36:10.719: ref CBEF37C1.C8A9606E (20:23:45.783 MDT Mon Jun 2 2008)
.Jun 3 03:36:10.719: org CBEF3A6A.BF8D3204 (20:35:06.748 MDT Mon Jun 2 2008)
.Jun 3 03:36:10.723: rec CBEF3A6A.C784D317 (20:35:06.779 MDT Mon Jun 2 2008)
.Jun 3 03:36:10.723: xmt CBEF3AAA.B614DB93 (20:36:10.711 MDT Mon Jun 2 2008)
.Jun 3 03:36:10.727: Authentication key 1

!! HERE R4 RECEIVES A PACKET WITH AUTHENTICATION-KEY 1

.Jun 3 03:36:10.771: NTP: rcv packet from 172.12.45.6 to 172.12.34.4 on FastEthernet0/1:
.Jun 3 03:36:10.775: leap 0, mode 4, version 3, stratum 2, ppoll 64
.Jun 3 03:36:10.779: rtdel 0000 (0.000), rtdsp 0002 (0.031), refid 7F7F0701 (127.127.7.1)
.Jun 3 03:36:10.779: ref CBEF3A89.50E1C8AB (20:35:37.315 MDT Mon Jun 2 2008)
.Jun 3 03:36:10.783: org CBEF3AAA.B614DB93 (20:36:10.711 MDT Mon Jun 2 2008)
.Jun 3 03:36:10.783: rec CBEF3AAA.C3941C78 (20:36:10.763 MDT Mon Jun 2 2008)
.Jun 3 03:36:10.787: xmt CBEF3AAA.C39DD400 (20:36:10.764 MDT Mon Jun 2 2008)
.Jun 3 03:36:10.791: inp CBEF3AAA.C565D38F (20:36:10.771 MDT Mon Jun 2 2008)
.Jun 3 03:36:10.791: Authentication key 1

R4 is now synchronized:

R4#show ntp status
Clock is synchronized, stratum 3, reference is 172.12.45.6
nominal freq is 250.0000 Hz, actual freq is 250.0004 Hz, precision is 2**18
reference time is CBEF3AAA.C565D38F (20:36:10.771 MDT Mon Jun 2 2008)
clock offset is 22.8825 msec, root delay is 59.68 msec
root dispersion is 15897.92 msec, peer dispersion is 15875.02 msec

Let's check R5 to see if it is still synchronized:

R5#show ntp status | inc Clock
Clock is synchronized, stratum 3, reference is 172.12.45.6

As you can see, R5 is still synchronized because it is not requesting authentication from R6. R4 on the other sends the request with an authentication key and requires that the time-source, R6, does so as well.

Well, this lab went okay. I have had many problems with ntp authentication before. Initially, I thought I would have problems again when R4 was not un-synchronizing with R6 when we first set up a key on R4. Well, it took a few minutes, so I guess patience is this key. You can't afford to be too patient though, you only have 8 hours on lab day :)

I will call this NTP blog NTP Scenario 2: NTP with access-group filtering.

R4, R5, and R6 are connected via full mesh frame-relay
R6 will be the NTP server.

In scenario 1 we set up R4 to sync to R6. In this scenario we
will prevent R5 from syncing with R6 by using ntp access-group command.

I have never used this command before, so I hope I am understanding it correctly. If not, please let me know how else to test this command.

On R6 we have the following ACL:

access-list 1 permit 172.12.45.4
access-list 1 permit 127.127.7.1

172.12.45.4 is the address of R4. 127.127.7.1 is the IP address that a cisco router uses as it's reference when you make the router an NTP master. This must be added to the ACL or it will become unsynchronized with itself.

In global config mode we enter:

ntp access-group serve-only 1

Let's check the current times:

R5#show clock
16:15:04.031 UTC Sat May 31 2008

R6#show clock
16:31:45.687 UTC Sat May 31 2008

R5 is about 15 minutes behind. I don't know a way to debug on R6 to make sure it's working, so what I will do is wait about 10 minutes then add R5's IP address to ACL. I think 10 minutes is sufficient to prove that R6 is not allowing R5 to sync to it.

...
...

Well it's almost 20 minutes now. Let's check if R4 is synchronized:

R4#show ntp status | inc Cloc
Clock is synchronized, stratum 3, reference is 172.12.45.6

How about R5:

R5#sho ntp status | inc Clock
Clock is unsynchronized, stratum 16, no reference clock

Some quick show clocks:

R4#show clock
16:49:06.505 UTC Sat May 31 2008

R5#show clock
16:32:29.755 UTC Sat May 31 2008

R6#show clock
16:49:12.335 UTC Sat May 31 2008

R5 is still unsynchronized and more than 15 minutes behind R6. Let's add R5's address to ACL 1 on R6 and see how long it takes to sync...

R6(config)#access-list 1 permit 172.12.45.5

We'll debug on R5:

R5#debug ntp sync
NTP clock synchronization debugging is on
R5#sho ntp status | inc Clock
Clock is unsynchronized, stratum 16, no reference clock
R5#
May 31 16:33:54.039: NTP: synced to new peer 172.12.45.6

Took less than a minute!

This is the second of many NTP scenarios to come. It really seems so simple, but I have always had problems understanding NTP server/peer relationships and authentication configurations. So these will be the topics of future NTP blogs. Hopefully I (and you!) will become NTP masters in time for CCIE :)

I will call this NTP Scenario 1: Basic NTP with no authentication. There will be more blogs about NTP but this is just to get it started.

R4, R5, and R6 are connected via full mesh frame-relay
R6 will be the NTP server.

First set the clock on R6:

R6#clock set 15:47:30 31 May 2008
*May 31 15:47:30.003: %SYS-6-CLOCKUPDATE: System clock has been updated from 01:
05:19 UTC Fri Mar 1 2002 to 15:47:30 UTC Sat May 31 2008, configured from consol
e by console.

Verify the time with show clock:

R6#show clock
15:47:36.651 UTC Sat May 31 2008

Configure R6 as the master along with stratum:

R6#conf t
R6(config)#ntp master ?
<1-15> Stratum number


R6(config)#ntp master 3
R6(config)#^Z

Verify that R6 is synced with itself:

R6#show ntp associations

address ref clock st when poll reach delay offset disp
*~127.127.7.1 127.127.7.1 2 22 64 1 0.0 0.00 15875.
* master (synced), # master (unsynced), + selected, - candidate, ~ configured

R6#show ntp status
Clock is synchronized, stratum 3, reference is 127.127.7.1
nominal freq is 250.0000 Hz, actual freq is 250.0000 Hz, precision is 2**24
reference time is CBEBF22C.1583DCD8 (15:50:04.084 UTC Sat May 31 2008)
clock offset is 0.0000 msec, root delay is 0.00 msec
root dispersion is 15875.02 msec, peer dispersion is 15875.02 msec

Now it's time to configure the clients, let's first check the time on R4:

R4#show clock
*03:03:44.823 UTC Fri Mar 1 2002
R4#conf t
R4(config)#ntp server 172.12.45.6
R4(config)#^Z
R4#show clock
*03:04:52.047 UTC Fri Mar 1 2002
R4#

Notice the time is 6 years behind the current date, I have noticed that NTP will not sync when the times are so far apart (Not sure how this works, but let's manually 'push' R4 a little closer)

R4#clock set 15:30:40 31 May 2008

Now we're about 15 minutes behind. Let's do some debugging:

R4#debug ntp sync
NTP clock synchronization debugging is on

.May 31 15:31:43.107: NTP: synced to new peer 172.12.45.6

Noe that we're synced let's verify on R6 and R4:

R4#show clock
15:59:42.466 UTC Sat May 31 2008

R6#show clock
15:59:41.135 UTC Sat May 31 2008

Looks good. I will use this topology for other NTP labs so stay tuned...

Atom

Atom