Upgrade de dos switchs Catalyst 3850 conectados en stack

Fecha: 23 de febrero del 2017

Escenario

Este upgrade se realizó para solucionar un problema con módulos SFP GLC-LH-SM que el switch detecta con el

CRC inválido y los deja en estado de error-disabled.

Para que el stack funcione ambos switchs deben tener la misma versión de sistema operativo, aquí repasamos

los pasos para actualizar ambos al mismo tiempo.

Si bien el upgrade no solucionó el problema, es bueno dcumentar un upgrade que no se realiza todos los días.

1.- Se verifican con MD5 las imágenes bajadas desde la web de Cisco.

2.- Copia desde TFTP al switch:

Cisco-3850#copy tftp flash-1:

Address or name of remote host []? 10.89.75.104

Source filename []? cat3k_caa-universalk9.SPA.03.06.06.E.152-2.E6.bin

Destination filename [cat3k_caa-universalk9.SPA.03.06.06.E.152-2.E6.bin]? (enter)

Accessing tftp://10.89.75.104/cat3k_caa-universalk9.SPA.03.06.06.E.152-2.E6.bin...

Loading cat3k_caa-universalk9.SPA.03.06.06.E.152-2.E6.bin from 10.89.75.104 (via Vlan75): !!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!

!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!

[OK - 303772864 bytes]

303772864 bytes copied in 1723.830 secs (176220 bytes/sec)

Cisco-3850#

Cisco-3850#copy tftp flash-2:

Address or name of remote host []? 10.89.75.104

Source filename []? cat3k_caa-universalk9.SPA.03.06.06.E.152-2.E6.bin

Destination filename [cat3k_caa-universalk9.SPA.03.06.06.E.152-2.E6.bin]? (enter)

Accessing tftp://10.89.75.104/cat3k_caa-universalk9.SPA.03.06.06.E.152-2.E6.bin...

!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!

[OK - 303772864 bytes]

303772864 bytes copied in 1723.830 secs (176220 bytes/sec)

Cisco-3850#

3.- Verificación de la copia:

Cisco-3850#sh flash-1: | incl .bin

26 303772864 Feb 23 2017 10:25:20.0000000000 +00:00 cat3k_caa-universalk9.SPA.03.06.06.E.152-2.E6.bin

Cisco-3850#

Cisco-3850#sh flash-2: | incl .bin

29 303772864 Feb 23 2017 10:56:46.0000000000 +00:00 cat3k_caa-universalk9.SPA.03.06.06.E.152-2.E6.bin

Cisco-3850#

4.- Verificación del entorno de arranque del switch:

Cisco-3850#sh boot system switch 1

---------------------------

Switch 1

---------------------------

Current Boot Variables:

BOOT variable does not exist

Boot Variables on next reload:

BOOT variable = flash:packages.conf

Allow Dev Key = yes

Manual Boot = no

Enable Break = no

Cisco-3850#sh boot system switch 2

---------------------------

Switch 2

---------------------------

Current Boot Variables:

BOOT variable does not exist

Boot Variables on next reload:

BOOT variable = flash:packages.conf

Allow Dev Key = yes

Manual Boot = no

Enable Break = no

Cisco-3850#sh version

Cisco IOS Software, IOS-XE Software, Catalyst L3 Switch Software (CAT3K_CAA-UNIVERSALK9-M), Version 03.07.02E RELEASE SOFTWARE (fc1)

---resumido----

ROM: IOS-XE ROMMON

BOOTLDR: CAT3K_CAA Boot Loader (CAT3K_CAA-HBOOT-M) Version 1.18, RELEASE SOFTWARE (P)

Cisco-3850 uptime is 2 days, 22 hours, 51 minutes

Uptime for this control processor is 2 days, 22 hours, 55 minutes

System returned to ROM by Power Failure

System image file is "flash:packages.conf"

Last reload reason: Power Failure

---resumiudo---

Model Number : WS-C3850-48P

System Serial Number : FCW1927D1TP

Switch Ports Model SW Version SW Image Mode

------ ----- ----- ---------- ---------- ----

* 1 56 WS-C3850-48P 03.07.02E cat3k_caa-universalk9 INSTALL

2 20 WS-C3850-12S 03.07.02E cat3k_caa-universalk9 INSTALL

Switch 02

---------

Switch uptime : 2 days, 22 hours, 55 minutes

Base Ethernet MAC Address : 84:b5:17:e7:c1:80

Motherboard Assembly Number : 73-15839-07

Motherboard Serial Number : FOC192774G9

Model Revision Number : F0

Motherboard Revision Number : A0

Model Number : WS-C3850-12S

System Serial Number : FOC1928X04C

Configuration register is 0x102

Cisco-3850#

5.- Modificación del arranque de los switchs 1 y 2:

Cisco-3850#conf t

Cisco-3850(config)#boot system switch all flash:/cat3k_caa-universalk9.SPA.03.06.06.E.152-2.E6.bin

Cisco-3850(config)#end |

Cisco-3850# esto comprende los switch 1 y 2

6.- Verificación del entorno de arranque del switch:

Cisco-3850#sh boot system switch 1

---------------------------

Switch 1

---------------------------

Current Boot Variables:

BOOT variable = flash:/cat3k_caa-universalk9.SPA.03.06.06.E.152-2.E6.bin;

Boot Variables on next reload:

BOOT variable = flash:packages.conf

Allow Dev Key = yes

Manual Boot = no

Enable Break = no

Cisco-3850#

Cisco-3850#sh boot system switch 2

---------------------------

Switch 2

---------------------------

Current Boot Variables:

BOOT variable = flash:/cat3k_caa-universalk9.SPA.03.06.06.E.152-2.E6.bin;

Boot Variables on next reload:

BOOT variable = flash:packages.conf

Allow Dev Key = yes

Manual Boot = no

Enable Break = no

Cisco-3850#

7.- Se guardan los cambios:

Cisco-3850#write (esto lo ejecuta simultaneamente en ambos switchs)

Building configuration...

Compressed configuration from 8218 bytes to 2933 bytes[OK]

Cisco-3850#

8.- Reinicio de los switchs:

Cisco-3850#reload

Reload command is being issued on Active unit, this will reload the whole stack (esto lo ejecuta simultaneamente en ambos switchs)

Proceed with reload? [confirm] (enter)

*Feb 23 10:58:03.785: %SYS-5-RELOAD: Reload requested by console. Reload Reason: Reload command.

*Feb 23 10:58:04.403: %STACKMGR-1-RELOAD_REQUEST: 1 stack-mgr: Received reload request for all switches, reason Reload command

*Feb 23 10:58:04.405: %STACKMGR-1-RELOAD: 1 stack-mgr: Reloading due to reason Reload command

*Feb 23 10:58:04.403: %STACKMGR-1-RELOAD: STANDBY:2 stack-mgr: Reloading due to reason Reload command (Cisco-3850-2)

*Feb 23 10:58:04.905: %IOSXE-3-PLATFORM: 1 process sysmgr: Reset/Reload requested by [stack-manager].

*Feb 23 10:58:04.503: %IOSXE-3-PLATFORM: STANDBY:2 process sysmgr: Reset/Reload requested by [stack-manager]. (Cisco-3850-2)

---resumido---

File "flash:/cat3k_caa-universalk9.SPA.03.06.06.E.152-2.E6.bin" uncompressed and installed, entry point: 0x8166a600

---resumido---

Front-end Microcode IMG MGR: Preparing to program device microcode...

Front-end Microcode IMG MGR: Preparing to program device[0]...594412 bytes.

Front-end Microcode IMG MGR: Programming device 0...rwRrrrrrrwsssssssssssssssss (esto formatea el microcódigo, ver mas abajo)

sspssssssssssssssssssspssssssssssssssssssspssssssssssssssssssspsssssssssssssssssssps

Sssssssssssssssssspssssssssssssssssssspssssssssssssssssssspssssssssssssssssssspssssss

---resumido---

sspssssssssssssssssssspssssssssssssssssssspssssssssssssssssssspssssssssssssssssssspss

ssssssspssssssssssssssssssspssssssssssssssssssspsssssssssssssssssssp

Front-end Microcode IMG MGR: Preparing to program device[0]...381970 bytes.

Front-end Microcode IMG MGR: Programming device 0...rrrrrrwssssssssssssssssssspss

Ssssssssssssssssspssssssssssssssssssspssssssssssssssssssspssssssssssssssssssspssssssss

Ssssssssssspssssssssssssssssssspssssssssssssssssssspssssssssssssssssssspssssssssssssss

---resumido---

ssssssssssssspssssssssssssssssssspssssssssssssssssssspssssssssssssssssssspssssssssssss

ssssspssssssssssssssssssspssssssssssssssssssspssssssssssssssssssspsssssssssssssssssssp

Front-end Microcode IMG MGR: Preparing to program device[0]...25186 bytes.

Front-end Microcode IMG MGR: Programming device 0...rrrrrrwssssssssssssssssssspsss

Sssssssssssssssspssssssssssssssssssspssssssssssssssssssspssssssssssssssssssspssssssssss

Ssssssssspssssssssssssssssssspssssssssssssssssssspssssssssssssssssssspsssssssssssssssss

---resumido---

Sssssssspssssssssssssssssssspssssssssssssssssssspssssssssssssssssssspssssssssssssssssss

spssssssssssssssssssspssssssssssssssssssspw

Front-end Mÿø

---resumido---

Bootable image at @ ram:0x6042d274

Bootable image segment 0 address range [0x81100000, 0x82140000] is in range [0x80180000, 0x90000000].

@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@

@@@@@@@@@@@@@@@@@@@@@@@@@@@@@

File "flash:/cat3k_caa-universalk9.SPA.03.06.06.E.152-2.E6.bin" uncompressed and installed, entry point: 0x8166a600

Loading Linux kernel with entry point 0x8166a600 ...

Bootloader: Done loading app on core_mask: 0xf

### Launching Linux Kernel (flags = 0x5)

All packages are Digitally Signed

Starting System Services

Press RETURN to get started!

Cisco-3850>enable

*Feb 23 11:20:51.883: %PLATFORM_HA-6-REDUNDANCY_MODE: ACTIVE supervisor initializing for SSO mode

Password:

Cisco-3850#conf t (no permite la configuración hasta que la sincronización entre ambos switch se complete)

Config mode cannot be entered during Standby initialization

Cisco-3850#

*Feb 23 11:21:56.751: %HA_CONFIG_SYNC-6-BULK_CFGSYNC_SUCCEED: Bulk Sync succeeded

*Feb 23 11:21:57.737: %RF-5-RF_TERMINAL_STATE: 1 ha_mgr: Terminal state reached for (SSO)

Cisco-3850#conf t

Cisco-3850(config)# (listo !)

Fuente: http://www.cisco.com/c/en/us/support/docs/switches/catalyst-3750-series-switches/64898-upgrade-3750-stack.html#cfbin1

Upgrade de microcódigo:

During an IOS image upgrade or downgrade on a PoE or UPoE switch, the microcode is updated to reflect applicable

feature enhancements and bug fixes. Do not restart the switch during the upgrade or downgrade process.

With the Cisco IOS XE Denali 16.1.1 release, it takes approximately an additional 4 minutes to complete the microcode

upgrade in addition to the normal reload time. The microcode update occurs only during an image upgrade or downgrade

on PoE or UPoE switches. It does not occur during switch reloads or on non-PoE switches).

Fuente: http://www.cisco.com/c/en/us/td/docs/switches/lan/catalyst3850/software/release/16-2/release_notes/ol-16-2-3850.html#pgfId-1127101

9.- Problema inicial con el módulo GBIC:

*Feb 23 14:37:39.189: %PLATFORM_PM-6-MODULE_ERRDISABLE: The inserted SFP module with interface name Gi1/1/1 is not supported

Cisco-3850#sh inventory

---resumido---

NAME: "Gi2/0/12", DESCR: "1000BaseLX SFP" (es el insertado previamente en Gi1/1/1)

PID: GLC-LH-SMD , VID: V01 , SN: FNS192416FQ

Cisco-3850#

Se configuró (sin resultados):

Cisco-3850(config)#service unsupported-transceiver

Cisco-3850(config)#no errdisable detect cause gbic-invalid

Cisco-3850(config)#errdisable recovery cause gbic-invalid

10.- Problema probablemente asociado a la misma falla:

Cisco-3850#sh spanning-tree vlan 100

VLAN0100

Spanning tree enabled protocol ieee

Root ID Priority 32768

Address 38ed.1882.6880

Cost 19

Port 48 (GigabitEthernet1/0/48)

Hello Time 2 sec Max Age 20 sec Forward Delay 15 sec

Bridge ID Priority 32868 (priority 32768 sys-id-ext 100)

Address 38ed.1882.7000

Hello Time 2 sec Max Age 20 sec Forward Delay 15 sec

Aging Time 15 sec

Interface Role Sts Cost Prio.Nbr Type

------------------- ---- --- --------- -------- --------------------------------

Gi1/0/48 Root FWD 19 128.48 P2p (conexión provisoria por media converter)

Gi1/1/1 Desg BLK 4 128.50 P2p self-looped (no es un bloqueo normal, ver que inclusive tiene mejor costo al root)

Cisco-3850#

Desconectando el media converter en Gi1/0/48 teóricamente descartamos el bucle:

*Feb 23 11:03:31.448: %LINEPROTO-5-UPDOWN: Line protocol on Interface GigabitEthernet1/0/48, changed state to down

*Feb 23 11:03:32.451: %LINK-3-UPDOWN: Interface GigabitEthernet1/0/48, changed state to down

Cisco-3850#sh spanning-tree vlan 100

VLAN0100

Spanning tree enabled protocol ieee

Root ID Priority 32768

Address 38ed.1882.6880

Cost 19

Port 48 (GigabitEthernet1/0/48)

Hello Time 2 sec Max Age 20 sec Forward Delay 15 sec

Bridge ID Priority 32868 (priority 32768 sys-id-ext 100)

Address 38ed.1882.7000

Hello Time 2 sec Max Age 20 sec Forward Delay 15 sec

Aging Time 15 sec

Interface Role Sts Cost Prio.Nbr Type

------------------- ---- --- --------- -------- --------------------------------

Gi1/1/1 Desg BLK 4 128.50 P2p self-looped (continuará…)

Cisco-3850#

(2017) Blocking my mind with spanning-tree

Rosario, Argentina