EFTA00593034.pdf

LSJ Preliminary Report Updated 12/14/15 This is a preliminary report and items will be added as I look thru the network and equipment configurations. Dragonwave Login Credentials The system had the default credentials. This was changed, because the dragonwave radios have public Ips, anyone from the internet could connect and this was a big security risk in terms of someone could change settings and cause the link to fail. Recommendations: Credentials were changed and the Ip will be changed to private. This will restrict access to the management to our VPN and will integrate the link to Choice Monitoring system. Firmware Bug Currently there are two bugs that I have identified, One is non service affecting and the following is service affecting: I've monitoring the link since Friday and there has been two drops in the dragonwave: LQ Severity Time Node Interface 92269658 Normal (s) (.1 12/7/ 15 12:30:52 [c] ['] custstt.14).2921 uehopennms.org/nodes/nodoUp (+1 Li Edit notifications for event Node cust.sttisj.2921 is up. 92286745 Major NI [•) 1207/15 12:15:30 (a) (>) cust.sttisj.2921 uei.opennms.org/nodes/nodeDown [-3 Edit notifications for event Node cust.stflsj.2921 is down. 92222554 Normal (+3 [.3 12/6/15 13:04:34 (.c) ( >) cu.t.stt.Isi.2921 um.opennms.ors/nodes/nodeUp (+) (-) Edit notification•, for event Node cust.stt.lsj.2921 is up. 92217839 Major (+3 (.3 12/6/15 11:21:36 (<) ( >) cust.sttisj.2921 uei.opennms.orlynodes/nedeDown [+3 (-3 Edit notifications for event Node cust.stt.lsj.2921 is down. Both times were due to a known bug on the firmware on the radios: Notice:Dec 07 12:12:36 C1N14DGM0075 alarm1544789-1]: Notice:Dec 06 11:21:08 C1N14DGM0075 alarm1544789-1]: Currently LSJ Dragonwave radios are running Version 1.2.2 Major: Queue depth threshold q4 Alarm raised Major: Queue depth threshold q4 Alarm raised EFTA00593034 Active: Bank A Component I Version I Validation OMNI Release I 1.2.2 I Valid Frequency File I 2.01.09 I Valid MIB I 3.0.0 I Valid All of our radios that are Compaq+ are running Version 1.3.9 Active: Bank A Component I Version I Validation OMNI Release I 1.3.9 I Valid Frequency File I 2.01.27 I Valid MIB I 5.0.0 I Valid This is a known issue with older version of this firmware. So far both times have been triggered by the rain (Sunday and today) the wireless link did not drop, but due to the rain were some data errors and as the radios retransmitted the data (this is normal behavior) this alarm is raised and crashes data communications. Once the radios are rebooted or power cycle normal transmission begins until the next event. Recommendation: To fix this issue a firmware upgrade needs to be done. I recommend once ViNGN is in place and this link can be taken offline to do the following: • Reset this link to factory defaults. • Reconfigured to Choice Standards. • Change Management Ips to Private IPs Choice Management. • Upgrade firmware on both radios. Configuration (Modulation) The radio currently is configured to a fixed maximum speed (256 QAM 279 Megs) . Recommendation: The link was coordinated with HAAM (Hitless Automatic Adaptive Modulation) with this feature the link will automatically drop modulation when it cannot maintain a signal without errors. Doing this, the link will be more stable and less susceptible to rain fade. Alignment (Signal Strength) The microwave link is outside of the optimal signal the coordination calls for -35.7 dBm (optimal would be +/- 4.0 dBm) and currently the signal strength is at -40.1 dBm. This can be caused by having the LSJ side of the link only at 10 ft. When you have links so low the ground can cause interference in what is call the Fresnel Zone. Basically this zone has to be obstruction free for a microwave link to be at its optimal. See following graph. EFTA00593035 TX antenna Height lOm 1st Fresnel zone RX antenna Height 49m ____________ ------------ Obstacle zone //// // Recommendation: raise the link to a higher height. For more detail about Fresnel Zone http://www.zytrax.com/tech/wireless/fresnel.htm Network Loop At LSJ shelter the following equipment are installed: • Netgear Prosafe JGS516PE • 1 Dragonwave Compaq — Link to St Thomas • 1 Rockus AP — Looking towards 5 Palms • 1 Ruckus Bridge — Looking towards the Gym There are 2 issues with this setup, 1) The dragonwave and the bridge are on the public side of this connection (internet) the AP (5 Palms) needs to be on the private side (internal network). Because there is no way to have communications to the internal network from the shelter, the AP to 5 palms was plugged into the public side of the network. This is why this AP is not working. 2) Both the Data and Management Ethernet cables of the Dragonwave radio were plugged in to the netgear. Creating a local network look between the Dragonwave and the Netgear. This is what created the unstable internet connection that caused between 5%-10% packet drop and caused real time applications and services like Netflix, Hulu, YouTube, and Etc. buffer or not to work. This can also cause websites to pause as the page was displaying (downloading) or a lag in clicking on a link. Recommendation: to fix the issues: • Issue #1 to fix the AP looking to 5 Palm the Rockus needs to be moved to the Choice 3560G switch by installing a POE power module. Once this is done I can create an internal port (Vlan) that will only talk to the private side. The Dragonwave and the Rockus Bridge will remain in the public side. ** Fixed • Issue #2 was fixed (network loop) was fixed by disconnecting the local management Ethernet cable. Issue #1 was fixed. 2 PowerOverEthernet (PoE) injectors were installed and both Ruckus were moved to Choice 3560 switch. A cable was installed to interconnect choice switch with LSJ switch to pass vlan 8 (internal Data) and the Ruckus that was pointing to 5 Palms is now in serviced. EFTA00593036 Recent Outages This is a list of outages and caused: • Dec 6 11:21 — Dragonwave Bug related • Dec 7 12:12 — Dragonwave Bug related • Dec 10 11:06 — Port on cisco router down — possibly switch at gym losing power • Dec 10 11:22 — Port on cisco router down — possibly switch at gym losing power • Dec 11 9:11 — Port on cisco router down — possibly switch at gym losing power • Dec 11 9:21 — Port on cisco router down — possibly switch at gym losing power • Dec 12 8:55 — Port on cisco router down — possibly switch at gym losing power • Dec 12 9:16 — Port on cisco router down — possibly switch at gym losing power • Dec 12 20:10 — nothing found in the logs could have been power but cannot confirm EFTA00593037 Configuration Drawing Choice IPAddresses 66.185.40.60/30 66.185.40.61 — Default Gateway 66.185.40.62— Free Mask 255.255 255.252 DNS 66.185.33.226 and 66.185.33.230 viNGN IP Addresses 208.50.70.188/27 20350 78190 — Default Gateway 20550 78.188— Free Mask 255.255.255.224 DNS 209 244 03 and 209 244 04 OW CP+ 5O Melts Slave 100 Ebi -13MENT-02neegs 100 Mess Vessup Beach Chace ernet stalley 3560 Truri: 39 Mngm 40 111 \4 3N Internet Southern Trust — Little St limos'\ Draft 12/7/2015 ver 1.0 EFTA00593038