We made it!  We are releasing our first Management Pack.  It has been a long road to get here.  We have met our goal of building an SNMP Management Pack to DOES NOT use WMI or external files or an external interface to configure or poll network devices.  We tried to build the Management Pack to be as native as possible, to avoid unnecessary complications in the monitoring.

To kick off our release of our Management Packs we are releasing our Cisco Catalyst 2950 Series Switch Management Pack for a free download.  This MP represents the first Management Pack that we developed.  Since there is not much to a Catalyst 2950 Switch, the MP is relatively basic.  We cannot monitor, for instance, Broadcast Storms, on the Cat 2950 because there simply are no counters for it.  There are, however, some valuable counters and health information that we can gather from the switch.  These are collected and monitored in the MP.

We are also releasing our Cisco Catalyst 2960 Series Switch Management Pack.  This will be made available to all paid subscribers.  When you become a paid subscriber, you get all current and future Management Packs that we will develop.  You can see the "To Do" list to the left of this article.  All of these MP's will be made available to all subscribers as they are developed.

Our release schedule for new Management Packs will be a 2 - 3 week release timeline, depending upon the complexity of the device.  Our next MP release will be the Cisco PIX Firewall and Cisco ASA Security Appliance.

If you have ever struggled to understand the cryptic codes within an SNMP Trap message, you know it is a challenge that needs something similar to a “Rosetta Stone” to decipher them.  Worse yet, once you have decoded the message, you might find out it was just a simple notification telling you something like a log file is starting to get full.

While SNMP Traps can be very useful, they need to be understood correctly to be of any use at all.  Some NMS can help to do this and SCOM has the potential to do this.  Out of the box, SCOM can receive SNMP Traps, but it is still the cryptic message that we are familiar with.  You also have the ability to build Monitors and Alerts around these messages, but it can be very time-intensive to create all the Traps you want, much less the ones you really need.

This is what prompted us to start thinking about a solution to this problem.  The result: the JaxMP SNMP Trap Enhancer.

What is the SNMP Trap Enhancer?

The Trap Enhancer is our solution to the cryptic SNMP Trap message problem.  After many ideas, we came up with a nifty little application (Windows Service) and a Management Pack to help integrate it into SCOM.  The Trap Enhancer can actually be a stand-alone application, if you wish, or it can be integrated into System Center Operations Manager with the Management Pack.

In a nutshell, the Trap Enhancer is a Windows Service that uses the SNMP Trap Service on a Windows Server to read incoming Trap messages, translate them based upon a translation file, and then writes the message to the EventLog, a Syslog server, or both.  If you are also using the Management Pack, it will monitor the EventLog and will generate an Alert if it sees a message from the Trap Enhancer.  The Alert description will contain the EventLog Description, so you are able to easily read the incoming SNMP Trap message and control the Alerts through SCOM.

How does it work?

First, the Trap Enhancer is a Windows Service.  The reason for this is that is needs to run all the time, no matter who is logged in, and it can be controlled very easily.  The Trap Enhancer uses the Windows SNMP Trap Service to listen to incoming Trap messages.  You have to configure the Windows SNMP Trap Service first before the JaxMP Trap Enhancer will work correctly.

The Trap Enhancer will actually read the values in the incoming Trap message and store information about the SNMP Trap.  It uses the OID value of the incoming Trap message to perform a lookup on a translation file.  It then reads the translation file and starts to generate a new message based upon the values it finds.

The translation file, or TIF as we call it, is completely customizable.  We have generated over 1000 TIF files for all Cisco generated SNMP Traps and as many Standard RFC SNMP Traps as we could find.  The contents of the TIF are directly from the vendor’s MIB file.  Since the Trap messages are defined in the MIB, we used it to help us create the content.  The TIF files are text, so they are relatively easy to read and change.  We have also added functionality that allows customers to create their own TIF for any specific device or application they may have.

Once the new message is created, we have several options.  By default, we write the message to the local Event Log.  This allows the Management Pack to read the message and generate an Alert or clear an existing Alert.  Optionally, you can send the message over to a Syslog server if it is important to track the Trap message for other applications or security purposes.

Enter the Management Pack

Now that we have stored this information in the local Windows Event Log, we have the message in the perfect place for SCOM to use it.  The Management Pack is nothing but Monitors and nearly all of the Monitors do nothing but read the Event Log.

When the Monitor sees a message from the Trap Enhancer, it generates an Alert.  The Alert actually contains the Event Description, so you can read exactly what the Trap Enhancer generated in the Event Log without looking at the actual event on the server.

All Monitors are disabled by default, but we built an Override MP that enables a few Monitors, to help users get started: Monitors such as the SNMP Trap that gets sent when someone saves the configuration on a Cisco device or a notification of a network device that has been reloaded.

What About SNMP Notifications?

When an SNMP Trap is generated, there may no be a corresponding SNMP Trap to clear the state.  This is usually true for Notifications.  A notification is a message that tells you something happened, not necessarily some GOOD/BAD event.  An example of this is when there is a reload of a device.  A reload message is generated, but there is no corresponding message that will clear this message.  What typically happens in SCOM 2007 is an SNMP Trap is sent for the device and an Alert is generated.  The Alert must be manually cleared because there is no corresponding event you can use to clear the Alert.  In other words, the Alert gets generated, but it must be manually cleared.

To fix that in the Trap Enhancer Management Pack, what we have done is to use a timer to clear the Alert after a set amount of time.  These timers are individual for each SNMP Trap and can be overridden for each device.  Our defaults are for 24 hours, but this will usually be changed depending upon the SNMP Trap that is generated and the device it is generated from.

Performance:

I am sure a pressing question is: What kind of impact will this have on my server?  Well, we have performed some stress tests on the Trap Enhancer Service and the results shocked us.  The highest level we pushed the Trap Enhancer was about 30 Traps/Second.  This is an extremely high rate of messages and would probably only be found in very large networks, if ever.

In all cases, the most interesting finding we made was that we could push the Trap Enhancer to the brink of failure and we could make it fail.  What is interesting about it is that it actually wasn’t the Trap Enhancer Service that ended up failing.  It was the Windows SNMP Trap Service.  We were pounding the SNMP Trap Service so hard, it would crash and stop receiving trap messages.  All we would do is restart the Windows SNMP Trap Service and the Trap Enhancer Service would continue translating all incoming Trap messages.

What Lies Ahead

Even before the Trap Enhancer Service and Management Pack was finished, we started looking to the future and the enhancements we wanted to make.  Some enhancements on the horizon are:

• Make TIF file creation and integration cleaner and easier
• TIF design and testing tools
• Convert TIF file format to XML
• Build APIs into the Trap Enhancer Service
• Add Performance Counters to the Trap Enhancer Service
• Allow for non-TIF content to be used for translations
• And many more.

The Trap Enhancer Service + Management Pack was built with a very specific purpose in mind: to take the messy, hard to read, cumbersome information from our network and make it presentable, readable, understandable, and easy to use.  Even though the Trap Enhancer Service is a major component to the solution, the complete user experience would not be possible without System Center Operations Manager.  Now we can take this information and do something useful with it, like manage networks efficiently and effectively.

We are putting the finishing touches on our first Cisco Management Pack.  This is formally know as the SNMP Poller Management Pack.  It is a big step forward for us and, we feel, for the industry.

Our first Cisco Management pack is targeted to the Cisco Catalyst 2950 Series Switch.  Why a 2950? you ask.  Well, first and foremost, we had a spare one available.  It was a good test device and it allowed us to really experiment with many of the System Center Management Pack features.  We had to resolve some scalability problems and performance issues.  A small 12-port low-end switch was a perfect starting point for us.  It also allowed us to develop and test our Rules, Monitors, Discoveries, and many other components of the Management Pack.

We have tried to keep as much functionality as native in System Center as possible.  Unfortunately, there was only one area of the Management Pack that we were not able to do that.  SCOM understands SNMP polling, but not nearly to the level it should be.  To get around this, we have built a library that the Discoveries use when finding the device.  Once the Discovery has used the library, it is no longer used in any other part of the Management Pack.  This means all Rules and Monitors are all native to SCOM and all SNMP Polling is handled within the SCOM environment.  This was a primary goal from the beginning; to bring as much of the Management Pack functionality into SCOM as possible.  We have successfully done that.  We are not using WMI or some other application to configure and poll the devices and then dump the information into SCOM through some convoluted method.  We are NATIVE.

Another design goal was to modularize many of the pieces of the Management Pack in order to easily generate new Management Packs for other devices.  As a results, it took many man hours to get this first Management Pack done, but, as you will see, we will start to generate a new Management Pack every few weeks.  This is huge for subscribers.  Subscribers will have ongoing access to all new Management Packs developed as well as any Support updates.

One last thing about the new Management Packs.  We have spent a considerable amount of time building and integrating as much knowledge into the Management Packs as possible.  Even so, it is still not enough.  We have added links from the MP Knowledge to our web site for additional information about a specific Alert or Rule.  We feel like you can't have too much information when you are trying to figure out a problem.

We are excited about the upcoming release of the new Management Packs.  We hope you will be satisfied with them.

As we wrap up the final testing of the SCOM Trap Enhancer, development has already progressed to our next product, tentatively called: SNMP Poller Management Pack.

Since this site is more about System Center Operations Manager (SCOM) and not necessarily specific product development, out focus going forward is going to more driven towards Management Pack development for SCOM. Actually, our first product, the SCOM2007 Trap Enhancer, actually comes with a Management Pack that can be installed on a SCOM system to help identify and alert on network, device, and application events.

Currently, we have one Product + Management Pack completed. Our next product will be specific Management Packs for specific vendor devices. We are starting with Cisco devices and we will publish a seperate Management Pack for each Cisco device. Each device is unique and needs to be treated differently from other devices. For instance, it would be foolish to think that the SNMP counters you find on a Cisco 7206 router are all going to be on the Cisco 1801 router. If you actually to a complete SNMP Walk of the devices, like we have, and compare them, the difference is very obvious. One size DOES NOT fit all, in this respect. This is why we were not satisfied with the current batch of SNMP polling software that "integrates" with SCOM. I use quotes around the word "integrates" because these vendors have shoehorned their software to fit into the SCOM Management Framework. See below how we are avoiding pitfalls like this and fully integrating into SCOM with our Management Packs.

Why did we need a "Trap Enhancer"?

If you have ever received an SNMP Trap from a device, you would understand why something needed to be done. We originally enabled our Ops Manager to receive SNMP trap and we configured our internal devices to initially send SNMP Trap information. What we found is that when Ops Manager receives the SNMP Trap, it is filled, literally, with OIDs and numbers. No text whatsoever is sent with the Trap. It is up to the NMS Manager to decipher the information in the Trap by using the corresponding MIB. We took the direction that it was more time than it was worth to try and "rewrite" the Trap into human readable information.

We first looked around to try and find something that could translate these Traps for us. Nothing was available. After a little bit of research, we decided that we could build something simple that could get the job done. The ultimate goal was not to build an application, but to build a Management Pack that could help us not only be alerted to issues, but for us to be able to translate what it means without spending hours researching.

That is how the SCOM Trap Enhancer was born. We reference the actual MIB to help us translate the descriptions and we literally did a cut-and-paste from the document. So the answer to the above question is: you really don't need a Trap Enhancer, but it sure makes your life MUCH, MUCH easier and will save you plenty of time.

What does the future look like for the SCOM Trap Enhancer?

We will be adding more vendors to the TIF library. Currently, we have the common Standard RFC MIBs and Cisco MIBs. As new SNMP Traps come out from these vendors, we will add them to the library for download by Premium Subscribers.

We also plan to add our own EventLog and our own Events with an improved error handler.

There is not many more major improvements that I can see us doing to the Trap Enhancer Service. We will be publishing a Roadmap in the Premium Content area of the web site.

SNMP Poller Managent Pack

This Management Pack (MP) will revolutionize the SCOM Management Pack world as it pertains to SNMP Polling of non-Windows devices. We are building a very unique framework that allows us to completely integrate SNMP Polling of all types of OIDs into SCOM. The real advantage is that we are not building a seperate tool or interface for you to configure your SNMP Polling environment, we are leveraging SCOM and it's Management Framework. All configuration of SNMP Polling of devices will be done through SCOM's Management Console.

But SCOM already supports SNMP Polling of devices, you say. Yes, it does. The dirty little secret is that it only supports individual, or non-Indexed OIDs. This means that you COULD monitor an interface on a router, but you MUST know the Index value of that interface as it related to SNMP and if you add another interface, your setting quite possibly could change, rendering your current SNMP configuration wrong. SCOM does SNMP Polling, but not on Indexed values, which is what most of the counters on a Cisco device consist of.

Current products require you to install a "management application" that is used to discover and configure the approriate Indexed values. We are integrating this into SCOM, so you DO NOT need an additional external application to administer your SCOM SNMP environment.

Our SNMP Poller Management Pack will be specific to the model of the device. If there are some extra counters, and we are including as many as we can find, you can configure them through the SCOM Console and set thresholds within SCOM to allow for Alerts.

We are just scratching the surface of possibilities of the different types of Management Packs that can be built. We are not going to stop at just hardware specific MPs. For instance, we have on our Roadmap of MPs to be developed the following:

QoS MP - This will include not only Router/Switch policy information, but also leverage EEM and IPSLA(within Cisco devices) to deliver very specific and exact results

VPN MP - This will detect Lan-to-Lan VPNs and provide performance and event monitoring on specific VPN Tunnels

Vertical Industry MP - We will start to focus on vertical industries such as Healthcare to provide application specific monitoring

We are only starting to explore the potential of Network Management and SCOM. We will continue to innovate and lead by example and drive technology forward.