It used to be that good preventive maintenance meant walking around, looking, listening, and generally being a good observer. Attention to detail was important. It was the sounds, and sometimes the smell that told you something was wrong. Writing down meter readings was a normal event. And while some of that remains, most of today’s preventive maintenance can be done at your desk. With a little scripting it can be sent to your email; allowing it to be done easily from almost anywhere.

Logs, SNMP traps, and a variety of other items need to be considered when doing good PM today. Not all SNMP traps end up in the logs. Notice I said logs, plural. Most systems have several logs that are for different subsystems. One log may contain entries that are specific to the hardware, while another may be for the operating system (OS). Some logs relate directly to the application. With computer-based systems you really do need to dig as deep as possible; and on a regular basis. For instance, a quick check of Windows 7 shows log categories that include Application, Security, Setup, System and Services. All of these can show problems that relate to basic system health. It is likely that most of these categories are not looked at until too late. Most groups that check logs may only look at specific logs that relate directly to their area of expertise. Yes, problems with the system will eventually find their way in to the application logs, but many times these problems can be caught much earlier when all logs are regularly scanned for issues.

A good example of watching a problem develop can be seen on most digital receivers. RF paths are unreliable, forward error correction (FEC) is used to correct errors that occur through the transmission path. At any given time there will be some number of errors, and (hopefully) all, or most, will be corrected using the FEC. Tracking those numbers over time can show degradation in the antenna and receiver. The same is true in a computer NIC, hard drive system or memory. There will be some errors that are corrected, and over time that number may grow. However, when the number of errors increases, either sharply, or over a predetermined threshold, it is time to find the source of the errors and correct them. The only way to find those gradual increases is by tracking the numbers over time, just as we did with meter readings.

Digital systems can easily lull you into complacency. Things work great…until they don’t. Then what? The truth is the system probably left plenty of clues, many of which were likely missed because no one was looking. There are tricks that can be used; management systems can gather and report log errors, scripts can search log files and their contents. Even something as simple as the size of the log file can be an indicator of developing problems. If the files are normally 2K, a 4K file might be a warning sign that needs closer examination.

Another thing to be aware of is the various thresholds employed by the manufacturers. Manufacturers might categorize events as warnings, minor errors and major errors. A lack of major errors does not mean all is well. Look at minor errors and warnings. Ask the manufacturer if logged events other than errors or warnings should be tracked. If you don’t understand the events in the logs, ask about them.

Manufacturers are usually open to sharing that information with their customers. Throughout the software development cycle, all sorts of items are written to the logs as part of the debugging process. Many of the logged events detail how the system is really functioning, much like the margin numbers on digital receivers.

Today’s systems have a remarkable and somewhat uncanny ability to adapt. While that often makes our jobs easier, it can also mask an incredible number of problems. Some of those problems will come back and bite. Some won’t. Like all PM programs the question is when do you want to find out? On your terms? Or after the system is down and it’s too late? Personally, I would rather find out during the day instead of receiving a frantic phone call at 4AM.

Lately, there have been several instances of confusion on the part of management that automated operations can be done without personnel, or, even worse, done with relatively unskilled personnel. There is this thought that the knowledge, skill and experience of top technical people can be saved in a knowledge base and then given to unskilled personnel and everything will be fine. Not true.

A knowledge base or playbook that details do ‘X’ when you see ‘Y’ is fine for common events and fixes that are known to have little effect on the rest of the system. However, in today’s highly integrated systems, fixing one problem can easily cause another if the fix is not handled properly or done in the right order. As an example, you could easily have a server in a network load balance that needs to be restarted due to an issue. Simply going to the start button and restarting it does not remove the server from the load balance, nor does it accommodate the current user sessions properly. Allowing the system to shed current sessions (and not starting new ones) before removing it from a load balance takes care of any users currently on that system. It also allows the other computers in that load balance to take on additional load properly. This is also true of devices connected to storage area networks (SANs). Proper task sequencing is crucial to maintaining the integrity of the files and systems that are connected.

Often, automation is used to accomplish complex tasks that are simply tedious or repetitive. Due to the complexity of the operation, things can go wrong on one pass that will work fine on the next. An acquaintance in the software industry is fond of saying ‘there are no gremlins’ and he is right, but in some scripts or automated sequences it is not worth the time to find the occasional random issues – it is easier to re-run the script and have it succeed on the second pass. But sometimes – and this is where skill and experience are important; things go wrong that cannot be solved by simply re-running the script. When this happens, having junior level personnel at the helm can be problematic. Do they know enough to correct the issue without breaking something else? Are they confident enough in their skills to know when to call for help?

Documentation is critical in today’s highly integrated systems. Having a knowledgebase that can be used and updated by all personnel is also critical. The key is making it meaningful. If all entries have ‘escalate to engineering’ as step 1 once the problem is identified, that is not much help. Nor, in the hands of junior personnel is a complex task sequence that should only be performed by knowledgeable maintenance personnel during a maintenance window.

Today’s automation allows a small group skilled individuals to accomplish things that could not have been done by an army 10 or 20 years ago. At the same time, it still requires someone to watch over it, and step in when things go wrong. Whenever large-scale, automated operations are running, the potential exists for any one of many sequences to fail. When that happens, things can snowball very quickly. Do you have the processes and personnel in place to stop that snowball effect and get things back without significantly impacting your business, operation or customers? If not, now is the time to address it – before that snowball starts rolling.

For the last several years I have been involved with specifying and purchasing a variety of systems. These normally start off with a salesperson offering up various solutions that might fit. As the system gets closer to reality, a sales engineer gets involved, and the discussion usually gets far more technical. Often, there is considerable give and take between what the sales person thought was needed and what the sales engineer specifies. None of this is really a surprise, the job of a sales person is to sell. Technical knowledge, while helpful, is second to salesmanship. Sales engineers, however, should have that technical knowledge and have a detailed understanding of what their company’s equipment will and will not do. Additionally, it is in the sales engineer’s best interest to understand the customer’s requirements and installation as well as possible.

Often the challenge appears when the field engineer arrives. The challenge comes out of the blue with a statement like “our management network has to be on its own subnet,” or “you can’t have both GigE ports on the same network,” or “even though the NIC is a GigE, it is limited to only 50Mbit/s ingress.” While each of these requirements may be reasonable within the equipment’s design, none should be hidden until installation. If a company is selling a network-capable device, it should act like any other network-capable device. If it has two NICs and both can run with a static IP, there should be no limit as to the addresses assigned. If there are requirements, they should be clearly spelled out on literature and through the sales process.

When working with multiplexed streams over networks, it is very easy to get caught by these items as technology changes. For instance, HD MPEG-2 was commonly distributed at 15Mbits/sec. At that time, 100Mbit/s NICs were also common. It was easy to put one stream on the NIC, and the designers may have limited the NIC drivers to 6 streams. This is because more than 6 streams at 15Mb/s would potentially swamp the NIC. Enter MPEG-4 at 5Mbits/sec, now you can easily playout 15 streams and not swamp the NIC (only 75Mbits/s). However, if that (long forgotten) driver code is still in place, the limit is 6. Gotcha.

With technology moving as rapidly as it is, items such as these become more common. Sometimes, it is as simple as a code upgrade. Sometimes it’s not. Code upgrades can take considerable time and that is not always part of your project’s timeline. Workarounds are not really how you want to implement new technology. Speaking from experience, code upgrades do not always correct the problem—and sometimes the new code comes with bigger issues than the old.

One solution is a detailed installation plan prior to purchase. In it, each and every IP address, connection and configuration setting is discussed, primarily with both a field engineer and the sales engineer. That way, any items that conflict with known implementation issues can be identified up front. Once identified, they can be resolved, or the purchase can be delayed, or potentially, other equipment can be considered for the project. Ultimately, all of that will information can be detailed as part of the purchase so that both sides fully understand what they are agreeing to. In the unfortunate event that it prevents a sale, that information gives the sales department the ammunition they need to go back to the technical side and get the situation corrected quickly.

I was once told that you can’t improve something, unless you can measure it. While I did not fully understand it at the time, the concept has stuck with me. Every time I am asked about improvements to a system, I think about ways to measure whatever needs ‘improving.’ Often that touches off a lengthy discussion about exactly what would be considered an improvement.

Part of measuring something is documenting the measurement and storing it for later reference. Unfortunately, the tool of choice is often a spreadsheet. Why is that? The best answer I have come up with, is that spreadsheets are easy, and databases are hard. Nearly anyone with computer skills can open up and fill out a spreadsheet. Databases require thought, sometime lots of thought. But what are we storing? Lots of data points. Data points belong in a data base (database). Period.

The problem with spreadsheets is that they are like Post-It notes. They just multiply. Pretty soon they are everywhere. Everyone has their own. All are similar but none actually have any relational hooks, and they are all out of sync. Often there are multiple copies of the same sheet and there is no way to merge them. Every system I know is made up of many data points. These might be meter readings, IP addresses, cabling connections, SNMP alarms, tuning parameters, and the list goes on. For your own sanity–Stop the madness. Learn how to use and build a database, or hire someone to do it. Insist that spreadsheets are moving into a common database that is regularly backed up.

A good database to start with is Microsoft’s Access. It is the one I learned on and the tutorials are excellent. Microsoft SQL is huge, but a much better long term choice, especially if you have an all Microsoft shop. My favorite is MySQL. It interfaces well with both PHP and Apache, it is easy to use from the command line, and GUI tools are available open source. It is free for non-commercial applications, but you should always verify that any software is being used within its license terms. The combination of MySQL, PHP and Apache running on a Linux server is an inexpensive and powerful addition to any shop.

First some terminology: A single sheet in a spreadsheet (commonly called a flat file) is the equivalent of a table in a database. Tables are a basic building block within a database. As a start, the next time you want to open up a new spreadsheet, create a table in a database instead. To be more specific, we are talking about a relational database where individual tables can have defined relationships between each other. Often this is done with a key, a unique value that is in two related tables.

Keys can be text, but are often numeric values. The relationships might be one-to-one, or one-to-many. A one-to-one relationship might be defined between two tables; one table containing names of employees, and another containing cubicle or office information. That way if someone changes offices, all that needs to be changed is the key that relates the two tables. One-to-many relationships are more common. One table might contain names, with another table containing phone numbers. One person could have several phone numbers including office, home, mobile etc.

Even if there are only a few relationships, databases can come in handy as a central location for numerous tables (spreadsheets). Record locking can prevent two people from changing the same record at the same time, and the database file set can be easily backed up as it is in a single location. The more you need to track, the better databases are at tracking it. Even better is the ability to query your data. Queries are the other basic building block of a database. Queries allow you to select, insert and update your data. In the end, with just a little extra effort, you will find that a relational database offers all the ease of use of any spreadsheet, and a whole lot more. Give it a try, and you will be rewarded for your effort.

When I started in broadcast, the majority of the station’s equipment was old. Some was five years old, some ten, but there was plenty of 15-20 year old equipment. At that time, anything less than 5 years old was considered “new.” Nothing contained microprocessors, but ICs were common, transistors more so, and tubes (besides those in the transmitter) were not completely out of the picture. Station management had long-since paid for and depreciated the gear. And, they were still seeing a return on that investment.

At that time, the challenge was to keep the gear operating at a reasonable cost throughout its useful life. Today, things are different. Most devices seem to get through their useful life without a problem. They are still fully functional, but no longer useful. They require little or no maintenance and repair, but are useless (and worthless) long before they quit working. The challenge today is to get the gear in, up and running, and paid for before it becomes functionally worthless. Sometimes that life span is measured in months rather than years.

Several considerations weigh heavily on how long you can expect something to be useful; proprietary systems cost more and typically are functionally useful longer than commodity systems which usually cost less. Many ‘cutting edge’ systems are based on a combination of high end server hardware running proprietary software. When that is the case, there is a possibility that upon replacement, the servers can be put to some use. That gets you a little extra life from the hardware, but does nothing for the software which can be the greater part of the cost.

Even getting extra life out of the hardware is a gamble. Will the system internals need to upgraded for the system to be useful in its new role? Is there time and personnel available to adapt the old systems, or is it more cost-effective to purchase the correct hardware for the job? With the constant change, often it is easier and cheaper to buy new. A quick test is to look at whether or not you have been able to utilize hardware you have recently removed from service. If you don’t have the time to adapt your current old servers and systems, unless you anticipate major changes, it is unlikely you will have the time or desire to re-purpose the next round of out-of-service systems.

There is little doubt that purchasing new systems can open doors to new or additional revenue streams. However, in light of the pace of technological change, make sure you include reasonable estimates of how long you will be able to keep that new system useful and in use.