"There are too many alarms. Can we turn the platform off first?" That sentence is not rare on equipment-room and power and environment monitoring projects. On-duty staff receive dozens or hundreds of notifications in a day. Slight temperature and humidity swings, equipment state changes, communication jitter, and real faults are mixed together. Over time, people develop alarm fatigue: the prompts cannot all be read, phone calls are not wanted, and a severe event can also be taken as one more false alarm.
From the years I have spent on power and environment projects and on-site operations, my experience is this: when alarms flood in, do not rush to reject the platform, and do not simply raise the thresholds. Many problems come from thresholds that are not graded, measurement-point data that is not reliable, alarms that are not deduplicated, and notification methods that are not designed around the time available for handling. The goal of governance is also not to push the alarm count to the minimum. It is to have the alarms that need a person to act reach the right person in time, while keeping enough information for later tracing.
Many alarms do not necessarily mean the platform is useless
A power and environment platform is responsible for acquisition, judgment, and notification. Whether an alarm is useful depends on the whole chain before and after: whether the measurement point is installed reasonably, whether the equipment data is trustworthy, whether the threshold matches the on-site operating condition, whether the alarm lasts long enough to be worth handling, whether the notification finds the person on duty, and finally whether there is confirmation and closure.
For example, a temperature and humidity probe stuck against a supply-air outlet may see readings change quickly with the airflow. An unstable contact on a door magnetic switch or a leak-detection rope can produce a momentary jump. A short break in the communication link can also show a batch of devices as offline at the same time. If data quality and the on-site installation are not checked first, and only the thresholds are adjusted, the alarms on the surface may be held down while a real blind spot is left behind.
So when I take over a site where "there are too many alarms," I usually look at three things first: whether the alarm corresponds to a clear risk or action; whether the same event has been notified more than once; and whether on-duty staff, after receiving it, can judge what to do first. Alarms that cannot answer these three things need to be sorted out again.
Thresholds should be graded by risk and by the time available for handling
Threshold configuration is not filling in one upper limit and one lower limit for every measurement point. Different objects have different risks, and the handling time also differs. Alarms can first be split into three grades—urgent, important, and advisory—and then the trigger condition, notification target, and response requirement defined for each.
| Grade | Where it applies | Common way of handling |
|---|---|---|
| Urgent | May immediately affect personal safety, equipment, or business continuity, and needs confirmation and handling as soon as possible | Phone, SMS, or a strong on-duty alert; notify the person on duty, and escalate when necessary |
| Important | A clear anomaly exists, but investigation and handling are allowed within a set time limit | Generate a work order or put it on the operations to-do list, and record the owner and the completion time limit |
| Advisory | A slight deviation, a trend change, or information that does not need immediate intervention | Roll it into daytime inspection, the daily report, or trend analysis |
A slight temperature or humidity excursion does not necessarily require a phone call to everyone. Based on the equipment-room design requirements and the on-site operating strategy, it can first enter the advisory or important grade, and be judged together with duration, the trend of change, and the situation in the area. Abnormal key states such as UPS battery, output, and bypass, a water-leak alarm, a distribution trip, or loss of power on a critical circuit usually need a higher priority configured according to the actual impact. The specific grading must be based on equipment capability, design requirements, and the on-site emergency plan. Numbers from another equipment room cannot be copied over.
Pay particular attention to the threshold unit, the sampling period, the direction of the upper and lower limits, and the measurement-point location. The same temperature number can mean different things in different areas, on different equipment, and under different operating strategies. At project handover, record the setpoint, the delay time, the deadband, the notification rules, and the owner together, so that what is left is not only an isolated threshold table.
False alarms and missed alarms often come from outside the threshold
A false alarm is the system indicating an anomaly when, after an on-site check, there is no corresponding fault or need to handle it. Common sources include sensor accuracy or installation problems, communication jitter, momentary spikes, thresholds that are too narrow, alarm delay set incorrectly, and no alarm inhibit or maintenance mark during equipment servicing.
A missed alarm is more dangerous: a real anomaly does not generate an alarm, or an alarm is generated but is not delivered and is not seen. The cause may be a threshold set too wide, a measurement point that is missing or inaccurate, an acquisition channel that is interrupted without a communication-fault alarm, a wrong notification target, or night escalation rules that do not cover the case. There is another situation that is easy to overlook: the alarm has already been suppressed, but the suppression logic also presses down a new and more severe state along with it.
When handling false alarms, do not only ask "should the threshold be raised." Also confirm the raw data at the time of the alarm, the duration, the equipment status, the environmental change, and the personnel actions. When handling missed alarms, verify the anomaly path in reverse: from the sensor to the gateway, the platform rules, the message channel, and on-duty confirmation, whether each segment can work. When needed, verify the path with a controlled alarm test. The test must be carried out according to on-site safety procedures, and the test status must be stated clearly, so that it does not create a real equipment risk or disturb the watch.
Alarm suppression should keep the root cause and the relationships
When the same point repeatedly crosses the limit in a short time, it is usually not necessary to place a new phone call every time. A repeat-alarm merge or a suppression window can be set: the first alarm notifies immediately, and within the window the duration, the highest and lowest values, and the state changes are updated. After recovery, record the recovery time. If it triggers again, treat it as a new event. How long the window is has to be set according to the alarm type and the risk. It cannot be applied uniformly to every point.
There can also be a causal relationship between upstream and downstream equipment. For example, after an air conditioner stops, equipment-room temperature rises step by step, and the platform may receive several alarms at the same time, such as an air-conditioner fault and temperature over limit. Alarm display and notification can highlight the possible root cause and its related alarms, and reduce repeated interruption. But a relationship does not mean the consequence alarm can be deleted: temperature over limit is still an independent risk signal. It should be kept in the event record, and whether to notify it separately should be decided by severity.
States such as a communication outage, an equipment stop, and maintenance operations also have to be designed separately. A communication fault should not be disguised as the equipment being normal. A maintenance inhibit should have a start time and an end time, an operator, and a reason. Urgent-grade alarms, in principle, cannot be held down indefinitely by a general deduplication rule. Every suppression rule must be able to explain "what was suppressed, why it was suppressed, when it recovers, and how it is audited."
From Tongwei's years of experience, judging alarm root causes, correlation relationships, containment relationships, and the like requires both a deep understanding of the characteristics of the industry and of the business, and a sorting-out of software and hardware data flows, synchronization of the business flow, and multi-mode adaptation of alarm rules, before alarm suppression can be done well: alarms made fewer, and alarms not missed. Of course, the rapid development of edge-side AI will provide important assistance in solving this.
Use a one-week trial run to verify alarm quality
After thresholds and the notification policy are adjusted, pick one site or one class of equipment and trial-run it for one week first, and review it together with the customer, operations, and on-duty staff. The trial run is not only a look at whether the alarm count has fallen. It checks whether alarms are more able to be handled, and whether important events can still be found in time.
The following metrics can be tracked:
- Valid-alarm rate: the share, among all alarms, of alarms that after checking truly need handling or follow-up. The statistical definition should be agreed in advance. Advisory information can be counted separately.
- Repeat-alarm share: the share of the same event and the same point that is generated again or notified again within the set window.
- Confirmation and handling time: the time from alarm generation to on-duty confirmation, the start of handling, recovery, or closure.
- Missed-alarm and late-alarm records: anomalies found in inspection, work orders, or fault review that the platform did not generate or deliver in time.
- Unconfirmed alarms and escalation: whether an owner did not confirm, an escalation rule did not trigger, or a message channel failed.
Before the trial run starts, keep the original rules and the historical samples, and agree the target and the escalation method category by category. During the run, spot-check high-grade events and suppressed events every day. After it ends, adjust the thresholds, the duration, the notification targets, and the suppression window. The valid-alarm rate can be used as a communication metric, but it cannot by itself represent the safety level. If the alarm count falls while missed alarms rise, the policy has failed.
Alarm governance is not a one-time "parameter tweak." Equipment replacement, seasonal change, load adjustment, equipment-room renovation, and a change in the duty system can all make the original rules no longer suitable. Important alarms need to be reviewed regularly, and rules need an owner and a version record, so the system can keep up with changes on site.
First turn alarms into information that can be acted on
When you meet a case of "too many alarms, and the duty staff do not want to look," you can proceed in this order: first verify the measurement points and acquisition quality, then split urgent, important, and advisory by risk; next set duration, deadband, and repeat merge for the same point; sort out equipment relationships, show the possible root cause, and keep the related risk signals; finally state who receives the phone call, the SMS, the work order, and the daily report, how soon each is confirmed, and how escalation works when there is no response.
You can first agree a one-week trial run with the customer, and evaluate it jointly by valid-alarm rate, repeat-alarm share, confirmation time, and missed-alarm records. Compared with turning the platform off directly, this is better at finding where the alarm noise comes from, and at verifying whether critical alarms are still reliable. The value of the platform is not in calling out every fluctuation. It is in having on-duty staff receive, at the right time, information sufficient for them to take the right action.