Monday, August 8, 2016

Alarm Management Needs Error Management

Does your system contain alarms that cannot actually ever occur?  Many do.

Alarm management focuses on understanding and controlling the alarms that can be generated in a system.  DBDOC provides techniques for walking through the configuration systematically to validate assumptions and look for problems. Effective alarm management is impossible without error management and configuration analysis.

Here are three example sets of alarms that do not work in a single powerplant configuration:
  1. Intermittent failure caused by duplicated exception report imports
  2. Detected failures to test blocks with quality
  3. Problems that can be found by using DBDOC to walk through the configuration easily
If you do not manage your errors and use the tools DBDOC provides to audit the configuration, you will not be able to manage alarms. Even worse, you can spend a great deal of money fooling yourself about alarms.

Example 1 - Frozen Import Block Error

INFI 90 allows block values from one module to generate "exception reports" when the block value changes status or value. In the case of an analog exception report, the new value is transmitted when it changes by an amount that is specified as "significant".

If a an exception report block value is imported more than once into a module, only one of the import blocks works. The values of any other blocks importing the exception report block value will remain frozen. Furthermore, when an on-line configuration is done, a second import block will silently begin working (!) and the block that was working will freeze at its last value. This applies to both analog and digital exception reports.

DBDOC detects and reports this error. However, when the module is compiled using Composer, the error is not detected.

      Error: Frozen import block [093]

      Duplicate off-module reference (DI/L, DI/L) to Module 1,01,05 Block 1230: 1020324 and
      1020342 [1010536]

The source value above is imported in two places in one module. This instance on a spare block page might have the active connection.

The following shows the intended logic.

There is a 50/50 chance that the correct logic is not working at any time. Simply put, tags 0D0730 "COND RETURN TANK HIGH/LOW LEVEL" and 0D0727 "AUX DEA HIGH/LOW LEVEL" might alarm, but they might not. It is a crap-shoot!

Note: It would seem reasonable to assume that AI/L block 1675, now spare, actually started life feeding S3 of block 1676. The duplicate import situation was discovered and the logic changed properly to get both values of S3 from a single AI/L block somewhere in the past. The problem was that the spare block still can be the active one.

Example 2 - Test Quality of a block that does not have quality to test

Powerplant A has 7707 TSTQ blocks. These four input blocks give a binary [1] as a result if one or more of their inputs have bad quality. Actually, 8592 blocks are being tested in all. Of these, 316 do not have quality to test at all.

Here is one example of an alarm (and shutdown) that will not work, identified by the message here.

       Error: TSTQ tests block with no quality [063]
       TSTQ Module 4,05,03 Block 2273 tests Module 4,05,03 Block 6170 (FC 15), which does not
       have quality [4050378]

Obviously, the simple solution to the problem of no alarm on bad quality for 1 GROSS MW TRANSDUCER A is simply to test the quality of the input at IREF P1E0388A instead of the output of the fudge factor sum block 6170.

Example 3 - Misconfiguration of a DAANG Block

DBDOC also allows you to go looking for trouble. Walking through all instances of a Function Code will allow you to look for things that were missed long ago. This example shows bad quality that will not alarm.

DAANG blocks are powerful tools when configured competently. Powerplant A has 6227 of them, the second highest number we have seen in a single plant. I was curious to see how consistently programmed they were. The table shows how a value places on Specification S8 can set bad quality.
This plant uses this extensively. This example shows how the constant 32.0 is applied to S8 with the green highlights. When the raw signal is found by the TSTQ block to be bad quality, the DAANG block will be marked bad quality as intended.
The following example shows that undetected errors exist. The red highlights show that bad quality on this raw input will result in good quality being forced. Oops!
Being interested, I started walking through the 6227 DAANG blocks looking for this particular issue.  I got bored after 200, taking 16 minutes, about 5 seconds per block. I found (and flagged with DBDOC annotations) the following reality:
  • 124 blocks with correct alarm configuration - setting 32.0 into S8
  • 22 blocks incorrectly setting 1.0 into S8
  • 1 block incorrectly setting 0.0 into S8
  • 53 of the first 200 DAANG blocks did not set the quality attribute in this way.

The bottom line is that 12% of the first 200 DAANG blocks (all tagged, by the way) would not alarm on bad quality. I have no reason to think this is not representative, which suggests that perhaps 800 of these tags will not alarm on bad quality in this one system.

By contrast, I checked a system with 7980 DAANG blocks. There were no alarm errors in the first 150 instances, with all quality inputs being generated correctly. At least in this respect, good configuration work would make alarm analysis valid.

In a second system with 5084 DAANG blocks, I found that only 2042 of them used the S8 to set bad quality if needed. In fact, this simple walkthrough with DBDOC showed that that 60% of the blocks I sampled were fed from signals that had no quality to test, but did not reinsert bad quality. That could mean about 2000 simple errors - tag does not show bad quality when it should.

Error and Configuration Management Trump Alarm Management

Alarms cannot be managed if errors prevent them from working. DBDOC can be used to improve alarms both by looking at errors and by walking through the project carefully. Otherwise, any alarm analysis will be incomplete and unacceptable.

No comments:

Post a Comment