Has Recent Work Introduced Errors?

You may have read this analysis of the errors DBDOC found that had been introduced into a client system over four years.

Here is another summary analysis of the important errors introduced into a different system over five years. DBDOC makes this sort of analysis trivial if you have backups of systems before and after work is done. Here are questions that should be asked:

1. How many errors that could hurt the plant are acceptable?
2. Why were these errors not in front of the DCS team for evaluation?
3. How could these errors, all breaking the rules, yield a clean Composer compile?
4. Should Composer have caught them?

Duplicate ADAPT Blocks and Multiple TSTALM Blocks

There are ten errors that result from the failure of Composer to force a cloned ADAPT or TSTALM block to have S2 or S1 set to -1 when the block is cloned.

Two of these errors are shown below. It is absolutely clear that S2 of block 1060 and S1 of block 1066 both should be 1062, not 2450. The value 2450 is residual from the working logic that was cloned into this logic. This is typical of a class of error made again and again by Composer users.

The block index for the prototype MSDRVR FC 129 Block 1062, centre of a wonderful set of beautiful functioning logic. The highlights show the four extra TSTALM block errors, so there are six of each. Not one of the errors was found when the work was done, compiled by Composer or loaded by the user.

Some Function Blocks are Defaulted as Disabled

This error message shows that one lag function was not enabled when it was added. The effect would show up as inexpicable unintended excursions that are supposed to be smoothed out. Clearly, the default should be to be enabled, not disabled.

Specification out of Range

What does this block do?

We could guess. Why should we have to? Why is this not an error in Composer, as the only legal values are 0, 1 or 2?

TSTQ Tests Block With No Quality

The work being analyzed has only two errors where a block is being tested for quality when it does not have quality to test. Only two!

First, note how the blocks with quality, shown by DBDOC as blue lines, lose that attribute when they go through logic blocks. Here, the cost of the one second lag function is the loss of quality for the pressure signal.

So, bad quality on the pressure signal will not be detected. What is it supposed to do? Simply put the FLOW CONTROL STATION into Manual. Oops!

Summary

There are fourteen potentially significant errors documented here, with many more less severe not mentioned. They should not have been made, and they should have been caught.

A System Without DBDOC - Analysis I

This is the first of a series summarizing the severe errors found by DBDOC at INFI 90 sites that do not have our product for system integrity protection. Be assured that there are more errors in this system than the ones shown here, but this article highlights some of the glaring ones. The system has 5723 configuration sheets at a large power plant. This site gets no warning from its existing tools about the errors described below!  DBDOC, however, would be able to alert it to all of them.

Adapt Block Cloning Errors

There are two adapt block cloning errors, caused by failure to change S2 to match cloned logic. Two subsystems are not adapted, even though they look like they are.

[S2 of block 6115 should be 6116, not 5116]        [S2 of block 7814 should be 7815, not 1115]

TSTALM Cloning Errors

There are nine multiple TSTALM situations. Every one of these comes from S1 not being changed when logic was cloned. Fourteen actions can happen mysteriously when they should not, triggered by the prototype logic instead of the new logic. Even worse, fourteen actions do not happen when they should, because the cloned logic is not being tested.

[S1 of block 3130 should be 3122, not 2920]

Why fourteen? One instance was cloned badly six times.

TSTQ Block Inputs Without Quality to Test

There are 209 inputs to TSTQ blocks being tested that do not have quality to test. In addition, quality is being tested on 10 AO/L blocks that do not have quality on their input signal. Three of these flawed tests control the manual interlock of control loops, so the loop will not go into Manual when the quality is bad.

[Analog Transfer block 4307 FC 9 does not have quality to test]

It is disconcerting that a number of the errors flagged look like OR and TSTALM block outputs are being handled incorrectly by being inputs to TSTQ blocks, instead of being OR'd with the outputs.

REDAI Blocks Used Without Being Tested for Quality

There are 46 instances where REDAI blocks can synthesize bad quality, but the value is then used whether or not it is bad. The first one I checked can be bad quality but still control a "FD AIR MASTER" control loop.

The one shown here is kind of cute, if you like scary productions. Good engineering was undone.

[Analog Transfer block 1835 FC 9 does not have quality to impart to the AO/L block and tag]

Function Block Errors

Among the function generator blocks, there are

• four that change from slope 1:3 to 1:1 instead of clamping
• four that change from slope 4:1 to 1:1 instead of clamping
• two that change from slope 7:1 to 1:1 instead of clamping
• three that with slope 1:1 that also do not clamp

[The large equal value (X,Y) pairs mean that the output rises at the same slope as the input]

Too Few Inputs
One 8 input Qualified OR block has four inputs, but needs seven to trigger it. It ain't gonna happen!

[Qualified OR block 7273 FC 36 will be a 1 if S9 (value 7) of the 4 inputs are value "1"]

Rung Block Errors

Two rung blocks ignore an input without documentation. If a test is in progress, too bad!

[S3 value 20 of FC 111 block 5026 means that value of "TTY TEST IN PROGRESS" is wiped out]

One rung block tests vibration limits, but does not disable the result on bad quality, as designed.

[S6 value 0 of FC 111 block 2606 means that value of  the test quality block is ignored]

Other Errors

There are 29 DAANG and AO/L blocks giving continuous exception reports.

There are 15 DAANG block tags that are designed to show some or all of 2nd and 3rd high and low alarms that will not do that.

One MA Station has type 5, not defined. Another has its initial mode after startup value set to 0, also not defined.

Summary

339 errors are outlined above. This is 339 errors more than the site knows about. What risks do these errors represent to the site? Why do these errors exist undetected? Should this system even be running?

System Integrity — DBDOC Detects Bad TSTALM Blocks

DBDOC is unique in giving you significant help in finding integrity issues in your system. Every INFI 90 system in the world will benefit from improving safety, reliability and operation with DBDOC's Integrity Plus approach. Our Error Browser and Error Marker development was designed to make it possible to manage errors and improve the integrity of your systems.

 

In August 2014, we showed examples of detectable TSTALM errors from one system, but we continue to find this error being made. We have been reporting it for a decade, since 2006. DBDOC is the only valid way to identify this problem.

 

We report Function Code 69 TSTALM blocks that test the same block and mode. This usually happens when a TSTALM block is copied without S1 the specification that tells what block is being tested being made to match the new block numbers. Any of these TSTALM problems could have an adverse affect on a plant.

 

I did a review of test data for 228 systems around the world:

• 22 had no TSTALM blocks - basically small systems averaging 1219 sheets each.
• 206 systems had 3 or more - average size was 4022 sheets.
• 90 had from one to 111 multiple TSTALM messages - average 8.2 errors.

The three systems with the largest number of messages looked like this:

• 111 errors - 114 actions missed, 114 actions falsely triggered
• 100 errors - 10 actions missed, 10 actions falsely triggered, 96 duplications
• 84 errors - 84 actions missed, 84 actions falsely triggered

 

Holy Crossfire, Batman! This is from http://holysmokesbatman.com/directory.

 

The problem is that good logic gets cloned to do the same job again and again. The block numbers get changed and the changes compile cleanly. However, S1 must be changed when the logic is cloned in the same module, because the cloned value of S1 is guaranteed to be the only block in the module that is wrong. The failure to do this is not caught by Composer or WinCAD compilers, so the error does not get corrected.

 

Let's start with an example. Here is our message, as shown in Error Browser, with the good stuff circled in green:

 

 

 

Block 2614 is clearly expected to be asserted when block 2609 is put into Auto mode.

 

DBDOC shows you the block index for block 2609, which you see here with the green highlight for the intended and correct logic.

 

 

 

  

What do you make of the purple, red and blue highlighted references? Click on each of them to go to the TSTALM block that is strangely testing block 2609.

 

 

 

 

 

S1 of block 2115 (purple) obviously should be 2111, not 2609.

S1 of block 2365 (red) obviously should be 2361, not 2609.

S1 of block 2490 (blue) obviously should be 2486, not 2609.

 

Count the errors:

• Blocks 2116, 2366 and 2491 will be asserted when MSDVDR block 2609 goes into automatic mode - three errors.
• Blocks 2116, 2366 and 2491 will NOT be asserted when MSDVDR blocks 2111, 2361 and 2480 respectively go into automatic mode - three more errors

 

Six errors for the price of one! Good value from DBDOC, eh?

 

Composer should be made to set the value of S1 to -1 when a TSTALM FC 69 block is placed on a page. The illegal value should force the user to enter one that would work.

 

Note also that the warning triangles (Error Markers) are in front of you when you get to the dysfunctional logic, so you can see right away that there is possibly a problem. The two messages they show are:

 

Tested block on different page (which should be a clue), and

TSTALM block tests subsequent block, which means they are out of natural sequence.

 

I have found as many as seven such duplications (that is, 14 errors) that came from the cloning of a single set of functioning blocks. Clearly, this error can be made and missed. DBDOC can find it for you.

 

When you review these using DBDOC Error Browser, you simply hide any that are of no consequence. You flag ones like this that affect the process. When they are fixed, they will disappear from the diagnostics. If you create such an error, it will show up as a new error, highlighted in yellow.

 

DBDOC helps improve your system integrity.

Surprise! Your TSTALM Blocks may put random device drivers into override (or do even worse).

TSTALM blocks can easily be misconfigured when working logic is reused. The most common error is simply failing to change the input specification to get the status from a different block (i.e. S1 is accidentally left unchanged when the function block is copied and reused, and thus both old and new TSTALM test the old block, and nothing tests the new block). When this error is made, there are invariably two consequences:

1. Action that is intended to be taken does not happen when it should.
2. That very same action does happen when it should not.

This note analyzes just one of thirteen instances identified by DBDOC in a particular system where a TSTALM block tests a block that already has a TSTALM applied to it (in fact two of these errors are not a concern, because previous examination has shown that the incorrect action is not taken because the signals are unused).  The remaining eleven messages in fact indicate twenty actions that will not happen when they should, plus twenty places where the same action will happen when it should not.

Of course, a situation where a TSTALM block triggers one or more wrong actions, and where the designed TSTALM block fails to trigger the desired action, can be serious for any block type.

Example:

The TSTALM error is typically caused caused by the DCS worker failing to change specification S1 to match logic that is put onto a sheet and given new block numbers.  So the block is copied, but the TSTALM S1 is not changed to match the new block it is supposed to be testing.

Take a look at this image:

 

 

 

[A] Error Browser: Multiple TSTALM: Module 1,10,04 Block 3623 is tested multiple times

This error is selected in the error browser.  It indicates that more than one TSTALM block is testing Block 3623.  The Error Browser shows a total of thirteen errors of this type.

 

 

[B] TSTALM Block 3632 -- The working original

 

The MSDRVR (Block 3623) is being tested correctly for being in Auto or Manual because the TSTALM (Block 3632) shown has S1=3623, indicating that it tests MSDVDR Block 3623. 

 

 

[C] Intended action on block in Auto or Manual

If TSTALM (Block 3632) is put into Manual, its output at block 3633 will be 0. However, if TSTALM (Block 3632) is put into Auto, block 3633 will be set to 1.

 

 

[D] OR block with this result in it

 

Clearly documented "OVERRIDE TO DEFAULT", this action works in the prototype logic.  When TSTALM (Block 3632) is in Auto, the S25 override input into MSDRVR (Block 3623) will be set.

 

 

[E] The MSDRVR block Override input

 

The diagrammed MSDRVR (Block 3623) will be put into Override when it should be.

 

 

[F] Working action

 

Tag 1SLWHS806 "FLUSH WATER PUMP" will be put into Override if TSTALM (Block 3632) tests the  MSDRVR (Block 3623)as expected.

 

 

[G] Seven errors:  MSDRVR (Block 3623) is spuriously tested by SEVEN other TSTALM blocks.

 

The block index shows seven more TSTALM blocks responding to the status of tag 1SLWHS806 in a spurious and possibly dangerous fashion.

 

These seven  TSTALM blocks, each with their S1 set incorrectly to 3623, are displayed below.

Each of these seven TSTALM blocks will put some random MSDVDR into Override when the unrelated MSDRVR (Block 3623) is in Auto, and at the same time fail to put the expect block into Override when it should. 

 

Ramifications:

 

Some of the failed or unintended actions might be noticed. Others will not be until a problem arises. It is very likely that the cause of the sporadic operation will not be found if the problem actions or failed actions are noticed, because there is no expected logical connection between these unrelated blocks.