Friday, August 22, 2014

Managing Errors with the Error Browser: Function Generator errors

DBDOC has noted CHECK messages for over a decade.  Errors are classified with a CHECK severity (as opposed to ERROR) when many or most of them will not actually cause a process problem, yet certain circumstances exist where they could.   ERROR severity errors should be investigated and corrected first, and then CHECK severity errors can be addressed.

One example of a CHECK error is "X coordinates out of order for F(x)", which is DBDOC's check on the order of the X coordinates in FC 1, F(x) Function Generator block.

The X coordinates (the even numbered specs from S2 to S12 for a Function Code 1 block) must be in ascending order for the block to work properly. Essentially, linearization is done stepwise, with only one pair of coordinates involved. However, many of these blocks have inconsequential errors where the output is flat, or there are minor round-off errors, or the blocks are simply not used. Some have the errors outside the design range of the calculation. It's important to identify the particular situations where an actual problem is present.

Suggested process for analyzing CHECK errors with the Error Browser

Below we work through this process in a system with 39 such errors. We will walk through the steps of figuring out which of these are potential process-affecting errors, and which can be safely hidden and ignored. This can only be determined in the context of each specific site

We open the Error Browser, set Show these errors to "All," set  Group by to "Severity" and then by to "Error Name," and press Rebuild Tree.  The tree now shows all the different types of errors in the system, organized by their severity.

In general, it makes sense to hide CHECK errors and other errors less severe than ERROR errors (e.g. COSMETIC, INTERNAL).  They will then be out of your way while you focus on the ERROR severity errors.  When you are ready, come back to the CHECK errors, and identify those that could cause problems.  In this example, it took me about a minute to analyze the 39 " X coordinates out of order" errors and identify the single one of those that was actually a cause for concern.

Here is an Error Browser procedure to follow for this error type, and generally speaking, for any other CHECK severity error:
  1. Set Show these errors to "All," set  Group by to "Severity" and then by to "Error Name," and press Rebuild Tree.
  2. Select the error type of interest, in this case "X coordinates out of order for F(x)".
  3. Mark them all as Hidden.
  4. When you have time, walk through each error, checking if the block it refers is actually used.  If it used, Star it.
  5. Mark the errors that you have walked through as Reviewed.  Now they are both Reviewed and Hidden, and need not be further worried about.
Analyzing the starred error is outside this minute, of course.

Marking the "X coordinates out of order for F(x)" errors as Hidden

Click on the "X coordinates out of order for F(x)" in the tree to select all errors of that type, then click on the Hide icon to hide them.  This will get them out of your way until you are ready to walk through them in detail.  Because they are CHECK errors, they are unlikely to be causing any problems, but a few of them might be.

Walking through the errors, checking which problem blocks are actually used

Click on the first "X coordinates out of order for F(x)" message. You will see both the message details in the lower panel of Error Browser, and the function generator block it pertains to on the CLD in the main browser window.
  • The block index shows that the F(x) block in question is used in only one place (that is, it is not imported anywhere else) [green].
  • Links to complete documentation of the message are available [purple].
  • The specifications [blue] are there to examine. The problem with the specifications [red] is evident. The X coordinates (even numbered specs) are decreasing, not increasing. This block will give an output of 0.8 for all inputs less than 120.0, and 23.2 for inputs more that 120.0. It doesn't work.
But you do not care, because the F(x) output is not used.  If it were used, there would be more than one entry in the Block Index.

Arrow down through all the error messages of this type, and all but one will turn out to be unused.
The F(x) block with "X coordinates out of order" that is actually used

Only one block with this error is actually used in the process.  You can see (below) that it is used because more than one item (the block source, as well as its use) show up in the Block Index.

The output value is wrong in the input range from 5.0 to 10.0. Like the negative side, it is supposed to rise linearly from 1.0 at 5.0 to 3.0 at 10.0. Instead, the output jumps from 1.0 to 3.0 when the input reaches 5.0. Analysis of the logic using the FC 16 multiply block is needed to determine how adversely this will affect the process. However, if you consider that the output is 3.0 for all values of input other than from -10.0 to 10.0 (by design), it is clear that 25% of the design effect is missing. Furthermore, one can worry about the effect of the step function in the output, something that DBDOC also has flagged.
Click on the Star to mark the problem error. You can also Unhide it. Now you can Rebuild Tree, which shows up whenever the Error Browser presentation would change. I also changed to show "Active" errors instead of "All", which makes all the Hidden errors not show up in the tree.
What is the error? You must be curious by now.
The messed up number feeds an adapt block that affects the overall gain of an APID block controlling a tag called 2LIC-459 "GREEN OIL TOWER REFLUX DRUM". The error means that there is no gradual ramping of the overall gain from 1.0 to 3.0, which might be nasty. On the other hand, GREEN OIL TOWER might no longer be used at all, or perhaps never installed, or whatever. DBDOC makes it all as easy as A, B, C.
  • A - The Function Generator Block 4008 is flawed, making a step change in the value where it should be smoothly changing.  This feeds into Multiply Block 4010.
  • B - The bad output from Multiply Block 4010 is fed into the Adapt Block 4011.
  • C - The bad value is set into S11 of APID Block 4016.
  • D - As a result, APID Block 4016 has a step change in overall gain in part of its operating range.
  • E - The output signal generated will jump if the error (my guess) exceeds 5.0.
  • F - Control of 2LIC-459 will be dicier than it should be, because control signals will be more extreme than designed.
Displaying the actual F(x) Block function, with live data
As an aside, DBDOC will show you the actual F(x) Block function, with its live data (but not here, because I do not have the plant data). You turn it on by right-clicking the output block number [A]  and selecting the live function graph feature [B]. Double-click on the graph [C] to dismiss it.
Getting more information: Help links in the Error Browser
This is as good a place as any to mention the fully featured help that is built into Error Browser, from the purple highlights above.

Wednesday, August 6, 2014

Surprise! Your TSTALM Blocks may put random device drivers into override (or do even worse).

TSTALM blocks can easily be misconfigured when working logic is reused. The most common error is simply failing to change the input specification to get the status from a different block (i.e. S1 is accidentally left unchanged when the function block is copied and reused, and thus both old and new TSTALM test the old block, and nothing tests the new block). When this error is made, there are invariably two consequences:
  1. Action that is intended to be taken does not happen when it should.
  2. That very same action does happen when it should not.
This note analyzes just one of thirteen instances identified by DBDOC in a particular system where a TSTALM block tests a block that already has a TSTALM applied to it (in fact two of these errors are not a concern, because previous examination has shown that the incorrect action is not taken because the signals are unused).  The remaining eleven messages in fact indicate twenty actions that will not happen when they should, plus twenty places where the same action will happen when it should not.

Of course, a situation where a TSTALM block triggers one or more wrong actions, and where the designed TSTALM block fails to trigger the desired action, can be serious for any block type.


The TSTALM error is typically caused caused by the DCS worker failing to change specification S1 to match logic that is put onto a sheet and given new block numbers.  So the block is copied, but the TSTALM S1 is not changed to match the new block it is supposed to be testing.

Take a look at this image:
[A] Error Browser: Multiple TSTALM: Module 1,10,04 Block 3623 is tested multiple times

This error is selected in the error browser.  It indicates that more than one TSTALM block is testing Block 3623.  The Error Browser shows a total of thirteen errors of this type.
[B] TSTALM Block 3632 -- The working original
The MSDRVR (Block 3623) is being tested correctly for being in Auto or Manual because the TSTALM (Block 3632) shown has S1=3623, indicating that it tests MSDVDR Block 3623. 
[C] Intended action on block in Auto or Manual
If TSTALM (Block 3632) is put into Manual, its output at block 3633 will be 0. However, if TSTALM (Block 3632) is put into Auto, block 3633 will be set to 1.
[D] OR block with this result in it
Clearly documented "OVERRIDE TO DEFAULT", this action works in the prototype logic.  When TSTALM (Block 3632) is in Auto, the S25 override input into MSDRVR (Block 3623) will be set.
[E] The MSDRVR block Override input
The diagrammed MSDRVR (Block 3623) will be put into Override when it should be.
[F] Working action
Tag 1SLWHS806 "FLUSH WATER PUMP" will be put into Override if TSTALM (Block 3632) tests the  MSDRVR (Block 3623)as expected.
[G] Seven errors:  MSDRVR (Block 3623) is spuriously tested by SEVEN other TSTALM blocks.
The block index shows seven more TSTALM blocks responding to the status of tag 1SLWHS806 in a spurious and possibly dangerous fashion.
These seven  TSTALM blocks, each with their S1 set incorrectly to 3623, are displayed below.

Each of these seven TSTALM blocks will put some random MSDVDR into Override when the unrelated MSDRVR (Block 3623) is in Auto, and at the same time fail to put the expect block into Override when it should. 
Some of the failed or unintended actions might be noticed. Others will not be until a problem arises. It is very likely that the cause of the sporadic operation will not be found if the problem actions or failed actions are noticed, because there is no expected logical connection between these unrelated blocks.