A Support Query: Importing the Wrong Value from an M/A Station Block

Dear Support:

I was wondering if you could explain this error to me.

Dear Client:

For sure.

One place block 3088 is used is as a block imported by an AI/L.

The exception report from an AO/L block looks like this:

• one byte - status bits, including H, L, Q
• one real - value of the AO/L

The exception report from a STATION block looks like this:

• one byte - status bits, including H, L, Q, Deviation
• three real values in this order
•     value of the PV (S1)
•     value of the SP (output N+1)
•     value of the CO (output N)

The AI/L block reads and supports the first two values - one byte, from which it gets the quality, and one real, which it imports as the value.

The message tells you that the value of the AI/L block imported equals that of the PV of the station block, something like 239 (S1), not the CO value of 46% (Block 3088), which is probably what is expected.

Please send the same screen shot, but also the AI/L block.




Dear Support:

The AI/L block

Dear Client:

As you see, the description PLANT AIR FLOW CONTROL OUTPUT just does not cut it. In fact, 46% would not be not high enough to trigger the 50% level in the subsequent H//L block, but the erroneously imported PV of 239 is.

You could insert an AO/L after the station block, to transmit the actual output value to the AI/L block.

However, a simpler way might be to put an H//L block on the page where the Station block 3088 exists and send the output of the H value to a DO/L, then import that DO/L here.

Does this now make sense?

Output Spikes - Sharp as Tacks!

This does not look good!

Here is a warning about derivative gain from a site, highlighted in the Error Browser.

PID Module 1,06,02 Block 214 uses derivative with PV from discontinuous input Module 1,06,02 Block 140

You can see the error marker next to the PID block it concerns.

Dan Reynolds sent me the data from the control loop identified. Here is what it looks like with DBDOC WatchWindow at 250 ms.

Zooming in a bit, we see that Block 280, the output, suffers spikes from its railed value of -5.0%. Block 214 shows a value of +1.7408%, that is, 6.7408 above the -5.0% it should be. The exception reported input Block 140 changes each minute and drives these spikes (not all of which are picked up in the 250ms reporting interval, since they are so brief). You can be sure that, if the control output were in the 0 % to 100% range, large positive and negative spikes would be suffered every minute. Ouch!

Just as clearly, if this PID output were not driven to -5 %, the spikes would be bi-directional and still significant. The effect is purely caused by the value of Kd on S8 for Block 214, only 0.2, but enough to drive the output in a way that is not valid.

By the way, you should notice how any exception report input has steps in its value. The values for Module 1,06,02 Block 140 and Module 1,03,04 Block 1901 show these characteristic steps. See Monitoring Exception Report Performance for details.

Alarm Management Needs Error Management

Does your system contain alarms that cannot actually ever occur?  Many do.

Alarm management focuses on understanding and controlling the alarms that can be generated in a system.  DBDOC provides techniques for walking through the configuration systematically to validate assumptions and look for problems. Effective alarm management is impossible without error management and configuration analysis.

Here are three example sets of alarms that do not work in a single powerplant configuration:

1. Intermittent failure caused by duplicated exception report imports
2. Detected failures to test blocks with quality
3. Problems that can be found by using DBDOC to walk through the configuration easily

If you do not manage your errors and use the tools DBDOC provides to audit the configuration, you will not be able to manage alarms. Even worse, you can spend a great deal of money fooling yourself about alarms.

Example 1 - Frozen Import Block Error

INFI 90 allows block values from one module to generate "exception reports" when the block value changes status or value. In the case of an analog exception report, the new value is transmitted when it changes by an amount that is specified as "significant".

If a an exception report block value is imported more than once into a module, only one of the import blocks works. The values of any other blocks importing the exception report block value will remain frozen. Furthermore, when an on-line configuration is done, a second import block will silently begin working (!) and the block that was working will freeze at its last value. This applies to both analog and digital exception reports.

DBDOC detects and reports this error. However, when the module is compiled using Composer, the error is not detected.

      Error: Frozen import block [093]

      Duplicate off-module reference (DI/L, DI/L) to Module 1,01,05 Block 1230: 1020324 and
      1020342 [1010536]

The source value above is imported in two places in one module. This instance on a spare block page might have the active connection.

The following shows the intended logic.

There is a 50/50 chance that the correct logic is not working at any time. Simply put, tags 0D0730 "COND RETURN TANK HIGH/LOW LEVEL" and 0D0727 "AUX DEA HIGH/LOW LEVEL" might alarm, but they might not. It is a crap-shoot!

Note: It would seem reasonable to assume that AI/L block 1675, now spare, actually started life feeding S3 of block 1676. The duplicate import situation was discovered and the logic changed properly to get both values of S3 from a single AI/L block somewhere in the past. The problem was that the spare block still can be the active one.

Example 2 - Test Quality of a block that does not have quality to test

Powerplant A has 7707 TSTQ blocks. These four input blocks give a binary [1] as a result if one or more of their inputs have bad quality. Actually, 8592 blocks are being tested in all. Of these, 316 do not have quality to test at all.

Here is one example of an alarm (and shutdown) that will not work, identified by the message here.

       Error: TSTQ tests block with no quality [063]

 
       TSTQ Module 4,05,03 Block 2273 tests Module 4,05,03 Block 6170 (FC 15), which does not
       have quality [4050378]

Obviously, the simple solution to the problem of no alarm on bad quality for 1 GROSS MW TRANSDUCER A is simply to test the quality of the input at IREF P1E0388A instead of the output of the fudge factor sum block 6170.

Example 3 - Misconfiguration of a DAANG Block

DBDOC also allows you to go looking for trouble. Walking through all instances of a Function Code will allow you to look for things that were missed long ago. This example shows bad quality that will not alarm.

DAANG blocks are powerful tools when configured competently. Powerplant A has 6227 of them, the second highest number we have seen in a single plant. I was curious to see how consistently programmed they were. The table shows how a value places on Specification S8 can set bad quality.

This plant uses this extensively. This example shows how the constant 32.0 is applied to S8 with the green highlights. When the raw signal is found by the TSTQ block to be bad quality, the DAANG block will be marked bad quality as intended.

The following example shows that undetected errors exist. The red highlights show that bad quality on this raw input will result in good quality being forced. Oops!

Being interested, I started walking through the 6227 DAANG blocks looking for this particular issue.  I got bored after 200, taking 16 minutes, about 5 seconds per block. I found (and flagged with DBDOC annotations) the following reality:

• 124 blocks with correct alarm configuration - setting 32.0 into S8
• 22 blocks incorrectly setting 1.0 into S8
• 1 block incorrectly setting 0.0 into S8
• 53 of the first 200 DAANG blocks did not set the quality attribute in this way.

The bottom line is that 12% of the first 200 DAANG blocks (all tagged, by the way) would not alarm on bad quality. I have no reason to think this is not representative, which suggests that perhaps 800 of these tags will not alarm on bad quality in this one system.

By contrast, I checked a system with 7980 DAANG blocks. There were no alarm errors in the first 150 instances, with all quality inputs being generated correctly. At least in this respect, good configuration work would make alarm analysis valid.

In a second system with 5084 DAANG blocks, I found that only 2042 of them used the S8 to set bad quality if needed. In fact, this simple walkthrough with DBDOC showed that that 60% of the blocks I sampled were fed from signals that had no quality to test, but did not reinsert bad quality. That could mean about 2000 simple errors - tag does not show bad quality when it should.

Error and Configuration Management Trump Alarm Management

Alarms cannot be managed if errors prevent them from working. DBDOC can be used to improve alarms both by looking at errors and by walking through the project carefully. Otherwise, any alarm analysis will be incomplete and unacceptable.

Output Cusps - Exceptionally Artistic!

We came across the pretty but alarming output forms shown here at a client. The scale is one minute per cusp and just over 1% amplitude. This is important because it means that the HMI and historian would not see this at all. DBDOC Watch Window handles it nicely, as is evident.

So what is going on here?

The blue line is the output of a FC 80 Station block and the output cusps were, in fact, a significant issue. The green line is the raw PV input (nice and smooth). However, the APID block was being fed by an exception-reported version of the PV, shown in red. The significant change specification was 0.1%, so the exception-reported value actually was competently implemented, changing in steps, as expected.

The problem simply is that the APID algorithm used a derivative term. As the two diagrams make painfully obvious, this simply does not work. Although the PV signal is changing at a nice fairly constant rate, the controller, fed by the red step input, is hit with a huge change accumulated over one minute, but seen as happening in 250 ms. Thus, the derivative is calculated as being 240 times bigger than it actually is.



The second image shows that, if the PV is changing so that the exception reports are triggered by the significant change, the cusps are smaller, as they have a better time base. but they are still misleading and problematic.

The bottom line is that PID and APID algorithms should never use a derivative term if the PV is affected by an exception-reported input PV. It cannot work mathematically, and it sure kicks the process in the teeth, again and again.

DBDOC can't yet detect this error situation automatically, but we are working on it.  Stay tuned.

Are Dated Design Compromises Affecting Your HMI and History Data Precision?

I got busy in the purported Christmas break and wrote a bunch of blogs.  This blog is a roadmap through several related areas. Read on to

• learn if you are overloading the exception report generation of nodes
• verify under-utilization and improve your console and history data
• automatically detect and report exception report delays

Most systems worry about the above, and most were configured a decade or two ago. Learn if you can get much more out of what you have.

For many INFI 90 systems, the power of the modern NIS21/NPM22 hardware capability will allow you to get much more precise data to consoles and history systems.

If you have DBDOC, you can find out, for every PCU / node that you think is heavily loaded, if it really is. Heavy loading is 50% to 90%, overloading above that.  This is how to do it:

Monitoring Node Communication CPU Load

If you find the communication CPU usage in the 10% range, you are not getting enough out of INFI 90 and are not giving operators, managers and engineers data that is good enough.  You can do better:

Improved Tag and History Data Precision in INFI 90 Systems

There is also a design for you to be able to permanently and validly measure whether you ever get delayed or lost exception reports.  If you find you have nodes in the 50% or greater node communication CPU usage, you should worry about this.

Monitoring Exception Report Performance

Have you tried any of these techniques?  It would be very interesting to hear your experiences.

FC 222 and 223 Can Have Severe Exception Report Problems

Summary

If you have FC 222 and FC 223 blocks, you should check for the following problems:

1. FC 222 S8 defaulted to 0.0 and FC 223 S9 defaulted to 0.0 which causes continuous exception reports if the block is tagged in any HMI.
2. Spare FC 222 and FC 223 blocks tagged but not used which cause absolutely wasted continous exception reports.
3. FC 222 and FC 223 blocks with all the significant change specifications set to 1, even when this value is not appropriate.

Details

At a client site, DBDOC was responsible for uncovering a significant loading problem involving FC 222 - Analog In/Channel that has been causing significant loading of the node communication capability. It turns out that FC 223 - Analog Out/Channel has the same problem. Here is an outline of the problem:

• site has significant number of FC 222 S8 and FC 223 S9 blocks at default value of 0 significant change
• even worse, many of these have "spare" tags identifying unused blocks brought into the HMI by exception report
• the result is exception reports generated by the node at once per second, many utterly wasted, and the rest far too sensitive to be valid.
• examination of other systems shows that some have all default values replaced, although there is clearly no understanding that the value is in engineering units (like FC 177) not in percent

DBDOC tools including the Significant Change Report and the extraction of all specifications made it easy to look for the problems once they were conceptualized.

In the site in question, a power plant installation, the bulk of the FC 222 blocks were done relatively recently, taking advantage of modern INFI 90 hardware.

There are 573 FC 222 blocks with default significant change of 0. Our tests with DBDOC Watch Window easily proved that each of these blocks, either if imported or in the HMI tag database, generated an exception report every Tmin seconds, that is, every second. In fact, 92 more FC 222 blocks had non-default significant change, because they had been properly configured.

What about these 665 FC 222 and FC 223 blocks?

• One had no tag, but was imported by an AI/L block, so it is generating an exception report anyhow.
• 196 were named spare blocks, none used in graphics or PI, so196 XR's per second wasted.
• The other 469 blocks were giving an XR every second.

The PCU distribution of the load was:

• PCU 2 - 640
• PCU 3 - 16
• PCU 4 - 8
• PCU 5 - 1

There actually were 2181 tags defined in PCU 2, which means that every Tmax of 60 seconds, each generated an XR. Thus, the base XR load was:

• 573 per second from default FC 222 blocks
• (2181 - 573) / 60 = 27 per second from the other tags

That is, the base XR load on the node is 600 XRs per second. It is a good thing this system has NIS 21 / NIS 22 communication cards. As it is, this showed that the average communication CPU usage in the node of about 50% was a valid figure.

Why worry about this?

554 of the 640 tags and export blocks had default significant change. If the defaulted FC 222 blocks had 1% significant change, the base load would nominally be 2181 / 60 sec or 36 XRs / sec. Allocating 600 XRs per second nominally would allow the analog tags to be increased in sensitivity by a factor of 20, with less load on the system than there is now.

This investigation also brought our attention to FC 223, the Analog Out/Channel block, which also has a default significant change of 0.

Guess what? There are 80 of these in PCU 2. They all have the default 0 significant change, so they are generating 1 exception report per second. 31 of them are spare blocks, so that load is doing nothing.

Bottom line is that, unbeknownst to our client, the exception report loading on the node included:

• 96 analog in and 31 analog out spares giving 127 utterly wasted XRs per second
• 469 analog in and 49 analog out tagged blocks giving 518 very, very, precise values every second.
• 26 XRs per second from the other 1536 "low class" tags, most at default significant change.

Thus, the load was about 600 XRs per second. If the spare tags were deleted and the defaulted ones set to 1% significant change, the load would be 518 / 60 per second (typically) or 9.

This node would generate 35 XRs per second as currently tuned. There is lots of XR capability to improve its tags and history data using the capacity that is nearly completely wasted right now.

How many other nodes with FC 222 and FC 223 blocks have this bad situation?

The good:

• Large Australian power plant has 10663. One has default significant change. It is also not tagged.
• Large Canadian power plant has 1610. None of the 161 that have 0 significant change is tagged.
• Large American power plant has 1312, none of which is defaulted.
• Medium size Canadian power plant has 192, all with sig change set to 1.

The bad:

• Large Australian process plant has 66 with default values. 16 are tagged, which is a meaningful load.

The ugly:

• These small American power plants with all values defaulted at 0 and over one-third tagged but spare.
• Small American power plant with 88 of 754 with default 0. 86 of these have tags, so they are a load. 21 are spares, so they are a wasted load.
• Medium size Canadian process plant has 496, all at default 0. 216 are in PCU 171 and 280 in PCU 172. Every one is generating one XR per second.
• Medium size American process plant has 559 FC 222, of which 368 are tagged with significant change 0. No spare ones are tagged.

It is clear that the significant change specifications and tags for all FC 222 and FC 223 blocks should be examined. The load can be large, and it can be a waste of a precious resource, even stalling or losing exception reports.

Postscript

The general situation with FC 222 and FC 223 blocks that are not defaulted is to have the significant change specification set to 1, no matter what the span.  This is probably an error, too, suggesting that the work was done under the misconception that the specification is a percentage value, rather than an EU one.  1 in a span of 5 is 20%, whereas 1 in a span of 1000 is 0.1%.  Since both appeared in the same system, with the only value used being 1, it is likely the values should be studied even when they are all not defaulted.

Monitoring Exception Report Performance

Are Dated Design Compromises Affecting Your HMI and History Data Precision?

In many INFI 90 systems, there is a significant loss of continuity between the system implementation and its current use and performance.  The experts that made the original decisions are long gone, with no documentation and justification for many decisions that were made.

How can you find out if the system performance tuning decisions made are still valid?  Why is it possible that the way things were done years ago is no longer valid?  

Systems have gone from NET 90 to INFI 90 (plant loop to INFI loop).  Additionally, the PCU now has much higher communication performance available in the form of the NIS21/NPM22 pair.  The old hardware could only generate (in a PCU) 400 exception reports (XRs) per second, whereas the new hardware can do many times that number.

What are the ramifications for INFI 90 systems?  Can limitations decided on long ago be backed off? Are the decisions made long ago still valid now after you spent all that money on new communication modules?

Especially if you have PCUs with new NIS21/NPM22 communication modules, you are very likely to have the ability to generate much more fine-grained data for your HMI / consoles and for your history system, essentially for free.  The capability is there.  Why not use it?  If your system has older hardware, it may have been set up so conservatively that there is actually lots of capacity available you can avail yourself of, too.

Here is my humble sketch of how you would use spare blocks (hopefully) and a lightly loaded module to monitor the performance of your system from the perspective of timeliness or delay of exception reports.

1. In a Module to be Monitored, Create An Exception Report Generated Once Per Second

The first outrageous suggestion is that you generate an exception report from the seconds value in any module you wish to monitor.  The sketch shows this.

To be able to stop the exception reports, you would simply put a transfer block and an ON/OFF block between block 22 and S1 of the AO/L block.  This would allow you to turn on the exception reports only when you wished to check.  However, it is very difficult to imagine that adding one exception report per second per module will really make a significant difference.

You can see that, as 1 second is more than 1% of 100, an exception report will be generated every second. The objective of this logic is to use the added load for good benefit.  You will have the seconds value in Loop 1, PCU 10, Module 4 available as an exception report.

2. Create Your Time Standard in Lightly Loaded Safely Manipulable Module

The rest of the implementation uses a lightly loaded module that you can manipulate safely.  Since the objective of this work is to increase the precision of a lot of data in the system, you have to decide if the effort is worthwhile, of course.

In most systems, the time is synchronized around the loop.  Do you know it works for sure?  If it does not, you will find out when you go further.  The second bit of logic, in the lightly loaded module (Loop 1, PCU 5, Module 4 in the example), simply puts an Output Reference on the seconds value in that module.

You can see that the value on the OREF "10504 SECONDS" ought to be pretty close to any other SECONDS value in the whole system.

3. Create an Exception Report Import and Comparison Logic

All that is needed now is to import the seconds value from the module being monitored and compare it to the reference value in our lightly loaded module.  Probably 5 seconds for an alarm is horribly high, but normal monitoring techniques will tell you easily enough how tight you can get the test.  If you define the tag "11004 SECONDS LAG" (or "11004_SECONDS_LAG"), you can get an alarm whenever the exception report is delayed (or simply log it).

Of course, your imagination is the limit.  You can send the difference to an AO/L block and monitor that using your historian.  You can filter and average and remove jitter if that is needed.  The sky's the limit!

Note how easily you could monitor all the modules this way that are purported to be heavily loaded.  The only significant work involves the lightly loaded module that you can manipulate safely.

4. Ramifications for Improved HMI and History Data

What happens if you find that there is no delay?  You also might have implemented the technique in the post "Monitoring Node Communication CPU Load" and found that your node CPU usage is very minimal.

The most important limitation in existing INFI 90 systems is the Significant Change specification.  This number tells either a percent of span (AO/L blocks, Station blocks) or an actual EGU value (FC 177, 222, 223) that triggers a new exception report.  Traditionally, 1% is the default value used to try to keep the load managed.  Is this still valid, if you have no exception report generation delay and idle communication modules?

If you have proven that you have excess communication capacity, you can start improving (reducing) the significant change specifications of important or too coarse tags to get better data to the operators, engineers and executives.  You can be sure that you have lots of capacity by keeping the CPU load reasonable (but higher than the minimal level found in many modern systems).  You have a direct test and warning if you are getting delays in the generation of exception reports.

5. Negative Ramifications of Current INFI 90 Practice with respect to Significant Change

In case you wonder, or have to justify even thinking about the monitoring suggested here, you will want to look closely at my post Improved Tag and History Data Precision in INFI 90 Systems.  It will give you "grist for the mill" as you try to make the data originating in your system better.  It will show you that there usually is a lot of room for significant improvement.

Improved Tag and History Data Precision in INFI 90 Systems

INFI 90 is based on the concept of Exception Reports (XRs) that reduce the communication load needed to get good values to HMI systems, historians, OPC servers and to be used in other parts of the system.  Analog values have a "significant change specification" to prevent load from analog values that are not changing significantly.

History

This history is approximate, as told to me by gurus over the past years.  

Net 90 and INFI 90 started with a 1 MBaud plant loop communication ring, plus the impediment that a single XR was sent for each enrolled user.  These two factors made it easy to overload the capacity of the system.

Infi Loop introduced both a 10 MBaud communication ring and a protocol that allowed multiple destinations on the ring for an XR, so the load caused by an exception report no longer changed if more consoles used the tag, for example.

Pragma

Practical considerations were developed to avoid the loss of exception reports in the early systems, and developed further as the systems got both bigger and faster.  Bigger, of course, meant more load, but faster meant more capability.  Initially, the practical suggestion and default was that a change of 1% of span in and analog value was a good compromise.  Sensors were often enough not very accurate and it seemed logical.

The Modern World

In the modern world, there are aspects you should take into account:

1. INFI 90 communication capability increases have not usually been translated into more precise data.
2. Sensors have often changed from 2 1/2 digits to 3 1/2 digits, ten times as precise.
3. Techniques now exist to easily monitor exception report delay and communication loading.
4. Decisions based on data that is not sensitive enough are made regularly, probably with affecting your plant negatively at times.  This applies to both console operations and historical data.

A Case Study

In a plant, examining the DBDOC Significant Change Report, we noted a flow rate with a span of 7000 and default 1% significant change.  Because it was being imported into another module by exception report, we could study the suitability of the significant change setting.

There were 23 exception reports in the period shown, 18 of them at Tmax of 75 seconds.

Here are some salient aspects of the raw and imported values:

• Raw value is the pink line.
• Imported value is the blue stepped line.
• Working range is under 400.
• Tmax is 75 seconds, and all positive-going exception reports were caused by the timeout.
• The negative-going exception reports were triggered by the drop of 70 before 75 seconds had elapsed.

The green sections show where the imported exception report value was less than the actual value.  The orange ones show where it was greater.  Interestingly enough (to me, anyhow), I had joked in front of clients about how bad it would be to be integrating exactly this sort of a variable.  You would find the accountants for the customer very happy when you told them you had used the blue step function to decide how much they should pay.  They would pay fast, and not quibble.

Out of curiosity, I estimated the performance using 0.1% significant change.  I did this by simply taking the raw value as a start and making a pseudo-XR when the value increased or decreased by 7.0 or more.  This gave me an estimate of how the imported values would look and what the performance would be.  The number of exception reports would have been about 140 in 1420 seconds, or about 6 per minute.  Only one period of 75 seconds went by when there would not have been a significant change. 

What does this look like from a numerical perspective?

Significant Change       1%      0.1%

Mean error            -8.1      -1.3

% of full span       -0.12%    -0.02%

% of working range   -2.02%    -0.34%

Mean error magnitude  29.0       3.5

% of full span        0.41%     0.05%

% of working range    7.24%     0.87%

The Bottom Line

By using the data available through DBDOC Watch Window (or Composer Trending, or any other block value monitoring package), you can see that it is perfectly possible for an exception reported value to be handled in a way that could be problematical.  The data comes from an exception report import block, but applies to every value on every graphic, and to every historical value in every INFI 90 system.

At a small cost in increased exception reports, using capacity that can be verified as being available safely, process values can be much more precise.  The full analysis that is done here is not necessary.  What you need to do is simply:

• Identify imported values, tags and historical data that needs more precision
• Monitor the node communication loading
• Where there is capacity, use it by making the significant change specification tighter

The improvements in console, history and perhaps control and shutdowns will be significant.  You probably have the capacity to do this right now.