Friday, December 20, 2013

FC 222 and 223 Can Have Severe Exception Report Problems

Summary

If you have FC 222 and FC 223 blocks, you should check for the following problems:

  1. FC 222 S8 defaulted to 0.0 and FC 223 S9 defaulted to 0.0 which causes continuous exception reports if the block is tagged in any HMI.
  2. Spare FC 222 and FC 223 blocks tagged but not used which cause absolutely wasted continous exception reports.
  3. FC 222 and FC 223 blocks with all the significant change specifications set to 1, even when this value is not appropriate.

Details

At a client site, DBDOC was responsible for uncovering a significant loading problem involving FC 222 - Analog In/Channel that has been causing significant loading of the node communication capability. It turns out that FC 223 - Analog Out/Channel has the same problem. Here is an outline of the problem:
  • site has significant number of FC 222 S8 and FC 223 S9 blocks at default value of 0 significant change
  • even worse, many of these have "spare" tags identifying unused blocks brought into the HMI by exception report
  • the result is exception reports generated by the node at once per second, many utterly wasted, and the rest far too sensitive to be valid.
  • examination of other systems shows that some have all default values replaced, although there is clearly no understanding that the value is in engineering units (like FC 177) not in percent
DBDOC tools including the Significant Change Report and the extraction of all specifications made it easy to look for the problems once they were conceptualized.

In the site in question, a power plant installation, the bulk of the FC 222 blocks were done relatively recently, taking advantage of modern INFI 90 hardware.

There are 573 FC 222 blocks with default significant change of 0. Our tests with DBDOC Watch Window easily proved that each of these blocks, either if imported or in the HMI tag database, generated an exception report every Tmin seconds, that is, every second. In fact, 92 more FC 222 blocks had non-default significant change, because they had been properly configured.

What about these 665 FC 222 and FC 223 blocks?
  • One had no tag, but was imported by an AI/L block, so it is generating an exception report anyhow.
  • 196 were named spare blocks, none used in graphics or PI, so196 XR's per second wasted.
  • The other 469 blocks were giving an XR every second.
The PCU distribution of the load was:
  • PCU 2 - 640
  • PCU 3 - 16
  • PCU 4 - 8
  • PCU 5 - 1
There actually were 2181 tags defined in PCU 2, which means that every Tmax of 60 seconds, each generated an XR. Thus, the base XR load was:
  • 573 per second from default FC 222 blocks
  • (2181 - 573) / 60 = 27 per second from the other tags
That is, the base XR load on the node is 600 XRs per second. It is a good thing this system has NIS 21 / NIS 22 communication cards. As it is, this showed that the average communication CPU usage in the node of about 50% was a valid figure.

Why worry about this?

554 of the 640 tags and export blocks had default significant change. If the defaulted FC 222 blocks had 1% significant change, the base load would nominally be 2181 / 60 sec or 36 XRs / sec. Allocating 600 XRs per second nominally would allow the analog tags to be increased in sensitivity by a factor of 20, with less load on the system than there is now.

This investigation also brought our attention to FC 223, the Analog Out/Channel block, which also has a default significant change of 0.

Guess what? There are 80 of these in PCU 2. They all have the default 0 significant change, so they are generating 1 exception report per second. 31 of them are spare blocks, so that load is doing nothing.
Bottom line is that, unbeknownst to our client, the exception report loading on the node included:
  • 96 analog in and 31 analog out spares giving 127 utterly wasted XRs per second
  • 469 analog in and 49 analog out tagged blocks giving 518 very, very, precise values every second.
  • 26 XRs per second from the other 1536 "low class" tags, most at default significant change.
Thus, the load was about 600 XRs per second. If the spare tags were deleted and the defaulted ones set to 1% significant change, the load would be 518 / 60 per second (typically) or 9.

This node would generate 35 XRs per second as currently tuned. There is lots of XR capability to improve its tags and history data using the capacity that is nearly completely wasted right now.

How many other nodes with FC 222 and FC 223 blocks have this bad situation?

The good:
  • Large Australian power plant has 10663. One has default significant change. It is also not tagged.
  • Large Canadian power plant has 1610. None of the 161 that have 0 significant change is tagged.
  • Large American power plant has 1312, none of which is defaulted.
  • Medium size Canadian power plant has 192, all with sig change set to 1.
The bad:
  • Large Australian process plant has 66 with default values. 16 are tagged, which is a meaningful load.
The ugly:
  • These small American power plants with all values defaulted at 0 and over one-third tagged but spare.
  • Small American power plant with 88 of 754 with default 0. 86 of these have tags, so they are a load. 21 are spares, so they are a wasted load.
  • Medium size Canadian process plant has 496, all at default 0. 216 are in PCU 171 and 280 in PCU 172. Every one is generating one XR per second.
  • Medium size American process plant has 559 FC 222, of which 368 are tagged with significant change 0. No spare ones are tagged.
It is clear that the significant change specifications and tags for all FC 222 and FC 223 blocks should be examined. The load can be large, and it can be a waste of a precious resource, even stalling or losing exception reports.

Postscript

The general situation with FC 222 and FC 223 blocks that are not defaulted is to have the significant change specification set to 1, no matter what the span.  This is probably an error, too, suggesting that the work was done under the misconception that the specification is a percentage value, rather than an EU one.  1 in a span of 5 is 20%, whereas 1 in a span of 1000 is 0.1%.  Since both appeared in the same system, with the only value used being 1, it is likely the values should be studied even when they are all not defaulted.

No comments:

Post a Comment