50% of Systems Have Certain Function Blocks Unintentionally Disabled. Is Yours One of Them?

DBDOC is unique in giving you significant help in finding integrity issues in your system. Every INFI 90 system in the world will benefit from improving safety, reliability and operation with DBDOC's Integrity Plus approach. Our Error Browser and Error Marker development was designed to make it possible to manage errors and improve the integrity of your systems. 

 

In our most recent versions, we have added tests for three function blocks that are disabled by default.

• FC 3 - Lead / Lag - S2 defaults to 0 disabling the function
• FC 8 - Rate Limiter - S2 defaults to 0 disabling the function
• FC 166 - Integrator - S4 defaults to 0 disabling the function

For a function block to be disabled by default is horrific. Only three of the 200 types or so are (the rest are enabled by default). The hapless users put function blocks in place, but they do nothing that they were intended to do, because they are disabled. Now and forever, they are booby-trapped.

What do you think? Can there be any justification for most function blocks being enabled by default, with only these three exceptions?

Am I wrong? Are there other booby-traps we have missed? Please let us know.

In my review of test data for 228 systems around the world, I found

• Lead / Lag - 605 disabled in 83 systems
• Rate Limiter - 392 disabled in 86 systems
• Integrator - 50 disabled in 26 systems

In actuality, 75 systems had one of these; 42 systems had two and 8 hit the TriFecta! In total 125 of the 228 systems had one to three of these errors - 55% of them.

What about the "real world" - places where DBDOC checking has not been used yet? Here is what this test detected in a small power plant being worked on now by extremely competent people:

• Module 1,02,02 Block 8507 FC8 disabled by spec values
• Module 1,02,02 Block 9541 FC8 disabled by spec values
• Module 1,02,02 Block 10295 FC8 disabled by spec values
• Module 1,02,02 Block 11049 FC8 disabled by spec values
• Module 1,02,02 Block 11803 FC8 disabled by spec values

 

Holy Rate Unlimiter, Batman! http://holysmokesbatman.com/directory contains 350 actual sayings, perhaps the best being "Holy Time Bomb".

 

The 5 errors out of 152 Rate Limiter blocks show that only DBDOC can protect you from these problems.

 

Here are examples of each from real systems.

 

The objective of this FC 3 Lead/Lag block is to apply a time constant of 60 seconds to changes in the PV as applied to the APID control algorithm. This is not happening here because, unbeknownst to (or unnoticed by) the DCS specialist, the block is disabled and feeds the input directly to the output.

 

The first Rate Limiter example looked like it was booby-trapped in the sense that sometime in the future an unsuspecting body would put meaningful values on the limits and expect them to work. As you can see here, the negative going rate limit of 0.00333 per second presumably is intended to prevent larger drops in the value from affecting the process. However, the block does not function.

 

 

 

In the case of the integrator, it is S4 that is the trap. Clearly, nobody puts an integrator and tag into a system not intending to get data. This one will not get data because S4 is not wired to a constant value 1.

 

Only DBDOC can improve your system integrity by detecting these relatively common oversights.

 



Are Dated Design Compromises Affecting Your HMI and History Data Precision?

I got busy in the purported Christmas break and wrote a bunch of blogs.  This blog is a roadmap through several related areas. Read on to

• learn if you are overloading the exception report generation of nodes
• verify under-utilization and improve your console and history data
• automatically detect and report exception report delays

Most systems worry about the above, and most were configured a decade or two ago. Learn if you can get much more out of what you have.

For many INFI 90 systems, the power of the modern NIS21/NPM22 hardware capability will allow you to get much more precise data to consoles and history systems.

If you have DBDOC, you can find out, for every PCU / node that you think is heavily loaded, if it really is. Heavy loading is 50% to 90%, overloading above that.  This is how to do it:

Monitoring Node Communication CPU Load

If you find the communication CPU usage in the 10% range, you are not getting enough out of INFI 90 and are not giving operators, managers and engineers data that is good enough.  You can do better:

Improved Tag and History Data Precision in INFI 90 Systems

There is also a design for you to be able to permanently and validly measure whether you ever get delayed or lost exception reports.  If you find you have nodes in the 50% or greater node communication CPU usage, you should worry about this.

Monitoring Exception Report Performance

Have you tried any of these techniques?  It would be very interesting to hear your experiences.

Improved Tag and History Data Precision in INFI 90 Systems

INFI 90 is based on the concept of Exception Reports (XRs) that reduce the communication load needed to get good values to HMI systems, historians, OPC servers and to be used in other parts of the system.  Analog values have a "significant change specification" to prevent load from analog values that are not changing significantly.

History

This history is approximate, as told to me by gurus over the past years.  

Net 90 and INFI 90 started with a 1 MBaud plant loop communication ring, plus the impediment that a single XR was sent for each enrolled user.  These two factors made it easy to overload the capacity of the system.

Infi Loop introduced both a 10 MBaud communication ring and a protocol that allowed multiple destinations on the ring for an XR, so the load caused by an exception report no longer changed if more consoles used the tag, for example.

Pragma

Practical considerations were developed to avoid the loss of exception reports in the early systems, and developed further as the systems got both bigger and faster.  Bigger, of course, meant more load, but faster meant more capability.  Initially, the practical suggestion and default was that a change of 1% of span in and analog value was a good compromise.  Sensors were often enough not very accurate and it seemed logical.

The Modern World

In the modern world, there are aspects you should take into account:

1. INFI 90 communication capability increases have not usually been translated into more precise data.
2. Sensors have often changed from 2 1/2 digits to 3 1/2 digits, ten times as precise.
3. Techniques now exist to easily monitor exception report delay and communication loading.
4. Decisions based on data that is not sensitive enough are made regularly, probably with affecting your plant negatively at times.  This applies to both console operations and historical data.

A Case Study

In a plant, examining the DBDOC Significant Change Report, we noted a flow rate with a span of 7000 and default 1% significant change.  Because it was being imported into another module by exception report, we could study the suitability of the significant change setting.

There were 23 exception reports in the period shown, 18 of them at Tmax of 75 seconds.

Here are some salient aspects of the raw and imported values:

• Raw value is the pink line.
• Imported value is the blue stepped line.
• Working range is under 400.
• Tmax is 75 seconds, and all positive-going exception reports were caused by the timeout.
• The negative-going exception reports were triggered by the drop of 70 before 75 seconds had elapsed.

The green sections show where the imported exception report value was less than the actual value.  The orange ones show where it was greater.  Interestingly enough (to me, anyhow), I had joked in front of clients about how bad it would be to be integrating exactly this sort of a variable.  You would find the accountants for the customer very happy when you told them you had used the blue step function to decide how much they should pay.  They would pay fast, and not quibble.

Out of curiosity, I estimated the performance using 0.1% significant change.  I did this by simply taking the raw value as a start and making a pseudo-XR when the value increased or decreased by 7.0 or more.  This gave me an estimate of how the imported values would look and what the performance would be.  The number of exception reports would have been about 140 in 1420 seconds, or about 6 per minute.  Only one period of 75 seconds went by when there would not have been a significant change. 

What does this look like from a numerical perspective?

Significant Change       1%      0.1%

Mean error            -8.1      -1.3

% of full span       -0.12%    -0.02%

% of working range   -2.02%    -0.34%

Mean error magnitude  29.0       3.5

% of full span        0.41%     0.05%

% of working range    7.24%     0.87%

The Bottom Line

By using the data available through DBDOC Watch Window (or Composer Trending, or any other block value monitoring package), you can see that it is perfectly possible for an exception reported value to be handled in a way that could be problematical.  The data comes from an exception report import block, but applies to every value on every graphic, and to every historical value in every INFI 90 system.

At a small cost in increased exception reports, using capacity that can be verified as being available safely, process values can be much more precise.  The full analysis that is done here is not necessary.  What you need to do is simply:

• Identify imported values, tags and historical data that needs more precision
• Monitor the node communication loading
• Where there is capacity, use it by making the significant change specification tighter

The improvements in console, history and perhaps control and shutdowns will be significant.  You probably have the capacity to do this right now.

Monitoring Node Communication CPU Load

INFI 90 systems are characterized by PCUs that are Nodes on a Ring.  Traditional systems include a NIS/NPM pair that handles the communication between the Modules on the PCU/Node and the rest of the system.  The performance of this interface can be monitored trivially using WatchWindow in DBDOC Hyperview.

Note that ABB Technical Bulletin TB1999054A gives details on the various performance statistics available from a node.  It also specifically mentions that this capability was introduced in INNPM01 firmware revision C.1, so it is not available in older versions.  It being at least a decade and a half since that firmware revision was introduced, let's hope nobody will be disappointed by not being able to do this.

Using DBDOC Watch Window to Monitor Communication CPU Usage

To see what percent of the communication module's processing power is being used, you simply monitor Loop L, PCU P, Module 0, Block 11.  The example shows Loop 1 PCU 4 being monitored and the monitoring of Loop 1 PCU 6 being defined.

Notes about this monitoring:

1. This fetch is never done in turbo mode, so you only have 10 to 15 values per second available, and these are shared with all your other Hyperview users.  
2. Running for one or two "Tmax" periods for exception report flushing should give you a good picture.
3. Turn off the Watch Window data collection by clicking on the green clock icon when you have a picture of the loading.
4. If you do a save of a module in the PCU while you are monitoring, you will get an idea of the maximum communication loading possible in the node.

Interpretation of the Data

Experimentation has shown that CPU loading above 90% shows a situation where exception reports (XRs) can be delayed or lost.  This is the bad side.

What about if you find the loading very low - under 10%? This is the good side. We believe this means you have a lot more exception report generation capability than you are using. Your system is capable of significantly better performance with respect to tighter values for the HMI and for the history system. Pushing the communication usage out of the idling range will potentially be very beneficial.

You might refer to Improved Tag and History Data Precision in INFI 90 Systems for more on what you can do to understand what you might be suffering from and how to get more out of your system.

Results

We would be happy to get a copy of the Watch Window data, which is in a .CSV file that is created by the monitoring.

Disclaimer

The monitoring indicated has been done on dozens of DBDOC systems with no known problems.  It is the same monitoring as done by Process Portal B and gets the same result (of course).  Monitoring only one block makes the load as minimal as possible.  However, if you have any question or hesitation, your consultation with your ABB support specialist should give you more insight.  We are very willing to work with your experts.