Wednesday, July 16, 2014

Your system depends on TSTQ to detect problems. Are these tests actually working?

I have taken the time to show and explain some of the many things that DBDOC checks for (and Composer does not). I believe there is no better tool for configuring INFI 90 than Composer, way ahead of WinCAD and DOS CADEWS. However, we have found that many errors exist "in the wild" and cannot stomach ignoring them.

One very common and pernicious error situation is when TSTQ blocks test the quality of signals that don't have quality. The test always succeeds, and this success allows whatever condition the test was designed to detect to go undetected. This is potentially dangerous, since the intention of some responsive action resulting from the detection of bad quality on a signal is silently thwarted.

To highlight problems of this sort, DBDOC checks to make sure that TSTQ block inputs actually can be bad quality, and reports errors when they are not. DBDOC also draws lines with quality in blue, making visual confirmation easy.

The following examples are from a system where DBDOC detected 82 instances of TSTQ being used to test signals without quality.

Overview:

In this first example, a signal called "LP EXT TO LP STM" is brought into the system to use in control. The TSTQ error found by DBDOC will cause bad quality on this signal to put part of the control loop into Manual, but leave part in Auto (not the designer's intention!).

Take a look at this image:


[A] Error message presented by DBDOC Error Browser

TSTQ 1SLPPC904@MI Module 1,02,02 Block 1668 tests Module 1,02,02 Block 1610 (FC 7), which does not have quality.

Clicking on the error in the Error Browser calls up the problem TSTQ block [1668].

[B] TSTQ block that is offending us

Input S1 is blue, showing that the signal can have bad quality. Input S2 is black, showing that the signal cannot carry quality ([E] on the second image shows where the quality was inadvertently stripped out).

[C] Intended action on bad quality of the second input

Bad quality is supposed to put the controller 1SLPPC904 "LP EXT PRESS CONT" in Manual Lock, but will not.

[D] Controller that is not handled correctly

Tag 1SLPPC904 "LP EXT PRESS CONT" will keep on in Auto, even if the input is bad.


As a contrast, here is a situation where bad quality is handled correctly, and will successfully trigger a controller to switch into manual.

[E] The Square Root function block that removes quality

The output of the square root block [1610], like all calculation and operation blocks, strips the quality attribute from the signal.

[F] Working action

Bad quality on the input correctly, with a 15 second delay, does cause an intended action.

[G] Intended action

After the delay, bad quality will put part of the control loop into Manual Lock.

[H] Controller that is handled correctly

Tag 1SLPFC904 "LP EXT TO LP STM HDR CONT" will be put into Manual if the input is bad.

Ramifications

Only 81 more of these TSTQ errors to analyze. Signals are never bad quality until they go bad quality. At that point, the plant is depending on the logic to protect it, and these error situations typically thwart the intent of the logic.

Wednesday, June 11, 2014

Coming soon: DBDOC 10.5.1, with NEW error sharing and improved SPlus/PGP and PG2 support

We're now in the home stretch for getting the next DBDOC release out the door.  DBDOC 10.5.1 includes major expansion of some of the new features introduced in DBDOC 10.5, as well as general bug fixes and feature improvements

There is good news for users of SPlus/PGP graphics.  As we receive feedback and gain experience from client systems using DBDOC 10.5,  support continues to be solidified, especially for systems converted from PPB and Conductor NT.

In addition, there is a whole new approach to 800xA graphics.  Using tools created for DBDOC by  ASSYST (included with your DBDOC installation) it's easy to extract more comprehensive PG2 or VB data for better integration of 800xA graphics into the DBDOC snapshot.

The other major new feature for DBDOC 10.5.1 is the introduction of error sharing.  Really this is just the first step along the road of allowing Hyperview users to share data in general.  Right now, each Hyperview user is relatively isolated, making independent annotation notes on projects, bookmarking significant items, creating and configuring Watch Window groups, and, as of DBDOC 10.5, managing and reviewing system errors in the new Error Browser.  The vision, however, is for all these activities to become more collaborative.  A whole group could take advantage of a predefined set of bookmarks, or view notes made by others.  It takes time to set up Watch Window groups, configured appropriately for monitoring particular situations.  Ideally these groups (and the data collected in them) could be easily shared among multiple users if desired.

So this is the direction Hyperview is heading, and the first example of this sort of sharing actually implemented is the sharing of error information, i.e. the stars and checks added to errors displayed in the Error Browser.

User data is shared via a shared folder accessible to multiple Hyperview users.  Usually this folder will be specified in BuildPlus, when a project is built, and will show up automatically in Hyperview.  To start sharing data with other users, all you need to do is "register" in this shared data folder.  Go to the Options menu in Hyperview, choose the Sharing tab, and (if a shared data folder has been built into your M14), just hit Apply and you're done.  If not, you can manually choose a data sharing folder.

 

When you open the Error Browser in 10.5.1, you will notice a new list on the left, listing all the users that have registered to share their error information.

 
By checking and the boxes next to the names of other users, you can cause their stars and checks to be displayed in your Error Browser.   By setting various filtering options, you have control over which errors you display.  For example, you could display only the errors starred or reviewed by a particular user.  Once the errors are displayed, you could select them all (for example) and hide them, or mark them as reviewed yourself.
 
This functionality is simple but powerful, because for the first time, it really makes it easy and convenient for groups of users to collaboratively review and manage errors.  A typical system will have hundreds of build errors, many of which are not serious in context, but all of which should be reviewed for safety.  The ability to split up and coordinate this error review process is extremely useful, and was requested by a number of early Error Browser users.
 
DBDOC 10.5.1 will be released and available within the next few weeks.  For an early look at the new features still under test, you can download DBDOC 10.5.1 Beta from the GMCL website.



Tuesday, June 3, 2014

DBDOC Visits the Peak District


Geoff and I are visiting clients in England for the next two weeks. On Monday we visited our first client, a cement works (Hope Works) in the Hope Valley, Derbyshire, a very pretty place. The villages are beautiful, set among large hills and occasional rock outcroppings rather than mountains. Of course, there are sheep everywhere. The weather was warm, approaching 28 degrees C with blue skies and sunshine as we started work in the morning.




The stone house is the B&B where we spent Sunday night. At several hundred years old it is an historic site, as the oldest of many similar ones are in the village. The ceilings are supported crosswise with large thick oak beams. In all very charming for our first night in England.

The sunshine, however, was not to last. By afternoon clouds had come up, and as we drove away in the late afternoon the rain started. Fog blurred the landscape and a picture of the rock outcroppings was not possible. The rain followed us to our next stopping place, Happy Guest Lodge at Warrington, Cheshire. Here we are today, with a day off, until we start visiting with clients for the rest of the week. Next week we will be visiting clients in the London area.

Thursday, May 1, 2014

Keeping XP secure as Microsoft's support slowly ceases

Today, Microsoft released a patch fixing a potentially serious (although some at Microsoft have called it "overblown") security bug that allowed for remote code execution through Internet Explorer 6 through 11. In their initial advisory for the issue, they listed affected Windows versions without mentioning XP. With this patch, they've apparently decided that it's important to patch XP too for this one. But rather than cause for celebration for XP users, this should be one last wake-up call; coming mere weeks after XP was dropped from support, Microsoft decided to patch through this time, but they're making no promises that there'll be a next time.

Here at GMCL, most of our physical XP machines have been retired, and we do most of our testing for it in hardware-accelerated virtual machines (which combines running a 'real' install with the ability to quickly snapshot and revert as desired). But a few machines still linger for one reason or another, so how do we approach maintaining them? Well, here's a few thoughts on the subject that I've arrived at, which I think apply generally:

  • Don't delay on upgrading: Obvious, but deserves stating first. Practical realities may make it seem too difficult, and one might imagine oneself to be secure, but even if XP was very amenable to being secured (and, in many folks' opions including mine, it really isn't), it's going to be an increasingly huge target. And even if you have an air gap in theory, in practice air gaps are nearly impossible to fully maintain; as Paul Ferguson of Trend Micro wrote in his white paper "Toward a More Secure Posture for Industrial Control System Networks", in "practical and operational terms...physically separating networks is not functionally nor operationally feasible in the real world." So for any systems still running XP, it should not be if you upgrade or replace them, but when.

  • Drop IE for Chrome or Firefox, wherever and whenever possible: Microsoft's policy for Internet Explorer updates is to tie them to support for the OS version in question, so while Internet Explorer versions for Server 2003 and 2003 R2 will continue to get updates until 2015-07-13, those that shipped with XP aren't officially guaranteed to recieve any more updates after April 8th---as Microsoft representatives have explicitly stated, today's update is "an exception". But Google's Chrome browser and the Mozilla Foundation's Firefox both will support Windows XP, and thus provide security updates on it, past the April 8th drop-dead date. Google is intending to update Chrome for Windows XP until at least April 2015, and Mozilla currently has no plans to discontinue support. Both browsers have proven in the past to be more secure out-of-the-box than Internet Explorer, and although contemporary versions of IE have significantly improved in this area (and many others), those versions won't run on XP anyways.

    You may well have internal sites or systems that rely on the older IE versions (especially if they involve ActiveX controls), but considering the web browser is perhaps the primary vector of infection, it is worth seriously evaluating the possibility of migration, and at very least using IE only in the specific instances you need to. In fact, for Google Chrome there is an official extension that can be installed on managed Chrome installations which allows you to specify automatic switching between IE and Chrome based on listed sites, thus allowing you to cordon off only specific internal sites to be browsed with IE.

  • Enabling Internet Explorer's Enhanced Security Configuration: To the degree that ditching Internet Explorer isn't an option (or that you truly prefer it as a browser), there's always the Enhanced Security Configuration. This is enabled by default on all versions of Windows Server, and can be enabled (piece by piece) on Windows XP. It beefs up security at the expense of some convenience--in fact, we've even run into some small bugs with the DBDOC Error Browser when ESC is enabled (relating to copy/paste). In this case, especially when XP is concerned, the convenience traded off seems potentially worth the added security, but that is indeed the classic dilemma.

  • Using the Enhanced Mitigation Experience Toolkit: Microsoft's EMET is a tool that hardens applications by preventing the kinds of behaviours and faults that are often exploited by malware. As such, it's quite useful for preventing unknown or unpatched vulnerabilities from being exploited. It's enabled per-application, so you can be granular and choosy with this one.

  • Uninstall anything you don't use: The converse of using the EMET to harden applications is to make sure any you don't actually need aren't there to be exploited. Alongside the news of the recent IE exploit, there's also an Adobe Flash exploit recently discovered in the wild. Many XP machines may have components like Flash and Java installed that they don't actually need for daily operations, and the machines in question will be safer without the additional attack surfaces lying in eager wait.

  • For anyone out there, what are your thoughts? Are you already working to upgrade? Is it out of your hands? Do you have any other suggestions for keeping XP as secure as possible as it sails well past its best-before date?

    Thursday, March 13, 2014

    Try DBDOC on a Tablet: It Just Works!

    Recently we brought a Surface Pro tablet into our office, and we've been pleasantly discovering that for the most part, DBDOC Just Works! Read on for more details.
     
     
    Lets look at Hyperview, which is the most likely of the DBDOC programs to be used with a tablet.

    First up, we have an option in Hyperview that happens to have prepared us for touchscreens entirely unintentionally. If you go into "Options..." you'll see in View the checkbox for "Large icon toolbar", which scales up the main buttons.
     

     

    Who knew we were preparing for touchscreens all this time? It's nice to be forward-thinking by accident!

    Most navigation works as you'd expect. Tap on the screen for left-click; long-press for a right-click, for example to call up a context menu on a block; scroll by flicking within a pane or by pressing down on a scrollbar and moving your finger. It can be a bit inconsistent sometimes, and we're investigating an issue where kinetic scrolling in the index pane stops working on some M14s, but in general it's quite fine with fingers or a stylus.

    Selecting an area to zoom into can be a bit tricky with the touchscreen, although I've found it to be fairly reliable if I start first with a distinctly horizontal motion, then start moving my finger vertically as well. Otherwise, Windows sometimes interprets the motion as an attempt to drag the viewspace. With the stylus that comes with the Surface Pro, however, box selection seems to work without hassle.

    If you're running Windows 8.1 on a relatively high-DPI device like the Surface Pro or a new high-end laptop, you may notice the fonts are a bit blurry. This is because Windows 8.1 by default scales up the UI element sizes when using a dense display, which is handy for keeping things readable (and for making it easier to touch interactive elements), but isn't done in a way that's terribly kind to applications built on the old standard Windows assumption of 96dpi. Right now, most third-party applications for Windows react in one of two ways: either upscaling with blurry fonts and other elements, or ignoring the scaling entirely. You can see this contrast with the current versions of Google Chrome and Mozilla Firefox for Windows, where the former looks the same as DBDOC does and the latter stays crisp but small in defiance of Windows' scaling settings.

    If you haven't noticed this (in which case, sorry for pointing it out!) you can leave it as-is, but if you're looking to disable scaling globally, you can find the option in "Control Panel\Appearance and Personalization\Display". You can also disable in on a per-application basis by right-clicking on a shortcut or an EXE, going into its properties, and choosing "Disable display scaling on high DPI settings" from the Compatibility tab.


     

    We're investigating and evaluating how we might hook DBDOC into the scaling functions of contemporary Windows, which will not only help with tablets like the Surface Pro, but also high-resolution desktop and laptop displays which are just now starting to edge into mainstream production; my own laptop has a resolution of 2560x1700 on a mere 13" screen!

    Well, that's the rote details of the current state of running Hyperview on Windows 8/8.1 tablet, but what's the actual use? Well, we've imagined that it could be quite helpful carrying around a plant, looking at live data (if you're connecting in through a local network or VPN) or even just to be able to easily browse your graphics and CAD sheets while standing in front of the actual machinery they represent. You could use CIUMon Relay to pass on data from a running CIUMon instance, or even hook up a USB-to-serial adapter to your tablet and plug in directly to a CIU with your tablet. And the guts of Microsoft's Surface Pro and Surface Pro 2 are fairly hefty, so you could even run BuildPlus to generate M14s if need be.

    We're just getting started on touchscreen and tablet support, so let us know what you would find useful and in what scenarios you can see using DBDOC on a tablet!

    Monday, January 13, 2014

    Are Dated Design Compromises Affecting Your HMI and History Data Precision?

    I got busy in the purported Christmas break and wrote a bunch of blogs.  This blog is a roadmap through several related areas. Read on to
    • learn if you are overloading the exception report generation of nodes
    • verify under-utilization and improve your console and history data
    • automatically detect and report exception report delays
    Most systems worry about the above, and most were configured a decade or two ago. Learn if you can get much more out of what you have.

    For many INFI 90 systems, the power of the modern NIS21/NPM22 hardware capability will allow you to get much more precise data to consoles and history systems.

    If you have DBDOC, you can find out, for every PCU / node that you think is heavily loaded, if it really is. Heavy loading is 50% to 90%, overloading above that.  This is how to do it:

    Monitoring Node Communication CPU Load

    If you find the communication CPU usage in the 10% range, you are not getting enough out of INFI 90 and are not giving operators, managers and engineers data that is good enough.  You can do better:

    Improved Tag and History Data Precision in INFI 90 Systems

    There is also a design for you to be able to permanently and validly measure whether you ever get delayed or lost exception reports.  If you find you have nodes in the 50% or greater node communication CPU usage, you should worry about this.

    Monitoring Exception Report Performance

    Have you tried any of these techniques?  It would be very interesting to hear your experiences.

    Friday, December 20, 2013

    The Art of DBDOC - Live Specs and NVRAM Failure

    Summary

    NVRAM failure can be a big problem.  Doing a DBDOC Live Loop Annotation fetching the block specifications will fail if the NVRAM has failed.  This could be useful information to you.

    Details

    A client reported that DBDOC Hyperview could not fetch the specs of a block.  In fact, the specs could not be fetched in any block in that module.  This is shown in this image.  Live data could be fetched - only the reporting of the specifications was affected.


    However, fetching specs worked fine in other modules.  The example in a clone of the problem one, as you can see.


    What could cause this problem? 

    The next two images show the difference between working and failing module status fetches. 




    Guess what?  The problem module shows NVRAM failure status: Fail.  The one that is working says: Good.

    From the Client

    "I believe a NVRAM failure will prevent looking at any of the configuration."

    Subsequent Perspective

    "The module will continue to operate with an NVRAM error because a copy of the configuration is held in SRAM and executes from there. The next time the module is reset it will not startup but will fail.

    "This module had failed earlier and was hard initialized during an outage and did not show the errors, but has NVRAM failure now."

    New INFI 90 "art" has been crafted.  This is the first time we know of that the module status fetch we created has been used to solve a problem.