So what can ISA-95 do for ME?

I had a conversation the other day, in which somebody made the comment “well there is the S95 standard, but it really doesn’t do anything for end users…” [I think I just heard screams of pain].

It is true that the ISA-95 Standard is, well a standard… as such, it is geared to bring commonality to the way that we do things.  Furthermore it is broad to allow for flexibility within the space (because not all manufacturers are the same).  As a standard it has been adopted [or used as a marketing point] for Vendors that sell/develop software products, which allow for open systems that can/should be able to communicate to other systems.

However, there are additional ways that end users can benefit or use the standard directly.  First, make use of the Hierarchy Model outlined in Part 1.  Use this model to determine ownership of systems and dividing lines between groups within your organization (such as Engineering and IT).  Second, use the standard to help define user requirements for software design or specifications.  The standard provides a good listing of all activities and definitions that can help identify what might be needed and in scope verses not needed [or needed yet].  In the next posts I will go deeper into each of these.

Quick Overview of the Parts of The ISA-95 Standard

The ANSI/ISA-95 Standard is currently divided among 5 parts with plans for a 6th part. Each of these “Parts” has a different focus, and with roughly 300 pages each, that can seem confusing. I put together a quick video that will [hopefully] give you a taste for the contents of each and how they relate to each other.  Over the next several weeks, I will dive deeper into each Part and give more detail on the contents and why you should become familiar with the Standard.

Microsoft In Process Manufacturing

Microsoft announced the launch of the ChemRA initiative, an endeavor led by Microsoft and its industry partners. 

“This initiative is not a Microsoft product-mandated system.  Rather, it seeks to develop an IT framework that allows for the easy flow of information across organizations.  ChemRA is based on a set of principles that map to the most common use cases of technology for users in the chemical and oil refining industries.” 

The five pillars of ChemRA are:  Natural User Experience, Application Interoperability, Enhanced Collaboration, Business Insight, Solid Infrastructure. 

The five pillars of ChemRA

For more information vist:http://www.microsoft.com/enterprise/industry/process-manufacturing/default.aspx andhttp://www.microsoft.com/enterprise/industry/process-manufacturing/solutions/chemra.aspx

MESA – “Cloudy With A Chance For Profits”

Greetings from Orlando, Florida where I am attending the 2011 MESA North American Conference.

This year some of topics slated for discussion are:The Cloud, Chance for Profits (the theme of turning problems into solutions to get us all thinking differently), Cut Through the Clutter (the good, the bad and the ugly about implementations) and Real Time (what is real time in my business?).  I providing key highlights in the weeks to come.  Check back tomorrow for a recap of an announcement from Microsoft.

For those not familiar, MESA (Manufacturing Enterprise Solutions Association) International is a global community of manufacturers, producers, industry leaders and solution providers who are focused on improving Operations Management capabilities through the effective application of technology solutions and best practices.

Stuxnet and What it Means to Our Security

If you have paid much attention at all to Control Systems news in the past year, you undoubtedly have heard of the Stuxnet worm.  For those of you not in the know, the worm was found to have infected many industrial systems that ran Siemens PCS7.  After analysis of the software and its payload, it was discovered that the worm was a targeted attack against a system exhibiting certain characteristics.  When it found these markers, the worm would inject itself into the Siemens PLC running the process and surreptitiously change VFD settings on certain drives.

It was speculated, then later confirmed, that the target of the attack was the Natanz nuclear facility in Iran, specifically destroying centrifuges used to enrich uranium.  The attack appeared successful in that hundreds of centrifuges were damaged at the site, causing replacements to be needed, and stalling, for a while, enrichment programs.

Should we fear an attack like this? Not really.  This was a highly sophisticated attack, which probably had some sort of government backing and resources that are out of reach to the large majority of hackers.  If a group is dedicated and has sufficient funding and time, they will be able to break through your security.  Our job is to make that threshold so high that it is unreasonable for them to do so.

The only sure-fire way of locking down your system is to completely disconnect it from the outside and disallow physical access to the controllers/servers.  This, of course, is impossible to do as more and more systems are being tied together using SCADA software so that real-time monitoring can be used on the enterprise level.  Listed below are some simple strategies that are effective, and when used together, forms stronger security by creating a “defense in depth” strategy.  This is not an exhaustive list, but it will get you started heading in the right direction:

Segregate your networks

You should never have your control and business networks on the same network.  The security implications are obvious in that one malicious email attachment could bring down both networks, but there are also performance gains that can be achieved by separating the two.

Utilize DMZs

A DMZ (demilitarized zone) is a section of a network which can be accessed by both your control network and your business network. It provides an intermediate layer of security in that the business network can only access certain servers that reside in the DMZ, such as a data historian, and the control network can push data into this DMZ, but the control and business networks never speak directly.

Anti-Virus

Anti-virus (AV) integration into control system networks can be a tricky thing.  For anti-virus products to be effective, they need regular updates to stay on top of new attacks.  In a locked-down or validated system, patching is almost non-existent and anti-virus products would not get the updates they need.  Another problem with AV solutions is that vendors require certain files and folders to be excluded from scans in order for the products to play nicely together.  This can cause a system to lose responsiveness and AV effectiveness can be lost.

One way to utilize anti-virus products is to have it sitting on a gateway server, so that any files transferring in and out of the system must pass through and be scanned before being allowed into the main servers.  This server could also vet any USB drives or CDs that would be used on the other servers.

Deny Access by Default

Configuring firewalls between networks is something that many companies fail to do adequately.  Many configurations are rushed, leaving them incomplete with gaping security holes.  It’s akin to barricading your front door while leaving your window wide open.  The best policy is to deny all traffic by default, and only allow connections on an exception basis, a concept called ‘whitelisting’.  This may be time consuming, as you need to figure out exactly what traffic or programs are necessary to allow through the firewall, but it provides much better security overall.

Restrict Physical Access

You’d be surprised how many installations have very good IT infrastructure security, but allow anyone to be able to walk up to a cabinet in the field and hook up their laptop directly to the PLC or network switch.  Simple solutions, such as locking control panels, and allowing only certain pre-screened engineering laptops on the control network can increase security and stop the proliferation of harmful worms and viruses.

Disable USB/CD Autoplay

The original vector for Stuxnet was through infected USB drives that integrators took with them around the world and plugged into control systems.  It is good practice to disable Autoplay in Windows, so these infections are not spread through merely inserting a USB device.

To disable Autoplay on Windows XP:

1. Bring up the Run prompt using Win+R
2. Type gpedit.msc and press Ok.
3. Navigate to Local Computer Policy > Computer Configuration > Administrative Templates > System
4. Under the settings in the right-hand window pane, double-clickTurn off Autoplay
5. Select the Enabled radio button and select All Drives from the drop-down menu to disable Autoplay on all drives.
6. Press OK.

I believe lasting legacy of Stuxnet will not be that of a new era of attacks on control systems, but an era of focusing more on the security of these systems. For too long has the industry relied on security through obscurity; it’s time to be more proactive in our security practices.

Below are some links for further reading about industrial control system security:

• www.tofinosecurity.com – ICS security devices and information
• http://www.digitalbond.com/ – Lots of good ICS information
• http://www.us-cert.gov/control_systems/ics-cert/ – US Government’s Industrial Control Systems Cyber Response Team

[Original post by:  Kevin Rawls] 

Why did “X” happen? (Or why did “X” not happen…)

Posts have been scarce lately… But there is plenty to come!  Now without further delay.

There is an easy and overlooked method to provide first steps to the question “Why did X happen? Or why did X not happen?”.   Not only is it an easy method, it is already there waiting for you to review…  The Windows Event Viewer!  OK, some of you are thinking, “Oh, that gee – so?”  For those of you who didn’t think of it, you are not alone… the Windows Event Viewer (or Dr. Watson for those reminiscing) provides detailed information about significant events on your computer. It can be helpful (and yet overlooked) when troubleshooting problems and errors with Windows and other programs.  A key point that I will make here is to look for information on the application and dependant technologies that the application uses:  DCOM, MSDTC and such for correlations.

Event Viewer

For a quick refresher, you can access the Event Viewer several ways, but I typically typically click Start, Run, and type eventvwr. There are typically three logs available:

• Application: applications running under Windows are supposed to log their events here.
• Security: when enabled Windows, can log a host of security-related events which are logged here.
• System: the operating system logs its events here.
• If you are really luckily, the MES or SCADA system you use might create its own heading too… (really, really lucky).

For those of you who haven’t bothered to look in the Event Viewer – don’t panic on your first view – there will be informational events logged aside from errors.  THIS IS NORMAL!  Another point is get a feel for what events are logged under NORMAL conditions so that you don’t chase a dead end for a missing printer driver (or somebody RDPing).

Now hop to it and check it out! 

Industrial Ethernet Reliability and Performance: Cisco’s “Errdisable” Functionality

Do you use Cisco Catalyst switches (or Rockwell Automation’s Stratix series of managed switches) on your network?  Have you ever had a port stop working, never to start again?  If so, there is probably nothing at all wrong with your switch.

Before I became acquainted with the Cisco IOS(Internetwork Operating System), I made the same mistake many people do; if a port stops working and I can get my device working again by just moving the connection to another port, the port must be bad.  In my experience with Cisco switches, this is rarely the case.  However, there is a feature that is enabled by default on many Cisco devices called ErrDisable.  This feature is designed to detect network problems and stop them before the rest of the network is affected.  The default behavior is to disable the port in question until  someone intervenes.  In order to re-enable the port, an administrator would have to issue the shutdown command followed by the no shutdown command.  There is also a feature that allows the user to set a recovery interval for the errdisabled state.  If the recovery interval is set, the switch will, on a periodic basis, check the disabled port to see if the error condition still exists.  If the error condition has cleared, the port will be re-enabled.

The guidance provided by Cisco and Rockwell is to set the recovery interval using the errdisable recover interval seconds command.  In conjunction with the errdisable recovery cause errortype command, the recovery configuration can be very granular based on the type of error encountered.  Playing devil’s advocate, I could argue that configuration of the errdisable recovery feature may cause further problems unless the switch logs are being monitored on a regular basis.  Assuming you have an intermittent hardware problem such as a sloppy cable termination that is causing a link flap (a condition in which the physical link is broken more than 5 times in 10 seconds, easily caused by poor terminations and vibration in an industrial environment).  In this case, if errdisable recovery has been established, the problem may never be discovered until there is a catastrophic failure, resulting in manufacturing downtime as opposed to  scheduled maintenance.  My point is, just because recovery keeps data flowing in the short term, the asumption that no problem exists cannot be made.

Monitoring is essential to technology systems reliability, but that is a whole other topic.  Here is a document that outlines some of Cisco and Rockwell Automation’s guidelines for plantwide ethernet: http://cisco.biz/en/US/docs/solutions/Verticals/CPwE/CPwE_DIG.pdf

Detailed information about the errdisabled state per Cisco’s documentation library: http://www.cisco.com/en/US/tech/tk389/tk621/technologies_tech_note09186a00806cd87b.shtml

[ Original Post by Jed Leviner]  

World Batch Forum 2011 Recap

I had the privilege of presenting at WBF this year.  There were a lot of good presentations this year.  For those who didn’t get to go, I’ll give a couple one-liner recaps of the most memorable parts (aka the parts I can still remember!).

I presented on a project we completed a little while ago.  It’s a 10,000 foot overview of the facility, the MES system, and the ties to the ERP system.  It was a great project to be on from a programmer’s point of view.  There were LOTS of cool problems to come up with even cooler solutions to.

I thought both keynote speakers were very good.  I won’t recap their presentations because they’re actually posted on the web.  If you get a chance, their well worth a listen (see links below).

Jim Porter’s Keynote Speech
John Berra’s Keynote Speech

Dennis Brandl got dressed up in a #88 race car driver suit.  His presentation was on an S88 implementation for a pharma packaging line.

The Honeywell guys had a lot of interesting thoughts on visualization of the future & good UIs.  They tossed out there the possibility of using the Xbox Kinect as an operator interface.  They focused on the face that (A) it’s a pretty darn cool UI device & (B) it’s really cheap for what it can do.  They also tossed out there using an iPad as a tablet HMI.  The catch is that it would use the camera to scan a barcode on the equipment or recognize the equipment itself & automatically load the right screen.

A British company presented on their PAT software.  In short, it analyzes mounds of multi-variable historical data and present it to the operator to make decisions on at run time.  Seemed like a cool piece of software at any rate.

A guy from AB had a presentation on how they implement S88 from a spreadsheet.  Each “task” is a bit in a step.  Each step does one or more “tasks” in parallel.  While the concept is really cool, it still doesn’t escape from the fact that the logic hasn’t been simplified.  What it does buy you is a structure that is much easier to reconfigure for new products.

Dave Chappell had a good presentation how to apply the GAMP V-model to a project to help reduce & mitigate risks during various types of projects (i.e. pure implementation vs R&D and the spectrum in between).

Finally, WBF is coming out with a series of books that looked well put together.  I didn’t get a chance to even skim through them, but they should be pretty good.  You can check it out here.

[ Original Post by David Goodman]  

What is in a Model? (The first in a multipart series…)

There are several models out there to help define the roles of a MES/MOM system and “the space” that such a system fills.  But why?  Because it is difficult to talk about…  It is no secret that not all people are alike, let alone have the same backgrounds.  As this “space” touches several groups within the company’s organization (again who have different backgrounds / interests / terminology), communication is often difficult.  With difficulty, often comes higher cost.  This is where Models can lend a helping hand!

In this “space” there are two sources of models that can assist:  MESA and ISA. 

MESA (Manufacturing Enterprise Solutions Association) International is a global community of manufacturers, producers, industry leaders and solution providers who are focused on improving Operations Management.  MESA International provides several models that can help visualize the functions that are typical within the MES/MOM area.

Probably, the most recognized is the MESA-11, first introduced in 1997 (pictured below).  The MESA-11 model has gone thru some refinements, and now is aligned to several Strategic Initiatives (also below).

MESA-11 Model

MESA Model, Version2.1

Please visit  MESA.ORG formore information.

Next time I will introduce the S-95 standard written by ISA and adopted by ANSI.

Industrial Ethernet Reliability and Performance: Cable Terminations

This post may seem very obvious to some and completely foreign to others.  The majority of Ethernet infrastructure is general Cat 5/5e/6 (from here on I will refer to the categories only as Cat5) cabling.  Terminations for each category of cabling are essentially the same and often done incorrectly; it is something I have seen in manufacturing facilities time and time again.
There are generally two types of Cat5 cabling, solid core and stranded.  This is referring to the copper conductor in the eight wires that make up the cable.  Each type has a specific use.  Solid core Cat5 is intended for permanent installation, such as from a patch panel to another patch panel (or other permanently mounted termination such as a wall box).  Most often solid core wire is terminated by pressing each of the eight small wires between an individual set of blades that slice through the insulation and make contact with the copper conductor.  Another consideration with solid core wire is movement.  The solid conductors have a much higher possibility of breaking from excessive movement than stranded cable.  Solid core wire can be terminated with a male RJ-45 connector, but the connector must be specifically designed for solid core wire.
Stranded core cable is intended for port to field device connections.  The stranded wire can stand up to far more movement and vibration and has a tighter acceptable bend radius than solid core cable.  Stranded cabling is typically used in patch cables and terminated with a male RJ-45 connector.  Stranded cable is not intended for long runs as the electrical performance is poorer than that of solid core cable.

Finally, the quality of the Cat5 cable used can make a significant difference in performance.  The outer jacketing should be pliable.  Some of the cheaper brands have a very brittle outer jacketing that can break in tight bends and get damaged as it is pulled through conduit leading to damage of the underlying twisted pairs.  Another feature that I find important is bonded pairs.  This means that the two wires in each of the four pairs are physical bonded to one another.  The benefit of this is that the twist construction of the cable is maintained better as the cable is bent.  Cable that does not feature pair bonding can actually have the pairs separate in bends which reduces cross talk cancelation.

[ Original Post by Jed Leviner] 

ARCHIVED COMMENTS:

1. Jeremy Gillett says:

   June 6, 2011 at 1:13 pm

   The use of the correct Ethernet cabling can be critical to creating a reliable network connection. I have been working as an engineer in the industrial automation controls field for more than 10 years. I have seen several occurrences where solid core wiring has not been landed to a stabile connection point or like the example of what not to use above. The outcome was a network connection that over time failed. The worst part has been the way the failures have occurred. For example we had a customer that called us in to help troubleshoot a drive problem. They have a few networked variable speed drives that were occasionally faulting. This was causing their cold rooms to fall out of spec limits. One of the main causes of the drives faulting ended up being the occasional loss of the network connectivity, and a simple correction to the Ethernet connection fixed the issue.
2. Vikram Pungaliya says:

   October 31, 2011 at 11:57 am

   Interesting article that was.
   One more point that i would like to add is the pins being used.
   1,2 form one twisted pair and 3,6 form another twisted par.
   Hence the wiring on both sides to be maintained in same pair.

An interesting approach to why do EBR…

I recently attended a good presentation on EBR systems [ok full disclosure, it was a system that we installed and a fellow colleague was presenting].  Naturally, a Q&A session followed the presentation; during the Q&A somebody asked a question about the payback. 

The interesting thing about the question was the frame of reference of the asker; paraphrasing the exchange…

Asker:  “How much reduction in cycle time did the EBR system provide to the operators”?
Presenter:  “None really… it was a wash.”
Asker: “Then why do it?”

I believe that the asker was from a CPG background and thusly so, was of the mindset that EBR/Work ticket systems should let to a direct labor time savings.  Of course the EBR system will lead to time savings, but where is the question.  [As the presenters, did answer]Some of the savings is in OPERATOR direct labor (transcription time, manual logging time, etc.) however, it is in QA REVIEW time that is dramatically reduced.  Naturally there are some other reasons supporting a system like this:

• •Better Data
  • •Improved accuracy and consistency of the batch record
  • •Increased speed of product introductions and process changes
• •Production
  • •Reduced cost of compliance
  • •(Some) Increased productivity – i.e. verified by

The major point here is there are a wide variety of reasons that systems are put in place, MESA has defined these as Strategic Initiatives, some of which are Lean Manufacturing, Quality and Regulatory Compliance, Product Lifecycle Management, Real Time Enterprise, Asset Performance and etc.  When considering implementing those, make sure to look up and down stream to fully recognize impacts and capture all benefits.

Industrial Ethernet Reliability and Performance: Multicasting

Industrial Ethernet works on the same principles and protocols as any other Ethernet network.  In a nutshell, devices place units of data on the wire called packets.  Packets generally have a source and a destination.  When one device sends a packet directly to a second device, the process is called unicasting.  When a device sends a packet to all other devices on the network, the process is called broadcasting.  Finally, when a device sends a packet to specific group of other devices on the network, the process is called multicasting.

You might be asking yourself:  “where would multicasting be used?” One example that is often used in the IT world is a live camera feed.  If twenty people want to view a live camera feed, the camera shouldn’t have to manage twenty individual conversations, so instead, the twenty devices that will be showing the feed subscribe to a multicast group associated with the camera.  The camera sends packets to a multicast group destination address and the devices subscribed to that group receive and process those packets.

So, where does this apply in an industrial controls network? The first example that comes to mind is the use of producer and consumer tags on Rockwell Automation’s Controllogix platform.  One processor is a producer while one or more are consumers.  The consumers subscribe to a multicast group served by the producer.

The potential problem with all of this is that many switches handlemulticasts as broadcasts which creates a large volume of traffic on the network and can cause performance problems due to the packets being forwarded to every port on the network.  The simplest solution to this is to implement IGMP (Internet Group Messaging Protocol) snooping.  This is a feature that is available on some managed industrial Ethernet switches and on many standard duty managed switches.  Once IGMP snooping is enabled, the switch will remember which ports have devices that are members of a particular multicast group and will forward multicast packets only to the devices that should be receiving them.  This will greatly reduce network traffic especially if you have a large number of devices utilizing multicasting on your network.

[ Original Post by Jed Leviner]

Paper-on-Glass? Don’t Throw Your Money Away

I recently generated the content for a mailing piece targeted at our Life Sciences Clients here at Avid addressing Paper-On-Glass… This is something that I feel strongly about, and decided to re-host here – so without further rambling…

For many, the concept of Paper-on-Glass (simply replacing the existing paper batch ticket/ink with a computer screen, i.e. glass) seems like the logical, low-risk evolution of automation systems.

Of course, we all know that implementing even this seemingly low-risk evolution will require a large team, representing every facet of the business (operations, quality assurance, IT, engineering and validation), due to the impact of the change. But a Paper-on-Glass system is worth it, because it can generate an EBR (electronic batch record), right?

The real question should be: While that team is together, can something better be done?
Paper-on-glass systems (like typical paper-based) require review after the fact and not in real-time (or by exception). The next step up, from Paper-on-Glass, focuses the exception handling during the EBR execution. Reviewing the process during the manufacturing process allows the operator on the shop floor to address exceptions as the manufacturing rules are enforced in real-time. This reduces wasted time, money and materials and decreases the review time.
Any change to the batch ticket will require cultural transition and careful change management.  Again, why limit your focus to simple remove the paper? Some additional areas to consider with the implementation of any EBR system should be:

• •  Total Quality Control – Tracking material lots and manufacturing within the process
• •  Scheduling – Queuing and optimizing production based on best-fit algorithms
• •  Recipe Management – more than weigh and dispense, formulation, set-up parameters, tolerances and labels, and prepositioning of set points
• •  Inventory Management – tracking “work-in-process” and state of assets, i.e. clean, used, in-process
• •  Maintenance – Monitoring equipment performance, tracking usage for preventive maintenance and downtime

Welcome to Level3!

Welcome to Level3…a manufacturing information space blog.  Here we will discuss a variety of topics related to the space that sits between the PLCs/DCSs controlling your manufacturing process and the Enterprise Resource Planning (ERP).  We will tend to focus on integrating from the ‘Shop-Floor’ up as opposed to ERP down or “Big Bang” MES packages.  Posts will be generated by several of us here at Avid, some leading experts outside of Avid, and hopefully comments by users too.

Some of the topics we have planned are:  Historians, Reporting, OEE, KPIs, Networking, Virtualization and so forth…