Keep ignoring the elephant in the room...

Recently, I was invited to participate in the panel on the topic of “Smart Manufacturing” organized by NIST and APMS 2013 Conference (Advances in Production Management Systems).  Many panel participants were from the SMLC (Smart Manufacturing Leadership Coalition) – “a non-profit organization committed to overcome barriers to the development and deployment of Smart Manufacturing (SM) Systems through an implementation agenda for building a scaled, shared infrastructure called the Smart Manufacturing Platform (SM Platform)”.

The invitation was an honor, as it allowed the opportunity to learn and share our vision for Intelligent Manufacturing.  I spent some time preparing by reading available material on Smart Manufacturing, SMLC and APMS. Three words jump at me as I was reading – “Cloud”, “Platform” and “Workflow”. Savigent is a provider of a Platform for the Manufacturing Operations Management and our Platform includes a Workflow Engine and Workflow Development infrastructure. We also designed our platform to support various deployment scenarios, including cloud. At the first glance SMLC was talking about exactly the same driving vision as Savigent for the last 15 years – Platform, Components, Workflows, Models, etc.

As I kept reading, my excitement was very quickly cooled by lack of relevance of the SMLC actual vision and the real world. There was nothing addressing Level 3 activities! (Enter the Elephant). The comments that I encountered during my reading echoed:  “It is too difficult and complex…”

I agree, yes, it is! However, it will not get easier nor go away.  That complexity will always be a part of any manufacturing enterprise and it requires new paradigm to design, build, test, deploy and manage systems. Beyond this, a level of maturity and commitment from the customer is needed to be successful.

Manufacturers need to keep pace with the customer demand for decreased product release cycle. The question of how to introduce and absorb timely operational system changes without significant errors becomes critical. We need “Smart Manufacturing“ to enable decisions in a complex, high volume manufacturing environment.  In the manufacturing enterprise, different components of the operational system are architected, developed, managed and used by multitude of users in different roles, all with different goals, needs, views, and skills. Enabling agility to support continuous change while delivering high quality operational systems requires purposely designed Manufacturing Operations Management (MOM) Platform.

“Smart Manufacturing” is not simply a protocol or data model problem – the realities dictate that whatever standard is selected, new technologies will arrive, outdating a standard.  

It is our opinion that “Smart Manufacturing” Platform should include support for IO and system integration, process workflow, complex event processing, “Big Data Historian”, all while providing development and management tools that support frequent changes as part of continuous improvement initiatives.  These development and management tools should also accommodate different levels of user expertise, enhancing developers’ productivity and lowering the complexity bar for non-developers. It is also critical to accommodate variety of IT deployment scenarios from a dedicated appliance to a private/public cloud. 

Purposely designed “Smart Manufacturing” Platform is the key to unleashing efficiency and ensuring effectiveness. “Smart Manufacturing” is a requirement to enable Predictive Space – Predictive Maintenance, Virtual Metrology, Predictive Analytics, Dynamic Sampling and next generation of advanced analysis techniques to find the ‘’hidden signatures’’.

Need is pushing early adopters into the space already. Their lessons are invaluable to help others to follow. “Smart Manufacturing” is not an application, it is a platform to enable decisions in a complex, high volume manufacturing environment and it requires organizational maturity to recognize and embrace the elephant in the room.

The Jell-O Effect

This is third installment in the series of posts related to Reactive Agents. The previous can be found here: http://blog.savigent.com/2012/05/merovingian-or-simple-explanation-of.html , http://blog.savigent.com/2012/08/to-tackle-complexity-we-need-to.html

My daughter loves to play with Jell-O on her plate - it jiggles and slides, and after a while starts falling apart.  The slab of Jell-O on the plate always reminds me of the conversation I had just before joining the team at Savigent.

In the mid-90s I was talking to a customer who described the brand new MES implementation as having the “Jell-O Effect”. I asked him to elaborate, I had never heard the term before and back then I didn’t have kids to know better.  I didn’t have to ask twice. “We can’t make any changes to the system”, he said, “it jiggles in many unpredictable places every time we make a single change”, he continued.

I spent couple of weeks at the plant as part of software vendor’s application group, helping to stabilize the system and learning an invaluable lesson in large scale architecture and implementation of manufacturing systems.

The MES vendor’s product was a database with an API layer to performed transactions against materials, machines, routes, scrap, etc.  During sales demos, everything was done using manual data entry screens, shipped with the product. Reality was very different from the demo – customer’s plant had highly automated manufacturing floor with multiple workstations, each running dedicated SCADA application. The “obvious” decision was to modify each SCADA application on every workstation to perform MES transactions and minimize manual data entries for operators. The resulting implementation created tightly coupled, fragile system with business logic spread across MES configuration, database and each and every SCADA application on the plant floor.

As soon as I opened one of the SCADA applications in the design environment it became obvious that technology was misapplied. Tag-based, scan-based SCADA was pushed very hard in an attempt to reliably detect plant floor events and execute transactions against the MES database. There were special precautions not to miss events, code to handle database call exceptions, timeouts and retries, even attempts to do concurrency. The amount of code and extra creativity needed to implement all the functionality earned my appreciation, but also raised a red flag.  SCADA was not designed to do event-driven, concurrent, transactional execution.  

Using SCADA for MES integration resulted in complex, fragile system that was not supportable and maintainable.  It was impossible to follow data flows through the system. Multiple scripts were changing different tags, creating complex dependencies. If an extra line or two of code was inserted in the script, it would break the timeout logic. For a person who didn’t develop the original SCADA code, even minor change was equal to the “red vs. blue wire” question – something might blow up. I also suspected that even for people who built the system, changing it after a while was not a trivial task.

Side Note – back then there were very few options for the tool or technology to build the system. The selection was between the C/C++, that not a single person at the plant would be able to support, and SCADA.  Unfortunately I see same SCADA packages pushed to do MOM implementations event today, 15 years later.

The Jell-O Effect was a result of taking a technology designed for one, very specific domain and using it to solve a problem in very different and much more complicated domain. CIM, MES, MOM, Level 3 or whatever next label is going to be is by its nature event-driven, concurrent and distributed, in other words – it is very complex.

The lessons learned studying the “Jell-O Effect” triggered a search for a better way to build MOM systems. Reactive Agents quickly rise as the top contender in the search. As we discussed before – reactive agents were invented to address complexities in the event-driven, concurrent and distributed world of robotics and artificial intelligence (AI). There are a lot of commonalities between these two worlds:

• Interact with environment
• IO
• People
• Work to achieve goals
• Schedules
• Recipes
• Targets
• Rules
• Communicate with command and control and peers
• Services
• Protocols
• Networking
• Maintain state
• Variables
• Data stores

Using Agents to build MOM systems becomes a very easy decision once we analyze the above list. Reactive Agents give us a blueprint to address the shortcomings we find with conventional tools:

• Encapsulation of the state – there are no shared variables, each agent is a black box to the rest of the system, internal state is protected from concurrent modification
• Event-driven – each agent receives and generates events
• Concurrent execution – each agent runs concurrently with other agents, in response to events
• Formal interfaces – each agent can only receive and send predefined events (event-driven form of the service-oriented communication)
• Individual agent changes while may change the reaction of the agent will not produce other side effects in the system
• Adding new communication links between agents, or adding  new agents to the system can be done at any time, while preserving existing functionality

The first implementation of the Reactive Agents-based Platform was deployed in production in the fall of 1999. Savigent’s second generation, .Net-based Catalyst Platform is implemented at Fortune 500 manufacturers in demanding 24x7x365 manufacturing environments, all without the “Jell-O Effect”.

It's been awhile

It’s been awhile since we have posted on the blog, and for that we apologize.  However, we are committed to maintaining the blog and providing you relevant and frequent information in the manufacturing operations management space.  Since our last post, we have:

• Coauthored an article with researchers from IBM in the third collection of Manufacturing Operations Management (MOM) methodology white papers by ISA - "The Role of Semantic Models in Smarter Industrial Operations”. [More here]
• Jointly presented with our customer, Uponor at the 2013 Industry Week Best Plants / MESA Conference in Greenville, South Carolina [More here]. Great story from the plant manager’s view about lessons learned and success found when implementing MOM to enable agile, continues improvements environment. 
• Refining the next generation of our MOM Platform – Catalyst 5. New Platform includes all new Historian, designed to scale to multi-terabyte installations. Also all new, Information Model provides uniform access to real-time and historical information in the Platform and to any other sources that typically found in complex MOM implementations.