Automating Automation

This post was originally published on www.tetranex.com, August 2016

Tetranex Solutions Inc. was incorporated 5 years ago based largely on the principle that we could make dramatic improvements to the way typical engineering projects were being executed in terms of quality, consistency and repeatability. It was obvious at first, that words were very easy to market, but actual follow through and actions were rare artifacts to be found in our industry. Our biggest dilemma: How to enforce quality, consistency and repeatability while, in parallel, encouraging internal innovation? As we explored our objective further, with specific regards to Control System Integration, we became faced with somewhat of a paradox.

The Inherent Paradox

Problem 1: A skilled control system programmer will constantly strive for a high level of personal engagement in the projects and tasks they are assigned. If unmonitored, this typically leads to a programmer developing their own configuration methods and styles that they are continually tweaking and revising as their experience and skills grow. This ultimately flies in the face of our mandate toward maintaining consistency. The resulting implementations are often difficult for anyone, other than the original programmer, to troubleshoot, maintain or expand.

Problem 2: Tetranex has an internal initiative toward innovation, with an objective of increasing the quality of our deliverables while, at the same time, reducing the time required to produce these deliverables. If our innovations constantly lead to modifications of our standard deliverables, we end up failing our mandate of repeatability. Blindly driving innovation often leads to simply making something different, but not necessarily better.

Solution: Build a quality control program around scrutinizing, testing and checking the project deliverables in line with our core goals of quality, consistency and repeatability (a.k.a. the destination). Give our resources the freedom and encouragement to innovate the means in which they generate these project deliverables (a.k.a. the journey). Simply put, develop a program to control the destinations, not the journeys.

Corporately, this solution was migrated into our Quality Management plan. For our controls integration team, the manifestation of this solution was the beginning of our ”Automation of Automation” program. We started off with a loose program charter, outlining a plan to consolidate project specific data, eliminate human error and quickly adapt to design changes. This charter was then implemented solely through mentorship, on a project by project basis, as opposed to a rigid corporate wide mandate. This led to the program evolving at a slow and steady rate. When we needed software libraries and applications for proof-of-concept, or to progress through expansive step-changes in this evolution, we utilized overhead development projects funded in part by the CRA’s Scientific Research and  Experimental Development Tax Incentive Program (SR&ED).

Early Lessons Learned and Program Adjustments

One of the initial goals of this program was to develop a suite of engineering software tools that would allow our resources to efficiently and effectively replicate standard control system configuration schemes across multiple control system platforms. What we learned early on was that this goal was ultimately defeating an important secondary benefit of keeping resources highly engaged. We were just moving the mundane task of configuring consistent control system code to the mundane task of executing our own engineering software against consolidated project specific data. While it was a vast improvement on the configuration quality and duration, it was doing nothing towards keeping our staff highly engaged in what they were doing.

We also learned that attempting to rigidly control the development of these engineering software tools was an unnecessary burden. All of our staff were eager to participate in the development of these tools but they all came with their own unique strengths in terms of programming languages and, similar to what was described in Problem 1, their own programming methods and styles. Tying a tool to a specific developer usually resulted in that individual being burdened with all future modifications and training for that specific tool. Our staff naturally learned to leverage the concepts and knowledge around each specific tool rather than depend on the tool itself. Extracting these concepts and knowledge from the original tool developer led to one of the most unique and open environments of mentorship I have ever witnessed within an organization.

The Key to Success: Staff Buy-In and Engagement

Getting our resources to buy into the concept was easy. We always start our resources off doing their configuration in the traditional manner. This keeps their attention while they learn the specifics of each control system platform, however, the challenge quickly diminishes when they have to start replicating these consistent templates throughout a project. Resources are then introduced to an easier way to reproduce logic by using basic replication methods, usually through excel and some rudimentary cell based formulas. Almost all modern ICS systems offer some degree of easily bulk building configuration which is accessible using csv files, excel files or text files through structured code representations.

Now that they have had exposure to “a better way”, they naturally become more engaged in their task once again. At this point, we gently nudge them to find an even more automated method of reproducing these template configurations. People with limited software background will naturally lean towards using Visual Basic for Applications (VBA) while those with some level of software experience will gravitate towards the other software languages they are already familiar with. Once they have experienced this “gateway drug” of automating automation, they now naturally become part of the mentorship environment throughout the rest of the team. Almost immediately, they are participating in both the mentor and mentee aspects based on what they have done to date and what they know they can achieve. In almost all cases, they quickly ditch VBA for more advanced and flexible programming languages like C#, Java or Python to further expand these skills.

Five Years Later

Five years in, we do not have a hard software suite of engineering software tools to automate automation across multiple ICS platforms. What we do have is a highly sophisticated team of individuals who have bought into the concept, are actively engaged in continually developing their own skills and continually finding ways to challenge themselves. These resources are now just as comfortable with manipulating XML, database administration, SQL queries, regular expressions and data mining as they are with the off-the-shelf ICS configuration software. All of their efforts are used to continually improve on our quality, consistency and repeatability; not to compromise it.

For Tetranex, the “Automation of Automation” has evolved from a program into a doctrine. What was originally intended to be a program to increase our quality while decreasing our project durations is now simply the way we execute our projects. Using these methods, we have been able to reduce our KPI, of configuration time to I/O count, by 50%. The new skills and methods the team now possess have surpassed the original intent of the program by a wide margin. As a senior resource with nearly 20 years of experience in the automation field, I am constantly in awe and humbled by the capabilities of this team. At Tetranex, we are excited to facilitate the development of this next generation of Controls Engineers and Integrators and their continued evolution.