Posts by lordandc

Keeping drinking water flowing and maintaining proper wastewater processing for millions of Americans is no easy task. Municipalities around the country typically have numerous remote pump and water distribution stations that must operate reliably to properly distribute water and manage the treatment of wastewater. To ensure optimal operation, municipalities continue to iterate on the deployment of supervisory control and data acquisition (SCADA) systems that eliminate the need to have operators monitor each station in person around the clock. But, as with any technological advance, remote SCADA HMIs present new challenges for municipalities to manage and host these systems.

While municipalities have experts to manage water distribution and wastewater treatment, they do not always have in-house experts to maintain their SCADA system. As a result, important tasks such as performing upgrades that keep the system running efficiently or focusing on the security of data being transferred are not always a priority. Therefore, working with a third-party integrator that specializes in the implementation and hosting of fully-integrated SCADA systems, such as Lord & Company, can be a more time- and cost-effective approach versus trying to host and maintain your own SCADA system.

Figure 1. An illustration of how all the components of a SCADA system interact.

The Lord & Company Approach: SMS SCADA

Internet of Things (IoT) technology has presented a tremendous opportunity to further simplify SCADA system management and hosting, especially for water and wastewater municipalities. At Lord & Company, we’ve used our decades of experience, talent, and our commonsense approach to SCADA to develop a cost-effective and reliable SCADA HMI hosting solution. The goal of our approach is to make SCADA system HMIs more readily available, interactive, and easier to maintain. Our SMS SCADA solution is based on the following key components:

With SMS SCADA, users can easily remotely perform the following functions for any of their pump or water distribution stations:

Changing the mode, status, runtimes, and rates of all pumps

Reviewing current and past alarms

Adjusting setpoints

Making notes in the operator notebook

Analyzing trends and data

At Lord & Company, we understand that your time and focus should be on running your plant, not upgrading and maintaining your SCADA system. With SMS SCADA, you can let us take care of the never-ending hardware and software upgrades required to maintain and secure your SCADA system. With our approach based on familiar HMI software, cellular communication, and Cloud-based data storage, you can rest assured that you have a long-lasting, worry-free SCADA solution in place.

New control systems have greater communication requirements than in the past. Tighter regulatory restrictions demand that control data must be passed between areas of the plant. Reduced staffing requires that plant information needs to be accessible anywhere in the plant, or sometimes outside of the plant. Advanced graphical interface tools and data logging place a greater demand on the control system’s communications bandwidth. All this combines to make the communications backbone of a Control System a critical component.Due to the increased demand for speed and reliability, specially designed industrialized hardware and redundant fiber networks were used. The redundant hardware made sure that any one break in a communication path would not restrict the communications. However, this break needed to be corrected before a second issue occurred. The decision was made to use the computer-based SCADA control system to monitor its own health and alarm if a problem was detected.
Hirschmann’s Rail-Switch family were selected from a variety of vendors providing industrialized, managed Ethernet switches because they supported the Ethernet/IP communication protocols used by the control system and could be integrated directly into the system without the need for additional hardware.
The Trihedral VTScada SCADA application was configured to monitor the health of the network switches. A loss of redundancy would be alarmed on by the computer system in the same place that process alarms (high temperature, low pH, etc.) are handled, providing a visual indication, audible indication, historical log, and remote notification by way of phone, email, or text.

The Solution:
Integrating the switch into the system was a three step process: configuration of the switch, implementing the Hirschmann-provided tools in the Rockwell Automation PLC control system, and configuring the computer-based SCADA software to monitor, display, and log alarm conditions.

Step 1: Configure the Switch
Using Hirschmann’s HiDiscovery software tool, the managed switches were configured for redundant fiber-ring topology. In addition to the basic configuration, each switch was configured to enable the Ethernet/IP protocol and the Signal Contact configured to activate on a loss of Redundancy in the ring.
It is important to note that the HiDiscovery software is provided by Hirschmann with all managed switches. However, the other switch configuration tools like the Web-based interface and the Command Line Interface can also be used.

Step 2: Program the PLCThe EDS (Electronic Device Signature) Files and Rockwell Integration tool were downloaded from the Hirschmann website. The EDS files were added to the Rockwell Software Logix5000 programming software using the EDS Hardware Installation Wizard, allowing the Switch to talk to the PLC processor as if it were an IO module. To establish the communications link between the PLC processor and the switch, the switch was simply added to the Ethernet Network tab as an “IO Module”, specifying only the IP Address of the switch and rate of data exchange.
In order to process the information from the switch, an AOI (Add-on Instruction) developed by Hirschmann was imported into the PLC program. This AOI drives the data transfer between the switch and the control processor, extracting the configuration and status information on the health of the switch directly to the PLC program memory variables.
An instruction was added to the PLC program to check the health of the communications between the PLC and the switch using a GSV instruction to monitor the “Entry Status” and generate a communications alarm.

PLC program logic was created to reference the Hirschmann_Switch AOI in order to manipulate information to/from the switch and populate PLC program variables. Information includes port configurations and status, switch temperature, model & serial numbers, diagnostic statistics, and configuration information. Almost all information available from the switch’s Web or Command Line Interface (CLI) are available.

Step 3: Configure the SCADA software

Finally, tags were created in the VTScada SCADA application to alarm on a loss of communications with the switch, or the loss of redundancy of the fiber ring. When a loss of redundancy occurs, the system will indicate the problem audibly and visually in the alarm summary. Visually, the operator can look at a display showing the entire plant communications and quickly identify the problem and begin the process of fixing the fiber ring.

Conclusion:

Utilizing a Hirschmann industrial Ethernet solution with Rockwell Automation control hardware and Trihedral’s SCADA monitoring software, Lord and Company was able to provide a cost-effective solution to increase the reliability of the customer’s Water Treatment plant control system. By tightly integrating control network diagnostics with the control system located at the plant the system health can be monitored in real-time using tools with which the plant personnel are familiar. This ensures that potential problems are brought to the attention of people who can act on the information in a timely manner, preventing interruptions in the treatment process or loss of data needed for regulatory reporting.

Lord & Company is a proven systems integrator providing control, monitoring, and information systems for improved, expanded, or new municipal water and wastewater treatments plants, industrial processes, or radio communication systems.

One critical component of these systems is SCADA technology. To understand the importance of a reliable SCADA solution, we will consider three things: current challenges around SCADA systems, key factors in successful systems, and how to choose a reliable solution partner.

Understand Challenges Surrounding SCADA Systems

In the vast networks of municipal water and wastewater plants, industrial operations, or radio systems, we know that SCADA systems are critical to uphold communication. Effective communication systems help organizations reach to remote locations and relieve those remote locations of the burden of requiring additional manpower. However, the time and costs associated with operating typical SCADA systems cause massive headaches for owners looking for seamless control, monitoring, and information systems.

With these challenges in mind, we created SMS (SCADA Made Simple) to offer an easy-to-use HMI solution that brings the benefits of SCADA technology.

Consider Critical Factors for Successful SCADA Solutions

Because SCADA systems manage many processes and withstand much interaction from operators, organizations need a SCADA solution that is secure, robust, and reliable.

SMS SCADA features a robust user interface that allows for information and data communication between monitored systems and the user. It allows operators to send and receive data, and control alarms and status through a user-friendly user interface. It saves time and money by using a cloud-based platform to mitigate the constant hardware and software upgrades required to maintain typical SCADA systems. With a highly secure data center served by a Tier 1 premium bandwidth supplier, Lord & Company has created an integrated solution with dual redundant servers for a safe, robust, reliable HMI solution

At Lord and Company, we are experienced in RF protocols, wireless technology, and the systems relying on your wireless network. Our teams use combined knowledge to help leading companies effectively use wireless communications for dependable and protected SCADA systems with solutions like SMS SCADA. Our CSIA certification, vast network of industry partnerships, and proven experience with large municipal water and wastewater systems like that of Brunswick County, NC, and fire, police, and EMS radio control systems have lead customers to trust working with us at Lord & Company time and again.

Lord And Company, Inc recently completed 3 Weeks of Motorola Toolbox, STS and Wonderware Intouch training for our highly valued customer in the North East. Training was conducted in co-operation with Motorola Solutions, Inc. 12 technical personnel each week for a total of 36 personnel were trained at a convenient facility in Brooklyn, NY. This training was specific to the customer’s System’s hardware, applications and software. Training format included a daily mixture of instructor Presentation and Hands On Lab (8 Labs) with Instructor support.

Training included:

Motorola MOSCAD Basics: The course provided knowledge for Toolbox software application backup methods, basic use of and troubleshooting with the Toolbox software, replacement procedures for hardware and reloading of the application programs and configuration files.

Motorola MOSCAD Advanced Training: Advanced level provided a detailed understanding of the System and Network Configuration, the data tables and ladder programming with procedures for online monitoring of programs and data tables.

Introduction to Motorola System Tool Suite ”STS” Software for ACE3600 and the differences between STS and Toolbox.

Motorola STS/ACE3600 Basics: This course provided knowledge for STS software application backup methods, basic use of and troubleshooting with the STS software, replacement procedures for hardware and reloading of the application programs and configuration files.

Motorola ACE3600 Advanced Training: Advanced level provided a detailed understanding of the System and Network Configuration, the data tables and ladder programming with procedures for online monitoring of programs and data tables.

Setup and Troubleshooting with a Protocol Analyzer: The course provided the purpose for and instructions of how to setup a protocol analyzer and how to export and interpret the data.

SCADA System Hardware, Software and Server Basics: The course provided an understanding of the functions of the hardware and software utilized in the system and how it communicates with the Motorola ACE3600 and MOSCAD processors. Includes procedures for rebooting the systems, making system backups and hard drive and data management.

Wonderware Intouch Basics: This course provided an understanding of the Tag Dictionary db, event logger, Modbus driver configuration for communicating to the FIU/FEP’s, links to the graphics screens and graphics development.

Six (6) MOSCAD RTU’s with 420 Processor, PS and I/O modules and configured and tested for comm to rental PC’s. One RTU per two students.

Six (6) ACE3600 RTU’s with Processor, PS and I/O modules and configured and tested for comm to rental PC’s. One RTU per two students.

A Big Thank You to the following Team members:

Kenny Brooks for configuring the Laptops & PLC’s and setting up the lab in our Fort Mill, SC Training Facility. This setup was invaluable to get the training flow synchronized for optimal learning.

Brooke Morris, for coordinating the shipment of equipment under complex delivery requirements.

Our Trainers, James Owens (Week 1), Michael Missaggia (Week 2) and Rafael Balderrama (Week 3) for extraordinary team work, combining all the different materials and co-coordinating amongst each other, so everyone could provide the same level of exceptional training.

A Special mention and Thank You to Dana Albina from Motorola Solutions, Inc for providing Training Site and managing logistics in Brooklyn, NY.

I just turned the Big Five-0 this year in July and my wife and kids threw me the biggest and best surprise birthday party that you can imagine. I had many friends and family surprise me with some unique gifts to remind me of how old I have become. We all had some great food and it was great to catch up with some friends that I do not see as often as I would like.

I did receive one gift that when I opened it; I thought it was just another joke or gag gift to see my reaction. I received a gift certificate to go skydiving from my kids. I joked it off and moved to the next card reminding me that I was older than dirt these days. A couple of days later in talking with my kids I found it was not a gag gift but I was going skydiving. I ask about the gift and they explained that they found my Bucket List in my desk drawer and skydiving was on the list. I explained to my kids that I am getting older and I do not even remember making out that list but I was told that I was going skydiving.

As a Project Manager with Lord & Company; I have been trained to avoid risk and if I cannot avoid it then ways to mitigate risks. I have been trained to define, identify, prioritize, analyze, plan, monitor and control risks. This type of analyzing sometimes drives my wife crazy but with my background in Engineering and Project Management it just comes natural.

Now I am faced with a pivotal choice, do I back out on the challenge of skydiving or do I face my fear and do this. Of course, it took me several weeks to analyze the risk and process jumping out of a plane at 14,000 feet. I finally made the decision that I would face this fear head on and schedule the big jump. I went online at Carolina Skydiving in Chester, SC and made my reservation to jump on September 24, 2016 at 9:00 AM. Since it was my first jump, it would be a tandem jump with an experienced skydiver.

I then started asking everyone at work to join me on my jump and figured a few would be glad to join me. There ended up being no takers and my wife and kids also decided that they did not want to join me. I ask around 50 people to join me skydiving and I got many different excuses. Some of the excuses ranged from “that is not a good time for me” to “I will do it the next time excuse”.

I had a great night’s sleep before my big jump. I woke up that Saturday morning and took a shower. When I got out of the shower I started thinking; I will be jumping out of a plane in a few hours. I immediately became nervous but it was time to face the fear. I drove down to the jump site by myself, so plenty of more time to think and pray. I arrived around 8:45 AM and checked in and was asked to wait until the ground training started in a few minutes.

While I waited, I was able to watch others skydiving and it was starting to sink in. You know that big lump in your throat and that feeling in your stomach. That feeling in my stomach may have just been hunger since I only ate a protein bar that morning. It was a beautiful bright sun-shinny day but a little warmer than I anticipated when I scheduled the jump back at the 1st of August.

The ground training was simple and straight forward. It is easy to do everything that you need to do on the ground, so after signing numerous waivers (too many to count). I am all set and ready to be called and fitted for my big jump. I can honestly say the worst part for me was the waiting. Once I made the decision to jump, I just wanted to do it quickly before I backed out.

I was finally called back at around 11:00 AM by my first name (William). Anyone that knows me never calls me William, it is either Tommy or Tom but never William. Scott my instructor could see how nervous I was and immediately started joking with me and told me this was his first jump also. He ask me about which parachute I wanted to select; number 22 or number 16. I explained that I would not make that choice, he went with number 22 after messing with me for a few minutes.

Suited up but back to waiting and getting even more nervous waiting for the plane to arrive. I see my wife and daughter have arrived to see me sweat, laugh and take a few photos. In just a few minutes the plane arrived and we all boarded the small plane. There were 12 to 14 people inside of the plane packed tight with 2 pilots at the controls. I start thinking about how many end up just landing with the plane but I immediately faced the fear again and decided that I am jumping out of this plane. Glancing down at the ground at around 6,000 feet and then again around 12,000 feet. The ride up took around 15 – 20 minutes. Helmet and goggles in place ready to move toward the open door on the plane. I am asked if I want to jump first or last; I immediately answer I want to go first since I am tired of waiting.

I tried to focus on my limited training and one, two, jump. That first 5 seconds was terror and then I found myself trying to get in the free-fall position. As I get into position at 120 MPH falling to the ground, all of a sudden this is not bad and I start enjoying it but in the back of my mind I am thinking I hope that chute comes out properly. After about 60 seconds of free-fall then a sudden stop as the parachute is deployed. Then it is quiet and peaceful coming down slowly toward the ground. You can now look around and enjoy the view. I enjoyed the decent down and I was given the opportunity to guide us for a little while. After a few minutes of the peace and quiet the ground was getting closer. I was given instructions again on what to do as we get closer to the landing. Scott proved why he was the professional by the nice smooth landing.

I am not going to lie, when my feet were back on the ground it felt pretty good. I was on an adrenaline high from the jump and I felt weird to just walk around. I was glad to see my beautiful wife and daughter as they ask me how was it. I was able to overcome the fear of skydiving to experience the birthday gift of a lifetime. Sometimes we have to push through our fears and challenges in life. I have been blessed with great friends and family and I want to enjoy each and every day that God gives me in this life.

I was asked if I would try skydiving again and I immediately said yes (maybe in another 50 years). Risk is a part of our life but sometimes you just have to go for it. We must first overcome fear in our mind to picture success. “When I am afraid, I put my trust in you.” Psalm 56:3

SCADA systems are essential to successful operations, and there are three critical aspects that organizations with small- and mid-size operations should consider when implementing or considering upgrades to SCADA systems:

However, there is another solution. The DRACS-RTU-3100 is an intelligent SCADA RTU solution for industries like municipal water and wastewater. The DRACS-RTU-3100 uses cloud-based communication for constant communication rather than requiring local hardware and software updates. With seamless communication capabilities through remote smartphone/tablet access, expandable I/O capabilities, and a UL-508A control panel, the DRACS-RTU-3100 is an efficient, scalable, proven solution for an RTU.

2. Use Functional Hardware
Experience with hundreds of water and wastewater applications has given us the insight to understand the most pressing needs of municipalities. Our teams use proven, off-the-shelf hardware to create reliable and efficient process automation and SCADA systems specifically for water and wastewater plants and communication networks. Using COTS hardware keeps costs down and maintenance simple.

The DRACS-RTU-3100 features a rugged NEMA 4x construction and is made with reliable, industrial-grade components for durability in all water and wastewater applications. In addition, it operates with low power consumption and includes a 12+ hour battery backup.

The platform also features a standard HMI display when used in conjunction with SMS SCADA. The two platforms combined create a total solution for data communication between a monitored system and the operator. The HMI available with SMS SCADA allows the user to receive data, control, alarms, and status updates through a user-friendly interface.

3. Choose an Experienced Partner
Our applications are already prevalent in the water and wastewater industry, and because of its rugged construction, ease of use, and reliability, the DRACS-RTU-3100 is also ideal for other industries, including:
• Industrial Processes
• Material Handling Systems
• Mining
• Network Fault Management
• Power Regeneration

At Lord & Company, our clients are the success story – through successful SCADA and automation systems, we provide high-quality, life-saving data that allows clients to impact their communities. We know that changes like a shift to the DRACS-RTU-3100 take time, and our teams work to complete implementations and testing within just weeks. We also support our systems with engineering and field teams after installation.

Lord and company vision is to be the world number one automation systems provider by completing each project on time, under budget and with excellence. In order to accomplish L&C vision, innovation and creativity plays a main key to meet projects expectations and exceed them. Troubleshooting hardware and software often require great amount of effort and time. Some of the tools for troubleshooting either software or hardware are generic for a specific system or a specific brand in the automation industry. These tools in some cases are not enough for a thoroughly troubleshoot process. For this reason Lord and Company has developed internal software tools that helps make this process more effective and less demanding on L&C employees to provide with a better project outcome.

On a current project engineers ran into a situation where a specific decoding software was needed to be able to process large amounts of data and find possible mistakes in the code. It is common on MosCAD networks to transmit data in frames where the programmer has the ability to program such messages. The messages transmitted can be via wireline links, radio link, and local RTU-RTU links. Radio link and local links can be captured via Motorola’s STS tool called Protocol Analyzer and in a more sophisticated wireline link a network sniffer software (Wireshark) can be used to capture all network traffic in a period of time.

Lord and Company has developed software to troubleshoot Motorola MosCAD system network communications. Packets being sent and received can be capture by using a port mirroring device connected to the system network and using any packet sniffing software such as Wireshark. Once the packages have been captured, Lord and Company software can convert and analyze them, providing the programmer with a readable format of the messages in the network. On the first release of the tool the programmer can copy and paste a captured message in hexadecimal format and transformed into a decimal format that is presented in a table format. The table format allows a programmer to understand, verify and validate the contents of the frame.

This software tool was developed using C# and Visual Studio 2013 IDE, it can be used on all Windows operating systems 32 and 64 bit. Programmers at Lord and Company keep improving, innovating and developing internal software tools that will help achieve the company’s vision. Future development of the software tool will have additional features to cross reference a messages with other data pre-defined data, save analyzed data into files, print and some others as needed.

This technical article for Motorola RTU networks using Ethernet links covers the messaging architecture of MDLC over Ethernet. How the MDLC is implemented within the IP structure, how the retry settings affect the messaging and what IP protocol is being used.
In order to document and validate some concepts of MDLC over Ethernet test in a lab environment have been performed. Two ACE 3640 processors with ID’s 1000 and 2000 communicate over Ethernet port (ETH1). An application was loaded into both units to send a message from site 1000 to 2000 using in some cases a TxFrm (transmit frame) and later a SndFrm (send frame) message type. (STS software version 15.50). In order to capture the network traffic between the units a port mirroring device in conjunction with a computer with running Wireshark software network diagnostics software has been used.
Motorola uses UDP (User Datagram Protocol) as opposed to TCP (Transmission Control Protocol). The difference is TCP guarantees the recipient will receive the packets in order by numbering them. The recipient sends messages back to the sender saying it received the messages. When using UDP, packets are just sent to the recipient. The sender won’t wait to make sure the recipient received the packet — it will just continue sending the next packets.
Test 1. The port configuration parameters of both units have default values. When a TxFrm type message is used between sites the following capture is observed on Wireshark:

Fig 1. Wireshark capture between site 1000 and 2000 using TxFrm. Successful communications between sites.

The 3 messages sent back and forth between sites are known as a three way hand shake. It is a three-step method that requires both the client and server to exchange SYN and ACK (acknowledgment) packets before actual data communication begins. The first message (denoted above as No. 1) is the “SYN”, the second message (denoted above as No. 2) is the “ACK” and the third message (denoted above as No. 3) is the actual Data Packet sent from the STS Application. Notice that when both sites are communicating the time delays between each event is the milliseconds order (as expected for a 10/100 communication link)
Test 2. The port configuration parameters of both units have default values. In this test the communication link between sites is failed or disconnected. When site 1000 use a TxFrm to site 2000 the following capture is observed:

Fig 2. Wireshark capture between site 1000 and 2000 using TxFrm. Bad communication between sites

Site 1000 did three attempts of the “SYN” message to the destination site (Site 2000) with a 10 second delay between each try. These results are correlated back to the Advanced Port Settings of the unit:

The Poll interval parameter is the time delay between retries (10 seconds in this example) and the Maximum number of poll parameter is the number of retries to send a message (3 in this example).

The importance of this parameters is that it allows a programmer to adjust the time delay between retries and the number of “MDLC” retries, without any additional programming at the application level. Configuring this parameter accordingly reduces the possibility of overruns when a message is transmitted. Message overrun is a term we use to describe when the application attempts a retry prior to the MDLC retry attempts expiring, this could cause the transmission buffer of one site to become full and the potentially start dropping messages and affect the performance of the CPU.

Finding the optimum balance between time and number retries and listening time (RTU is not sending messages) can be key to fixing communication issues and streamlining communications. The optimization procedure can be extensive and have multiple iterations before finding the correct balance; considerations to the network structure, speed, data reliability and the particular application will have to be considered to make proper adjustments.

Another important feature of the test was to demonstrate the use of network capture tools such as Wireshark to do diagnostics and help on the parameter settings configuration using reliable data.

On the next article a similar test will be performed using a transmission type SndFrm (Send Frame) between sites. We will show the parameters effect on the sites that communicate using this type of frame.