Managing intelligent electronic devices via automated systems

As part of their recent funding for smart grid initiatives, utilities in the USA have been deploying large numbers of intelligent electronic devices (IEDs) to implement advanced automation systems. However, they are discovering issues related to security and the high cost of maintaining these devices.

Originally considered as essentially capital intensive, projects in the electrical sector have seen increasing operational costs as utilities are faced with devices that require complex commissioning, potential regular firmware updates and issues related to security management. In Australia and New Zealand, we can leverage the US experience and take advantage of their learnings.

Power distribution systems have traditionally been relatively static in the context of local and/or remote binary operations with binary feedback of equipment. Recent years have seen an increase in the amount of intelligent activities utilities are expected to carry out, including advanced, coordinated, real-time functions such as fault location isolation and service restoration (FLISR), real-time power network status reporting, VAR management and voltage optimisation and control.

This technological evolution can provide a wide variety of business benefits — but it also brings a number of operational challenges. Advanced automation systems are built out of large numbers of programmable intelligent electronic devices, which can come from a broad variety of vendors. All of these devices are expected to operate in conjunction with one another, while also exchanging data in a secure and reliable manner.

This is a significant change in paradigm; traditional power distribution systems were designed to be long-lasting, incorporate simple technology and use simple protocols, with little or no security. Once operational, they were not expected to change until they required repair or replacement. Automation systems were utilised to ensure functionality, but not much more. However, many modern automation systems are adopting or inheriting functions that have traditionally been part of IT systems:

• Greater system dynamism, with a shorter device life expectancy.
• More regular device upgrades, due to obsolescence or discontinuation.
• New generations introduce new technology and protocols.
• More complex devices, requiring regular firmware updates to address security vulnerabilities, programming errors and new functionality.

Change has therefore become a fundamental characteristic of modern automation systems, and it must be considered as part of a business’s wider IT and security strategy.

The ongoing evolution

Traditionally, communications, interoperability and security have generally been secondary considerations for automation systems as engineers selected ‘best of breed’ devices, according to their functional capabilities.

From a practical standpoint, commissioning a device requires numerous manual operations, often using a proprietary software tool connected to a serial maintenance port. Up until now, this was not an issue since the automation system was often deployed as part of a larger project, with very few changes expected or planned once the device was installed. It was time-consuming in the initial outlay, but required little attention afterwards and most ongoing maintenance was carried out via manual means.

But with the expansion of automated system functions, they are no longer static and require greater attention, including maintenance and device updates or replacements. As utilities extend their communication networks to integrate ever-increasing numbers of devices across new applications, it will no longer be cost-effective to manage the system manually and some form of automatic configuration is inevitable. While vendors of networking devices and proponents of the Internet of Things (IoT) have been promoting a vision of plug-and-play devices and zero-touch deployment, the whole process of commissioning electrical controls such as protection relays, voltage regulators, recloser controls and capacitor bank controls, from a variety of vendors, is much more complex and will remain so for the foreseeable future.

Moving into new territory

As this is relatively new territory for power and IT specialists alike, Australia and New Zealand are still to develop a set of compliance standards surrounding the use of these technologies. Though Standards Australia and/or Standards New Zealand are likely to formulate them in the future as the technology becomes more widespread, at present it is recognised that the US-based North American Electric Reliability Corporation (NERC) CIP provides the most comprehensive standards in physical and cybersecurity.

Along with cybersecurity, there are currently efforts to extend standards such as DNP3 and IEC 61850 to include device properties and configuration settings, but these have not yet been widely adopted by the industry. As such, the task of managing devices used in automation systems requires specialised knowledge and experience that can only be expected from vendors in the electrical sector.

Electrical substations powering smart grids are host to a large number of IEDs, from a variety of vendors. These are used for protection, metering, monitoring and communications. Utilities that operate transmission substations were the first to implement automated device management systems; this was originally to reduce operational costs, but also to help meet compliance requirements.

One of the first applications to be automated was the retrieval of power system event and fault data from digital fault recorders (DFR), in order to meet NERC PRC reliability requirements. Many utilities thus chose to connect their substations and deploy automated event retrieval software to eliminate the high cost of having skilled technical personnel drive to the substation simply to manually retrieve event data.

Deploying TCP/IP networks to the substations and installing advanced data concentrators had the additional benefit of providing the capability to manage devices remotely, further reducing operational costs. However, providing remote access to devices also opened a potential security issue which was rapidly addressed by NERC CIP.

Why automation needs to begin now

Complying with the NERC CIP standards involves much more than complying with technical requirements. Utilities have to provide evidence that their operational processes comply with the requirements. Managing lists of devices, firmware versions, users, access permissions and passwords update history are all operations that can be much better handled by a device management software, rather than manually (eg, via spreadsheets). For many utilities, the business case for automating device management has thus been the ability to reduce operational costs through secure remote access, in addition to implementing tools to automate NERC CIP compliance.

Utilities that operate transmission substations must comply with the NERC CIP standards and also put in place policies and procedures to ensure the security of the devices used to implement critical functions in the bulk electric system (BES). While most large utilities have deployed some type of asset management software at the enterprise level, these systems are typically static and the information must be entered manually.

IT network management software, also typically used by large utilities, has generally been unsuccessful in handling IEDs because of the lack of standard protocols to interrogate substation devices and programmatically extract information such as firmware versions, serial numbers and device settings.

Specialist device management software designed for the electrical sector instead implements device drivers for all common devices used in the electrical sector, and supports the functions required for managing the device’s life cycle for NERC CIP compliance.

Reaping the benefits

Utilities that have deployed device management software to provide secure remote access and automate many of the most labour-intensive aspects of NERC CIP are now reporting additional unexpected benefits. Because substation devices can be accessed securely, departmental silos are breaking up, leveraging the communications infrastructure to provide access to multiple departments that require substation data for a variety of applications such as power quality or device condition monitoring. The time and effort required during the investigation of outages and power restoration is reduced through improved access to devices and the capability to remotely change settings during emergencies and weather events. Device management systems are thus providing proven benefits for the operators of transmission substations. While not yet being used on a large scale, some businesses with Australia and New Zealand are already taking advantage of the opportunities these systems present.

The high cost of compliance to NERC CIP standards provides a clear-cut business case for implementing automated management of substation devices — but what is the business case for devices used as part of distribution automation systems?

While the transmission substation can be characterised by the large variety of different devices installed in a single location, distribution automation is characterised by the large number of identical devices deployed throughout a large geographical area. The operational challenges will thus be very different.

Experience gained while working with utilities that are deploying large fleets of distribution automation devices has provided us with a better understanding of the areas where operational costs are the most significant. Distribution automation devices are geographically dispersed, ensuring that remote management becomes a cost-effective alternative to an expensive truck roll. As an example, bulk settings and firmware updates can be performed remotely.

As utilities deploy their distribution automation systems, they gain a better understanding of the operation of their electrical network and will often need to change device operational settings from the original values. While some setpoints are mapped to DNP3 data points, not every setting is available through SCADA or DMS, and some changes need to be performed using the device configuration tool. Providing remote maintenance access to the device will obviously be less costly than driving to the site.

With thousands of devices to update, these operations rapidly become tedious, error-prone and very time-consuming. Firmware updates are the new reality in power system automation. As devices become more complex and perform more functions, they expose more programming errors and security vulnerabilities. While updating device settings is generally a short and straightforward operation, firmware updates require the transfer of larger data files and take much longer to perform with the communication speed provided by most distribution automation communication networks. Having qualified technical personnel sit in front of a maintenance tool watching a progress bar slowly creep forward is simply not a cost-effective operation.

As a ballpark figure, imagine an operation that requires at least 30 minutes performed, on a thousand devices, at an hourly cost in the range of $100 per hour to appreciate the potential cost savings of having the operation performed automatically, with no operator intervention. Firmware and setting updates are applications where the reality of distribution automation is very different than the realities of substation automation. Because of the criticality of substation devices, utilities generally prefer to be on-site to perform operations such as firmware updates. Device management software can accordingly be adjusted to meet the requirements of distribution devices that are typically less critical and where it is simply not cost-effective to perform updates manually.

Automating the process of keeping track of device configuration settings also makes sense from the perspective of operational best practices, even if it is not a compliance requirement.

Finally, most utilities will not consider managing device passwords as a very high priority for distribution devices. However, this will change in the future as it is expected that cybersecurity requirements will certainly extend to the distribution network.

Businesses looking at investing in this area of ‘smart grid’ with substation automation should be planning and anticipating now. The power distribution utilities will expand in their deployment and the functions and increase in complexity over the coming years. Investing in the proper management systems now can help prepare for future changes, and also reduce costs in the process.

Image credit: ©stock.adobe.com/au/kinwun