By Isiah Jones, Director & Principal – ICS Cyber Security Engineering
While attending the EnergySec Electric Distribution Security Forum March 22 – 23, 2018 in Washington, DC, the topic of best practices came up between trade organizations and state utility commission speakers and attendees. I informed them that in security “best practices” are already defined and not something that wiggles with legal interpretations. A security best practice is something that can be universally applied at different maturity levels regardless of how small or large the organization or what regulatory or non-regulatory jurisdiction or sector vertical they belong to. Each sector and jurisdiction all use PLCs, RTUs, I/O devices, sensors, actuators, laptops, servers, both unmanaged and managed network switches, wired communications transports (e.g. ethernet, fiber, electric power lines, leased phone lines etc) and some form of RF/wireless communications (e.g. 900 Mhz, cellular, ISA100, WirelessHART, Wi-Fi, ZigBee, satellite, microwave etc). These assets and devices range from geographically dispersed to sitting and operating within the same building or facility.
The below top 10 list is a universal list that all our most critical infrastructures should implement internally–regardless of size or jurisdiction–if they have the budget and staff or seek help externally by prioritizing these requirements in all their third-party agreements and all requests for services and funding from their trade organizations as well as local, state and federal stakeholders. Organizations that don’t have the staff and budget for grandiose commercial based solutions should strongly consider seeking out those with critical infrastructure security experiences and leverage open source tools. Those owners and operators should consider “ICS Security Manager as a Service” as a viable option.
The top 10 list is in no particular order because applying all 10 is very crucial to the security and resilience of our critical infrastructures, especially our interdependent industrial infrastructures such as water, oil, gas, electric, transportation (e.g. pipelines, rail, aviation, maritime) and telecommunications.
Practice 1: Operational Resilience and Business Continuity
Seek out trained individuals or resources from your sector peers and trade organizations that can help you with conducting a business impact analysis. Then seek resources to help implement and enforce in your contracts that backup, recovery, exercises, testing and redundancy be built into your assets and operations. Leverage open source tools and public resources from organizations like NIST. NIST SP 800-34 and other sources have templates that can be leveraged to help even small organizations know what areas they should consider and how they should go about doing it.
Practice 2: Governance, Risk and Security Program Management
For industrial control systems (ICS), automation, and operational technology (OT) in general (includes SCADA, DCS, PCS, PLCs, HMIs etc) there is a need for organizations to designate in writing a person responsible for ICS security and critical infrastructure operational resilience. Often this role is best filled by someone with an operations and ICS background who has also been trained in ICS specific security tactic, techniques and procedures. This role should also work with the CISO and CSO collaboratively for ICS security. An ICS security council and change or configuration control board (CCB) should be created where these folks meet monthly to hash out and make execution decisions on issues related to ICS and dependent operations.
Practice 3: Asset and Inventory Management
You cannot protect what you do not know you have. You cannot protect what you know you have if you don’t collect or require by contract the collection, maintenance and documentation of ports, protocols, services, real-time OS, OS, firmware, interfaces etc. Leverage tools and ICS trained and experienced individuals to help you keep up with an inventory. Tag assets by name, type, operational business unit, site location etc.
Practice 4: Insider Threat and Personnel Security Program
Nation states and terrorist are not the only issue. Sometimes your trusted integrator will make mistakes on your assets in the field and not fix it, know it was broken or tell you about it if they did. You also have former employees you should pay attention to. Third-parties and employees can and have caused more ICS and operational resilience impact than nation states have combined in many cases. Do not overlook all threat sources.
Practice 5: Configuration Management and Patch Management
ICS and operational infrastructure is not IT. Do not patch, scan, fix in ICS. Instead by contract and by procedure and policy create at least quarterly maintenance windows, a spare parts inventory, and a test lab and jump kit culture so that patches, drivers, updates, retrofits and other changes can be simulated, tested and scoped prior to a phased zone by zone implementation in production operations. This is very important for ICS. Leverage virtualization and simulation tools as much as possible.
Practice 6: Vulnerability, Risk, Threat Assessments and Penetration Tests
At least annually but especially after mergers, acquisitions, divestments or major new greenfield projects or even major brownfield retrofits you should do at a minimum a vulnerability assessment and at best a full risk, threat, consequence, and impact assessment plus a penetration test for ICS and dependent operations. Call in ICS focused security professionals to help you do this on all Commissioning, Factory Acceptance Test (FAT), Site Acceptance Test (SAT), annual exercises and annual assessments.
Practice 7: Network Management and Security
ISA/IEC 62443 security zones and conduits is not just a buzzword it is something you must implement in ICS environments. You should always have DMZs with separate domains and zero trust between domains with shared data services locked in the DMZ as needed to avoid direct two-way communication between safety, primary ICS and any other system outside of ICS (e.g. IT, internet and third-party connections). Always enforce this in all designs and all contracts.
Practice 8: Incident Response, Analysis and Monitoring
For ICS leverage the US Cyber Command Advanced Control Systems Tactics, Techniques and Procedures (ACI TTP) for ICS, ICS-CERT recommended practices, NIST SP 800-61, ISA/IEC 62443 and SANS Incident Handlers Handbook to create an ICS specific Incident response program with policies, procedures and guides that define what an event is, when an event becomes an incident, chain of command, chain of custody, chain of communication both internal and external to ICS operations and the approved tools, roles, responsibilities, tactics and techniques. Ensure ICS focuses on ICS devices and protocols.
Your third-parties and where you shop is your weakest link. Ensure all technical specifications sections of all contracts and agreements list out the specific security controls you want to implement from NIST SP 800-82, ISA/IEC 62443, NERC CIP and other critical infrastructure and ICS focused security standards, best practices and regulations. Also ensure controls from the NIST SP 800-161 for Supply Chain Risk Management are implemented within all contracts and agreements as well as organizational practices, procedures, cultural processes and policies.
Practice 10: Systems Security Engineering
Secure system design, testing and validation, acceptance testing, commissioning and continuous lifecycle checkups is the best way to ensure your ICS assets arrive secure by design and remain secure by design until they are properly decommissioned. Ensure controls from NIST SP 800-160 for Systems Security Engineering is implemented in all processes, procedures and contract agreements.
In addition to requiring, teaching and educating asset owners on this top 10 list, Regulatory agencies such as one I worked for, Federal Energy Regulatory Commission (FERC), and all NARUC members in general, should do their duty and create cost recovery rules, regulations and orders focused around security, safety and resilience. This needs to be done especially for the small to midsize organizations that don’t have large investor or profit based revenue streams or the staff nor can they afford to contract the staff or tools without cost recovery being granted by the regulatory agencies. All proposals for rate cases, retrofits and new projects should be required to include operational security and resilience, particularly ICS focused, needs and cost estimates that address mitigations of impacts and consequences that require budget and resources to mitigate successfully.
Implementing these best practices and regulatory support for cost recovery methods will go a long way towards expediting the improvement of many of the various infrastructures society depends on daily for our way of life. If the regulators, asset owners, vendors, EPCs, integrators and the community doesn’t implement these top 10 practices as the daily standard then the insurance industry should step in and build premiums based on the maturity levels of organizations measured against these top 10 practices for ICS security and operational resilience within critical infrastructure. If you want to learn more of what you should do and how to get help doing it then just reach out to LEO’s ICS team, we are the community and the community is us.