An operator recently experienced damage to their pipeline when a metal pipeline marker post was driven into the ground by a falling tree – the ‘leaning tree incident’. This is certainly not a common pipeline failure scenario. How much criticism against a risk assessment is warranted if this obscure event is missed?
CERTIFICATION
With growth in both the number of pipelines and their neighbouring receptors, there is more at stake from pipeline failures. Formal pipeline risk management is now an essential aspect of owning and operating pipeline facilities. Stakeholders are requiring increasing levels of assurance that the risk management program is truly effective. This article begins a discussion on certification of risk management processes, i.e., gaining assurances for stakeholders that currently used processes are at least appropriate, if not robust and optimum.
Terms like validation, verification, and calibration, while not universally defined, logically seem intertwined. Let’s adopt the term certification to cover all and say that a certified process is one that meets or exceeds minimum acceptability requirements. More on that in a later article.
FOCUS ON RISK ESTIMATION
Until a long track record demonstrates how well risk management was done, a program’s effectiveness is best evaluated in terms of its components. First and foremost, good risk management requires good risk assessment. If the risk is not well understood, how can management of risk be effective? So, assurance of good risk management logically begins with an examination of the embedded risk assessment process.
Risk assessment involves the general steps of data collection, data integration, assignment of values for missing information, and production of risk estimates. As a first step in certifying an overall risk assessment, it makes sense to begin with the last task – producing risk estimates.
Why focus on risk estimation, i.e., the risk models, first? Two compelling reasons include that 1) it is a current area of US regulator concern and 2) good risk estimation offers some assurance of effective ‘downstream’ processes.
The first general question in the certification effort is: Can the risk assessment model produce true risk values? That is our topic now. For future certification steps, the words ‘does’ and ‘will’ replace ‘can’ in the same question. Answering this requires subsequent evaluation of the other parts of the overall risk management process – again, a future topic.
Ideally, subjectivity will be largely removed from the certification process. In this first certification step, objectivity is achieved by having certification-seekers produce risk estimates from a set of information for which the risk issues are well known. That is, perform a risk assessment with a standardised, assumed-accurate dataset and compare results with the previously determined risks. Isolating the mechanics of the risk estimation by assuming perfect information inputs allows a better evaluation of the appropriateness and capability of the risk estimation itself. The role of data accuracy is very important but confuses the evaluation of other components.
COMPLETENESS
It is not practical to capture all possible risk issues in a certification or ‘test’ dataset. Therefore, a test using a provided dataset will likely not prove model performance against all possible risk issues.
Think again about our ‘leaning tree incident’. At first glance, it is tempting to say that missing a ‘one-in-a-million’ threat like that is not as serious as missing a more frequent threat. However, what is generally a miniscule threat when viewing thousands of miles of pipe over many years can be the primary threat for a specific location at a specific time. The ‘one-in-a-million’ scenario is only appropriately ignored when it truly is that low everywhere (and will not become significant when aggregated).
OPPORTUNITY FOR COMPLETENESS
For certification purposes, we make a distinction between actually recording the threat versus having the opportunity to record the threat. If we see the dead tree leaning over the marker post directly over the pipeline, but have no way to capture this in the risk assessment, the risk assessment is flawed. On the other hand, a risk assessment that is ready to capture and assess this obscure scenario meets minimum requirements, even if that threat was not input. Falling objects should already be a consideration, and this particular scenario should be additive to all similar scenarios – e.g. falling buildings, utility poles, rockslides, etc. All threats are analysed via independent evaluations of the exposure, mitigation, and resistance elements
(see previous articles).
ACCURACY
So, a certification-seeker has produced risk estimates using their risk estimation processes on the test data. What if their risk estimates differ significantly from the benchmark results? Without agreement on ‘true’ risk estimates, how can certification be accomplished? The answer is that ‘correct’ risk assessments can produce a wide range of risk estimates for exactly the same scenario, depending on factors such as:
- assumptions employed when information
is missing or weak - target level of conservatism desired.
Furthermore, since our risk estimates must contain elements of probability, we will usually not know their true accuracy for decades, so insistence on matching certain numeric values is not appropriate. For certification purposes, as for many other uses, the risk profile is the key. The profile is often the most useful output of the risk assessment. This means it is also a central element of a certification.
A profile shows changes in risk along the pipeline route and demonstrates aspects central to acceptability of risk estimation:
- locations of directional changes (up or down)
- magnitudes of changes
- drivers of changes
- aggregations of multiple issues at the same location
- comparisons between any points
- comparisons between similar pipelines
(e.g., perhaps identical routes with different products or operating characteristics).
All of these profile-demonstrated aspects should be fully consistent with the underlying science and engineering of the pipeline’s failure potential. That is what makes the risk estimation process acceptable and worthy of certification. Matching exact numerical estimates of risk are not necessary – other objective criteria that allow for numerical differences can be employed.
GAINING CERTIFICATION
To conclude this initial discussion, let’s recognise that 1) growing stakeholder concerns can be at least partially addressed by independent evaluation of pipeline risk management processes and 2) producing a fair and useful evaluation of risk management processes requires some thought and planning. As described here, the effort is underway!