Process Quality in the Office for National Statistics




Дата канвертавання24.04.2016
Памер66.59 Kb.

Process Quality in the Office for National Statistics


Sarah Green1, Rachel Skentelbery and Rachael Viles,

Quality Centre, Methodology Directorate, Office for National Statistics


Paper presented at the Thirteenth Government Statistical Service Methodology Conference: London, 23rd June 2008.
1. Overview
Quality is one of the key principles of the United Kingdom (UK) National Statistics Code of Practice. At the European Union (EU) level, the requirement to improve the measurability of statistical outputs and processes is embedded in the European Statistical System and also, progressively in legislation.
The aim of the Process Quality Project in the UK Office for National Statistics (ONS) is to develop a Continuous Quality Improvement (CQI) framework for the statistical production processes, and tools to enable this with the view to roll out across the ONS and wider Government Statistical Service (GSS).
An ONS specific version of the Eurostat 'Handbook for Improving Quality by analysis of Process Variables' has been developed. The handbook will assist the implementation of the CQI techniques into the ONS, and includes updated, ONS specific examples. Additionally it provides guidance on applying the technique for each step of the Statistical Value Chain (SVC) as well as links to a defined set of process quality measures and E-Learning courses.
A comprehensive list of quality measures for statistical processes has also been developed to support the handbook. By providing a list of process measures and indicators for the statistical production process, it will enable producers to measure and monitor the quality of their processes over time and to facilitate CQI. Process measures also provide effective feedback to allow managers to be responsive to problems and objectively highlight areas for improvement. The document promotes a standardised approach to measuring process quality.

2. Introduction

The word ‘quality’ has many different meanings, depending on the context in which it is used. All users have different expectations and as such the definition of quality is a relative one, allowing for various perspectives on what constitutes quality, depending on the intended uses of the products. Quality can therefore be most usefully defined in terms of how well products meet user needs, or whether they are ‘fit for purpose’. Indeed assessing the quality of statistical outputs is concerned with providing sufficient information to judge whether or not the data (products) are of sufficient quality for their intended use(s).


Within an organisation there are a number of different perspectives of quality (figure 1). Ultimately users are concerned with the quality of products but to achieve this there are other areas where quality must be considered. Firstly, business quality: how the organisation is run and administered; following this, the quality of processes must be considered. These three aspects all contribute to the reputation of the organisation which has a direct impact on users’ perceived level of quality of the organisation and its products. It can be seen, therefore, that process quality is an integral component of any organisation and as such it is vital it is understood and controlled within the ONS. The word ‘quality’ has many different meanings, depending on the context in which it is used. All users have different expectations and as such the definition of quality is a relative one, allowing for various perspectives on what constitutes quality,
Figure 1: The relationship between the different types of quality that affect the statistical outputs produced by ONS (adapted from Statistics Sweden model)

The importance of process quality to a statistical organisation was highlighted by the Eurostat Leadership Expert Group (LEG) final report (Eurostat (2002)) which states that:


'in theory, good product quality can be achieved through evaluations and rework. However, this is not a feasible approach since it is costly and time consuming. Instead, it is believed that product quality will follow from improvements in process quality'
Therefore for any organisation with constraints on business, process quality is the sensible way forward. The LEG recognised that improving process quality has a direct effect on product quality, and that this is often more feasible than improving product quality directly.
The LEG called for the development of a handbook on the identification, measurement and analysis of key process variables, which are those factors that can vary with each repetition of the process and have the largest effect on critical product characteristics. The “Handbook on improving quality by analysis of process variables” (Eurostat (2004)) was produced by a project team consisting of members from the National Statistics Institutes of Greece, Portugal, Sweden and the UK, who led the project. The Eurostat handbook consists of guidance on how to identify, measure and analyse process variables. Once a general approach was developed, examples of applying the method to statistical processes were identified. These examples are presented in the handbook, categorised into the key processes used to produce statistics.
This paper summarises two of the initiatives taken by ONS under the process quality project to develop a continuous quality improvement framework for statistical production processes, and the tools to enable this, with the view to roll out across ONS and wider GSS.

3. ONS Process Quality Handbook
The “Handbook on improving quality by analysis of process variables” is a long and technical document. While it is ideal for reference it is not easily accessible in order to roll out process quality techniques across ONS. Due to this, one of the initiatives from the process quality project was to produce a summarised version of the Eurostat handbook for ONS. This was to include updated guidance on improving quality by analysis of process variables and examples relevant to ONS.
The ONS Process Quality Handbook describes methods for improving quality by analysis of processes based on the Eurostat handbook. This handbook applies the general approach taken in the Eurostat handbook to the ONS and describes useful tools for the task of identifying, measuring and analysing key processes in order to achieve Continuous Quality Improvement (CQI). Section 2 includes guidance and tools while Section 3 looks specifically at examples of the application to ONS statistical processes using the ONS Statistical Value Chain (SVC). The ONS SVC was developed during 2001 and describes the key ONS processes to produce statistics. There exist several other similar representations of the statistical production cycle, for example Eurostat uses a tool called the Cycle des Vie de Données (CVD) or Data Life Cycle.
The ONS handbook does not aim to provide a list of recommended variables to measure process quality across all statistical processes; such measures are being developed in a separate process quality measures document. It is however a component of the process quality tool kit being developed in the ONS.
Process quality can be judged using Key Process Variables. A process variable is a factor that can vary with each repetition of the process; those that are key are those which will have a large effect on key characteristics of the output. Morganstein and Marker (1997) developed a standardised approach to process quality, that will produce an output (product) meeting user needs. The ONS handbook uses an adaptation of this approach, Figure 2, tailored to be specific to ONS needs. The standardised approach is planned in a process map made up of six steps. The first three steps cover identifying, the fourth measuring, the fifth and sixth analysing key process variables.
Figure 2: Implementing and improving process quality for ONS processes, a step-by-step guide (adapted from Morganstein & Marker, 1997)

Clearly, a vital part of this approach is the ability to identify the key characteristics of the process that will produce an output to fit user requirements. It is not enough to identify a process; in order to implement process quality and effectively achieve CQI it is essential to understand the internal and external output characteristics that are important to users; these are the key characteristics.


An important consideration when identifying key characteristics is to ensure adequate coverage of the European Statistical System (ESS) dimensions of quality through the processes to be monitored. The six dimensions of quality for statistical outputs are:


  • Relevance: a statistical product is relevant if it meets user needs.

  • Accuracy: accuracy is the difference between the estimate and the true parameter value.

  • Timeliness: this is important for users since it is linked to an efficient use of the results.

  • Accessibility and Clarity: results have high value when they are easily available and understood.

  • Comparability: reliable comparisons across space and time are often crucial.

  • Coherence: statistics originating from different sources are coherent insofar as they are based on common definitions, classifications and methodological standards.

Identifying the key characteristics will also identify the processes to focus on. Once a process has been identified for monitoring and the key characteristics identified, the next step is to map the process by developing a comprehensive flow chart of the process, known as a process map. Process maps can be used to take an objective look at a process and identify:




  • who owns the process or parts of it

  • sources of variation in the process

  • ways to monitor the process

  • any activity that uses resource but adds no value

An effective process map will establish the following:




  • pictorial representation of a process

  • clearly defined start and end points

  • the flow of the process and decision points within it

  • what actually happens in a process - not what 'ideally' should occur.

Once the process is fully understood it can be looked at more closely to identify, monitor and analyse key process variables.


3.1 Identifying, Monitoring & Analysing of Key Process Variables
Process variables observed conveniently and continuously during the process can be used to assess the quality of the process and as such control the quality of the output.
It would not be efficient or effective to measure all process variables and so the aim must be to identify those factors that can vary with each repetition of the process and have the largest effect on key characteristics of the output, that is the key process variables.
A simple and effective tool for identifying key process variables is the Pareto chart. This tool tries to find the relatively few error types that account for the majority of all errors, hence enabling staff to be more effective in allocating resources. The error types identified, that account for the majority of all errors can be labelled as the key process variables to measure over time.
Figure 3 is an example of using a Pareto chart to identify key process variables in ONS e-commerce survey. With an aim of reducing item non-response for e-commerce it can be seen that the process variable is the contribution to item non-response of each question. The Pareto chart shows that out of thirty questions (on the e-commerce questionnaire) four are identified as contributing to 33% of the total item non-response for the survey. As such these four questions will be the key process variables to consider.
Figure 3: Example of Pareto Chart to Identify Key Process Variables in E-Commerce Survey

Some process variables can not be summarised quantitatively in a Pareto chart. Cause & Effect diagrams can identify key process variables when numeric information on the variables is not available.
After identifying the key process variables, it is important to know how accurately they can be measured. Ideally this measurement should require as little effort as possible on the part of the producers. The key points to consider when evaluating how to measure the key process variables are:


  • what can be measured that is useful to understanding and monitoring the system

  • can the measures be used to assess changes in the quality

  • what does success look like

  • can the measures lead to potential for improvement

In the same way that efficiency and effectiveness are ensured by identifying a subset of process variables that are key to measure, it may not be efficient or effective to measure a key process variable continuously in order to assess the quality of the process. If this is the case then a sampling and verification approach could be used: assessing a sample of work from the process.


Figure 4 is an example of a measuring key process variables for ONS UK Census, monitoring the accuracy of occupation coding for each estimation area.
Figure 4: An example of measuring key process variables in ONS UK Census
UK Census uses occupation coding and the accuracy of this was identified as a key process variable when looking to monitor and improve processes.

Rather than checking all codes a 2% sample of codes was verified for each estimation area, independently of coders.

Sample data were used to estimate the key process variable which could be measured as a consistency rate.

The consistency rate can be considered over time to identify any positive or negative changes in quality.

Success will be consistently reaching an agreed level of consistency in the accuracy of coding.

If the consistency rate is found to be too low or drops then the process can be considered to see where problems may occur that lead to this and how these problems can be rectified. In this case data checks may be required at more strategic points in the process.


Once key process variables have been identified and measured, analysis of them can begin. This begins by testing them for stability: process stability is a state where the process variation consists entirely of random components, that is when the variation is not systematic.
In the ONS handbook, the main tools highlighted to monitor stability are control charts and analysis of Pareto charts. Through this analysis step, we can identify the sources of process and output variations, and examine the effects that changes to the process have on variation. Control charts have control limits that are typically calculated at three standard deviations from a centre line (possibly a group average). Observations that fall out of the control limits are deemed incapable or 'out-of-control'. A special cause for out-of-control observations should be identified and addressed. Tools such as control charts can be used to measure the effectiveness of improvements and accurately predict likely outcomes in the future.
Figure 5 is an example of a control chart for the ONS UK Census, monitoring the accuracy of occupation coding for each estimation area as discussed in Figure 4.
Figure 5: Example of control chart monitoring consistency rates in ONS UK Census

Points of interest in this example are clearly from February to April 2002 where the consistency rate is out of control and over the top limit. This examples shows that the limits are in place in order to help spot good and bad special causes of variation in processes. The higher-than-anticipated rate should be investigated in as much detail as if the consistency rate had dropped below the lower limit: when things get better spontaneously it is important to investigate the reasons for this as it may be something that can be incorporated permanently into the process.
Once the key process variables have been identified, measured and monitored then it must be determined whether the process produces an output that, over time, continuously meets user requirements. For processes that are reasonably stable, staff can determine the limits of the expected process variation, or evaluate the capability of the process to predictably meet specifications defined and driven by user needs. A stable process is ‘capable’ if its random variation is such that the system will consistently meet the user requirements or limits for that process.
Once a process has been shown to be capable of meeting user requirements, it is important to ensure that a system of continuous quality improvement is implemented and maintained. A continuous monitoring system is needed to:


  • keep staff informed

  • provide feedback

  • assist in controlling process variation

  • help achieve continuous reduction in process variation through improved methods

  • evaluate process changes

It should also aim to reduce the burden of measuring and monitoring by automating or improving procedures. Effective procedures for managing the implementation of identified improvements are important.



4. Process Quality Measures
To complement the ONS Process Quality Handbook, a definitive set of quality measures are being developed. This set of process quality measures will enable producers of ONS statistics to measure and monitor processes over time, be responsive to problems and highlight areas for improvement. Moreover, it should create a standardised approach to measuring ONS process quality and ensuring comparability between outputs, since standard quality measures could be checked across ONS. It will also provide a means for users to quality assure their processes, again in a standardised manner across the organisation.
The ONS process quality measures are being designed to be counterpart to the statistical output ONS Guidelines for Measuring Statistical Quality. Measures will link to steps in ONS SVC and one of the six process quality attributes documented by the Eurostat LEG. The process quality attributes are an accompaniment to the ESS six dimensions of quality for outputs. As with the dimensions the attributes are definitions to usefully describe and determine process quality, which would otherwise be intangible and certainly inconsistent across processes.
The definitions, from the LEG, for each of the process attributes are as follows:


  • Effectiveness: successfully delivering the desired outcomes, meeting business needs.

  • Efficiency: producing the desired outcomes cost effectively.

  • Robustness: delivering results against challenging demands.

  • Flexibility: processes that readily adapt to changing needs and demands.

  • Transparency: processes that are open, visible and easily understood.

  • Integration: processes that are complementary and consistent with other processes.

In terms of considering standardised approaches for measuring process quality the most important aspect of the process quality measures work will be to derive a subset of key process quality measures (KPQMs). For the Guidelines for Measuring Statistical Quality, out of over 160 quality measures and indicators a subset of 12 key quality measures (KQMs) has been identified (Annex A). The KQMs are considered to be the most important and informative in giving users an overall summary of output quality and it will be important to have the same approach for process quality.



5. Implementation
Process quality is not a new concept in any organisation. However the implementation of process quality measurement, monitoring and improvement, as an inherent part of business-as-usual in organisations is challenging to achieve. Even in organisations or areas where variables within processes are measured it can often be seen that this information is not used to its full advantage to achieve continuous quality improvement.
As an organisation ONS wishes to continually achieve acceptable quality processes and make continuous quality improvement where necessary. This being the case, it is important that the ONS Process Quality Handbook and the associated process quality measures are rolled out effectively to ONS. As such a progressive approach to roll out across ONS is to be taken.
Roll out of process quality will largely be achieved through piloting the ONS Handbook in specific statistical output areas. If gains in process quality can be made in these pilot areas and they can quantify the benefits they have found from implementing the approach then these success stories can be used to create interest from other areas of the office.
Bespoke support to statistical output areas will be provided from ONS Quality Centre, since from examination it can be seen that processes fall into two categories:

  • those amenable to the whole process quality approach, including identifying measurable process variables

  • those where mapping the process and identifying key factors may lead to valuable quality improvement actions

Such support will broaden the process quality knowledge of Quality Centre as well as enable all areas of the office to have sufficient and effective process quality procedures in place.


Detailed information about when and how to use the corporate process quality tools described in the ONS Handbook will be addressed through e-learning courses and supported by expert led training sessions where required.


References

Eurostat (2002) Quality in the European Statistical System - The Way Forward, Office for Official Publications of the European Communities: Luxembourg.


Eurostat (2004) Handbook on improving quality by analysis of process variable.
Jones N and Lewis D (2004) Developing a handbook on improving quality by analysis of process variables, Paper for the European Conference on Quality and Methodology in Official Statistics: Mainz, Germany, 24-26 May 2004.
Morganstein D and Marker D A (1997) Continuous Quality Improvement in Statistical Agencies, in Lyberg L, Biemer P, Collins M, De Leeuw E, Dippo C, Schwarz N, and Trewin D (eds.), Survey Measurement and Process Quality, New York: Wiley, pp. 475-500.
ONS (2007) Guidelines for Measuring Statistical Quality v3.1
ONS (2008) ONS Process Quality Handbook, forthcoming publication.

Annex A UK Office for National Statistics: Key Quality Measure (KQMs)
Key Quality Measures have a dual function: to provide a summary of overall output quality, and to provide management information to monitor office performance. In their latter function, they fulfil the requirement for statistical key performance indicators (other non-statistical key performance indicators may include Human Resources or financial information, but these are not covered here).

Key Quality Measures can be either quantitative (that is calculated with standard formulae) or qualitative (that is textual statements). For both, guidance is available: formulae are given for quantitative measures; for qualitative measures examples are given as well as 'things to consider' when completing the quality measure. Guidance is available the National Statistics website in the Guidelines for Measuring Statistical Quality at http://www.statistics.gov.uk/qualitymeasures.

It is important to note that not all of the KQMs will be relevant to all outputs. When writing a quality report only relevant KQMs should be reported upon.


Key Quality Measures

Quantitative / Qualitative

(Included in)

ESS Quality Dimension

Where possible, describe how the data relate to the needs of users

Qualitative

(Summary Quality Report)



Relevance

Provide a statement of the nationally/ internationally agreed definitions and standards used

Qualitative

(Summary Quality Report)



Comparability

Unit response rate by sub-groups, weighted and unweighted

Quantitative

(Basic Quality Information)



Accuracy

Key item response rates

Quantitative

(Basic Quality Information)



Accuracy

Editing rate (for key items)

Quantitative

(Basic Quality Information)



Accuracy

Total contribution to key estimates from imputed values

Quantitative

(Basic Quality Information)




Accuracy

Estimated standard error for key estimates

Quantitative

(Basic Quality Information)




Accuracy

Compare estimates with other sources on the same theme

Qualitative

(Summary Quality Report)



Coherence

Estimated mean absolute revision between provisional and final statistics

Quantitative

(Basic Quality Information)



Accuracy

Identify known gaps between key user needs, in terms of coverage and detail, and current data

Qualitative

(Summary Quality Report)



Relevance

Time lag from the reference date/period to the release of the provisional output

Quantitative

(Summary Quality Report)



Timeliness and punctuality

Time lag from the reference date/period to the release of the final output

Quantitative

(Summary Quality Report)



Timeliness and punctuality




Address for correspondence: Sarah Green, Office for National Statistics, Government Buildings, Cardiff Road, Newport, Gwent, United Kingdom, NP10 8XG (sarah.green@ons.gov.uk)


База данных защищена авторским правом ©shkola.of.by 2016
звярнуцца да адміністрацыі

    Галоўная старонка