14.9.1 One of the terms of reference for the Commission is “to examine the need for instituting statistical audit of the range of services provided by the Government and local bodies and make suitable recommendations thereof”. For efficient execution of any such project for providing services to the general public, Statistics are required at every stage – project formulation, execution and evaluation on completion. Apart from the basic information required in setting the targets of the project and assessing resource needs and availability, a benchmark survey is usually carried out to record initial conditions at the start of the project. During the execution of the project, collection of information is also necessary to monitor its progress, while evaluation and audit of the performance conclude the exercise. Since statistical methods used generally vary from project to project depending on their nature, the Commission is not in a position to make any general recommendation, except that the Government should make use of the statistical expertise of the NSO in planning, monitoring and evaluation of large projects of national importance.
14.9.2 Instead, the Commission has examined the more pointed issue of control and certification of quality of statistics and the process of their generation.
14.9.3 Statistical audit is necessary to assure the user not only of the quality of data presented, but also of the soundness of concepts, definitions and of the entire system of collection, processing, summarisation and dissemination of data. Currently, the CSO is responsible for setting up standards in respect of concepts and definitions, but there is no system of certification of either the process of generation, or the generated statistics. It is envisaged that the proposed National Commission on Statistics would be formally responsible for certification of the ‘quality’ of Core Statistics generated by the official statistical system, including the data collected through administrative records, surveys and censuses and would get this accomplished departmentally through the NSO. Certification of the quality of statistics may be extended to private producers of statistics on a voluntary basis.
Experience in Different Countries
14.9.4 Different countries have different approaches to the assessment of data quality. Since 1988, Statistics Sweden has produced an annual quality report. This report deals with the external factors affecting the quality of the statistical product This provides a basis for the development of the quality in their statistical products. For American Housing Survey, Quality Profiles are produced gathering together into one document, information on all aspects of data quality. The quality profile includes sections on sample design, data collection procedures, non-response error, measurement errors, data processing, weighting, sampling and a comparison of survey data with external data. Canada and Australia have laid down Statistics Quality Guidelines. These guidelines have to be followed before publishing the results. The guidelines require that an assessment of the quality of statistics produced should be published with the results.
Determinants of Quality of Statistics.
14.9.5 The three most important determinants of the quality of statistics are validity, reliability and timeliness. Validity is concerned with the relevance of the collected statistics to the subject of the study. Reliability is assessed in terms of the extent to which the data are free from sampling and non-sampling errors. Statistics are timely if they are available when they are needed.
Internal and External Audit.
14.9.6 Concurrent audit of statistical activities is necessary for early detection of errors and mistakes during the progress of work, and their rectification in time. This is essentially an internal activity of the data-collecting organisation. Assessment and certification of the quality of the end product are done through an audit - by an external authorised auditor - of the final results embodied in a report and other related records and documents. The most important amongst all the items of information that should be made available for such an audit are listed below.
Recommendations.
Items for Audit.
14.9.7 A fairly long but not exhaustive list of items, which could be audited is given below:
Theoretical concepts and their modification into operationally feasible definitions, covering individual respondents, population of such respondents, sampling unit and information unit, frame and its adequacy, information to be collected from the respondents, etc.
Methods of data collection and handling: Interview, Direct observation or measurement, copied from records, mail enquiry, etc.; Steps to avoid respondents' or interviewers' bias; Treatment of sensitive questions; Deliberate redundancy to check consistency of information; Data transcription and scrutiny; Classification and coding; Choice of reference period and survey period; Design of questionnaire or schedule; Instruction manual.
In the case of secondary data acquired from administrative records – design of the form for recording and summarising, incomplete coverage and treatment of missing data, definitional consistency, checks on arithmetical errors.
Sampling design: uni-stage or multi-stage, procedure for stratification and selection at very stage; Formula for estimation of parameters.
Procedure for control and assessment of sampling and non-sampling errors; Manual and computerised procedures of scrutiny and editing of data; Methods for imputation of missing or rejected observations.
Data handling errors - Errors in data capture, editing, coding of open-ended textual responses, data processing, etc.
Obligation of Producers of Core Statistics.
14.9.8 It will be obligatory for every producer of Core Statistics to supply a copy of their report to the NCS for audit. The producer of the report should certify that all standards set by the NCS have been met, or draw attention to deviations, in case there are any. Their reports must contain information on all the above items for audit. In addition, the report should give its own assessment of the magnitudes of sampling and non-sampling errors associated with the more important statistics produced by them.
Audit Procedure.
14.9.9 The NCS may accept the producer’s certificate or carry out an audit of the Report and related documents through the NSO. For the purpose of the audit the NCS may call upon the producer to supply additional records or documents. The result of the audit would be communicated to the producer and would also be included in the annual report of the NCS.
