A report is given on a feasibility study for monitoring Australia's progress towards the effective use of online services. It is explained how the concept of "effective use" is adopted to indicate sustainable and valuable use of services, as seen from the context of all players in the delivery chain, viz. users, content providers and other service providers.
A prototype of an ongoing structure for acquiring and presenting information, across a number of key sectors (government services, business, education, health, residential and groups with special needs) has been developed. The paper concludes by establishing the critical need to relate such a monitoring process to national and state strategies, and objectives, for online services development.
Governments and industry in many countries have a major commitment to the use of online services. Much of what is happening is, however, still exploratory. How can we understand and share this experience? What assessment can we make of the effectiveness of strategies to promote the use of these services? What gaps and needs for action can be observed?
This paper reports on a feasibility study of monitoring Australia's progress towards effective use of online services. A prototype of an ongoing structure for acquiring and presenting information, across a number of key sectors - government services, business, education, health, residential, and groups with special needs - has been developed as a Web site for critical review. The approach taken in the feasibility study is outlined in the following sections with key conclusions about the development of an ongoing process.
A core issue in the study is the consideration of the concept of "effective use". This term has been adopted to indicate sustainable and valuable use of services. Value should be present for the key elements of the delivery chain - users, content service providers (government agencies, businesses, schools, etc.), and communications service providers.
Much of the available data regarding online services focuses on measures of access to services (through ownership or other means), and volume or nature of use of services. However, access to services does not necessarily mean they are used, using them does not mean they are used effectively - access and use are necessary but not sufficient conditions for effective use.
Initially the project was conceived as the development of a "report card" on Australia's progress towards effective use of online services. However, because the term "report card" was found to project negative connotations in various sectors, particularly in the context of national-state relationships, its use has gradually been reduced. The term also suggested a critical appraisal of the state of development, against some identified standards or expectations, or even State to State. In contrast, the value of a monitoring process has increasingly been seen in its contribution to development, the engendering of a learning process about the use of online services.
The nominal objectives of a monitoring process are to assist the evaluation of national, state, and organisational strategies through:
While all of these objectives are seen as relevant, the first two have inherent difficulties in establishing stable bases of comparison over time and context. The latter has been emphasised in consultations, leading to an intended focus on identifying and monitoring:
|
Conclusion 1: Monitoring process as a developmental, evaluative activity An ongoing monitoring process should emphasise developmental aspects, rather than critical appraisal. A primary contribution of a monitoring process should be to support the evaluation of national and state strategies for development of online services. |
A desirable starting point for monitoring the outcomes of strategies is to utilise understood and accepted statements of strategic objectives. Australia has had a number of vehicles at the national level for the development, expression and implementation of online, interactive services development in recent years. For various reasons they have either not set objectives or these objectives have not been sufficiently supported to be taken as bases for assessment.
An examination of statements of vision and objectives from various sources led to adoption of the following as a point of reference for this project:
The use of online services in Australia should
in order to improve the social and economic well-being of its citizens.
More explicit, and best described as instrumental or intermediate objectives can be found among agencies in different sectors; one being the objective of 90% of businesses in targeted industries being online, another being the reasonably general objective of all government departments and agencies having a Web presence and the capacity to conduct some/all of their transactions online.
There are arguments for and against explicit policy objectives. There is probably no argument, however, that if we are to seek to monitor progress on strategies we wish to do more than gather all the interesting information that is possible. There is a clear task to agree upon a framework of objectives that should be monitored.
Coincident with the production of this report, the Ministerial Council for the Information Economy released a statement Towards an Australian Strategy for the Information Economy (July 1998) as a basis for consultation. This included the following mission statement:
To ensure that the lives and work of Australians are enriched, jobs are created, and the national wealth is enhanced, through the participation of all Australians in the information economy
It also provides a source of relatively detailed objectives, such as:
Through this structure a set of long- and medium-term national objectives is becoming apparent. The integration of this set with the framework of this report is a clear next stage of development.
It has been made clear to us in many discussions that focusing on online services - the technology enablers of activities - is unduly reductionist. Organisational strategists and managers are recognising that online services provide one channel for delivery of, or access to, other "consumable" products and services. The use of online services needs to be examined in association with other more traditional physical or electronic channels, in a context of broad business, economic and social objectives.
Conclusion 2: Relationship of a Monitoring Process to National Objectives
A monitoring process needs to be related to a concise statement of a national strategy for online services development. This strategy should have
- online services as one of a number of delivery/access channels,
- the need for cooperative endeavour between government at all levels, industry, business and community to build a national base of expertise, and the need for commitment to continuity of national purpose and implementation approaches.
The consultation document Towards an Australian Strategy for the Information Economy now serves as the basis for this development.
A significant part of this feasibility study has been the development of a framework for identifying potentially useful information. We have:
|
Equipment enabling service delivery has been purchased or the service can be subscribed to; or without purchase, it is physically possible to gain access to such a service or enabling equipment. |
|
Quantification of the actual use or operation of the service, eg. in terms of usage volume or frequency of use. |
|
A measure of the actual or perceived value gained from use of the service, eg. satisfaction, affordability, return on investment, value for money, usefulness, extent of repeat use. |
Applications by Sectors
Market Presence
Consumer / Client Management
Supply Chain Management
Work Group Collaboration
Electronic Services Delivery (ESD)
Inter & Intra-Department/Agency Communications
Whole-of-Government Information Locator (or Directory)
Classroom/remote delivery
Development of learning resource materials
Management of learning
Collaborative projects
Professional development
Information for prospective clients
Administration
Health Information Management
Professional Development and Continuing Education
Clinical Consultation and Diagnosis
Family/Social
Money
Shopping
Education
Entertainment
Health
Transport
Compliance
From the point of view of:
Users Usefulness (meets needs)
Affordability
Ease of use
Content service providers Achievement of business/organisational (business, government agencies, strategy through, e.g.,
schools, etc) - Increased productivity
- Increased range of services
- Increased market access
Communications service providers Return on investment
Extension of range and quality of service
Market access/market share
Our primary focus has been on the consideration of the points of view of the first two groups of stakeholders, with the approach to addressing the questions about value to communications service providers an outstanding matter.
A general framework for consideration of possible measures is included as Attachment 1.
Overall, we have found relatively few attempts to assess aspects of effective use; the focus being at this stage more on the preconditions of access and use. We have, however, found a general recognition of the appropriateness of an objective of effective use, tempered only by some observations that political and other strategic imperatives to be online may override consideration of the issues of value.
A focus on effective use is one way of beginning to understand the relationship between the use of online services and broader objectives. Figure 1 illustrates conceptually how the "value to content service providers" and "value to users" dimensions may relate to the broader economic and social objectives that the use of online services may be intended to serve.
Figure 1: Relationship between Macro Objectives and "Effective Use"
|
Conclusion 3: Effective use as a long term objective; change issues as a primary focus An objective of effective use of online services should be an integral part of national, state, and organisational strategies. It is, however, a long term objective. Intermediate, more monitorable, policy objectives are likely to be expressed in terms of access or use; their expression may be influenced by recognising the nature of longer-term objectives. A primary focus of any policy review process should be on the informed identification and analysis of the change issues to be resolved to achieve effective use. These change issues should also form a basis for identifying intermediate policy objectives. |
As part of the project we conducted a survey of the range of Australian and international data sources. In examining potential sources of information we distinguished a variety of ways in which data may be obtained. These sources include:
a. Services related
An increasing amount of data is available from the systems underlying the services themselves; e.g., the kinds of transactions, time of day, etc., for government electronic service delivery, or usage statistics associated with Websites. While these are clearly useful sources of information, there needs to be greater clarification of the meaning of certain statistics (e.g., number of hits or pages opened on Websites).
b. Studies
There are a variety of studies from which information can be obtained, and which have characteristic differences. They include:
Samples of total populations:
The most broadly based and regular are those conducted by the Australian Bureau of Statistics (householder and small business).
Online surveys:
Various organisations specialise in online surveys of users of the Internet. These provide valuable information about online users, but inherently provide a skewed sample of the total population of users and non-users.
Qualitative research/case studies:
More intensive studies of particular groups, based on interviews or observation, are conducted by a number of organisations.
c. Panels
A number of questions will be best addressed by the views of expert panels. These will generally need to be formed for the purpose.
In practice a mix of these approaches will be desirable. Linkages between them will provide some powerful sources of information. Measures of access and use are likely to be obtained from population surveys and services-related data gathering measures. The issues of value in effective use are likely to be best considered through qualitative, case study and panel approaches.
In seeking to obtain information from a number of different sources some clear issues of comparability of measures arise.
Taking the example of residential use of the Internet:
It is from the approach to data collection and to questions such as these that significantly different statistics emerge. There is much data we have chosen not to present because of issues of this nature, particularly in the international domain. Is it, for instance, useful to compare Australian statistics of numbers of individuals using the Internet for online banking in the last year (0.3%) with available international data on time spent on the Internet on online banking (2.0% US)?
We have only used data available from public sources, or proprietary data which has been made available. There is a considerable amount of proprietary data being collected, but the pursuit of such data would require greater clarity of requirements.
|
Conclusion 4: Australian Data Sources While there is a limited amount of data available in the form sought to populate the frameworks in this report (access and use in the residential and business sectors being best served), there is a significant network of data gathering and research organisations who are already gathering information, are prepared to extend their current focus, or who could contribute special expertise to assist ongoing monitoring. Some relevant data could be available reasonably simply through building data-capturing procedures into systems design. The Australian Bureau of Statistics has a key role to play in further developing its established focus on usage statistics. It should continue its consultative process to establish commonly recognised measures. |
In examining international approaches, we have found few instances of attempted systematic monitoring of objectives. We have assembled elements of data from a wide range of different sources.
An important point of reference has been the project conducted by Spectrum Strategy Consultants for the UK Department of Trade and Industry. Focusing on the use of information and communication technologies by business (particularly small and medium), this study is now in its third year. Initially based on secondary sources, it now largely utilises a primary survey conducted by the National Opinion Poll of businesses in the UK, US, Japan, Germany, and France. While we have endeavoured to take a broad cross-sectoral approach, there are important commonalities with the Spectrum project; particularly in their recognition of multiple paths to the "information society", their increasing focus on effective use and barrier or change issues, and their integration of various data sources.
The Australian Bureau of Statistics is actively involved in promoting the gathering of comparable international statistics through the OECD Information, Computer and Communications Policy (ICCP) Committee. From this engagement they have confirmed the dearth of comparative points of reference.
|
Conclusion 5: International links Comparison of experience with other countries is an underlying requirement for effective monitoring of strategies. To this end:
|
The structure of the material from this study, and the audience for whom it is initially intended, is suited to presentation online. Critical review is sought of the prototype Web site at: www.circit.rmit.edu.au/monausol. This site provides reports on each sector, reviewing the overall status of development and access and usage measures where available. For each application considered, the nature of indicators of effective use is examined. Since, generally speaking, data is not readily available to assist the understanding of effective use, trial measures are suggested.
The government electronic service delivery application provides the following examples of suggested measures for users and government agencies. In this application particular reference is made to the activities of the Victorian Government in utilising the maxi system and the Australian Capital Territory Government in developing the Austouch system.
Table 1: Government Electronic Service Delivery - Trial Measures of Effective Use: Users
|
Indicator |
Trial Measures |
Available Data* |
|
Usefulness (in meeting needs):
|
|
|
|
Affordability of service (incl. cost effectiveness) |
|
|
|
Ease of Use:
|
|
|
* Data of this nature is presently collected for the maxi and Austouch systems but the results are not in the public domain.
Table 2: Government Electronic Service Delivery - Trial Measures of Effective Use: Government Agencies
|
Indicator |
Trial Measures |
Available Data* |
|
Improved productivity/leverage |
|
|
|
Improved range of services |
|
|
|
Increased market access |
|
|
We have sought confirmation of this approach through ongoing consultation with funders and other interested parties. We have sought to develop panels of experts to review each sector, and conducted workshops and seminars in most sectors. The significant interest shown suggests the merit of further development of this process.
However, there have been limitations on cooperation in obtaining information. Some of these are time limitations of busy people; others are more fundamental. It is apparent that an ongoing process will need to have characteristics of:
Within a number of sectors, while interest is clear, the view has been expressed that the provision of detailed information is dependent on the acceptance of the bona fides of the process as a legitimate vehicle for review of strategies.
Even if conducted under the auspices or endorsement of key bodies, the monitoring process is unlikely to be successful unless there is an appropriate degree of ownership within the sectors. Bodies with interests in the development of online services in each sector are identified as possible owners of this process.
The process of establishing ownership will need to deal with some tension that appears to exist between central agencies promoting online strategies and service agencies in the same jurisdiction, and in a somewhat competitive form between different jurisdictions. The monitoring process has been positioned as a co-operative, developmental activity, which will need continual reinforcing and interpreting.
Contributions of information are also unlikely to be readily forthcoming if they impose additional resource requirements. The process of development and maintenance will need to be closely integrated with existing, or desirable new, initiatives.
With this ownership and integration could come, however, limitations on the maintenance of a critical reflection of developments. This should be nurtured through review fora and other processes.
|
Conclusion 6: Developing an ongoing process An ongoing monitoring process should be developed and owned by the National Office for the Information Economy and supported by the Ministerial Council for the Information Economy and the Online Council. It should be closely associated with the development of the National Strategy for the Information Economy. Specific bodies should be approached to develop and oversee monitoring activities in the different sectors. Consideration should be given to inclusion of other sectors; in particular a focus on cultural and other information and entertainment applications. Individual States have their own potential use of a monitoring process for evaluation and development of strategies, as well as being important contributors to a national monitoring process. Common frameworks and approaches linking state and national developments are likely to be of value. |
The contributions to this project from John Burke, Sarah Miller and Supriya Singh are gratefully acknowledged.
Attachment 1: Framework for Measures by Sector
|
ACCESS (Access by members of group to specific services) |
USE (Use of specific services) |
EFFECTIVE USE (Value in relation to particular kind of use/application) |
|
By identified group: No. (%) who - have (own/subscribe to) services - have, without ownership
access to services (define levels)
Change/Barrier issues (identify relevant issues and define indicators for sector); e.g., Location Cost Infrastructure availability
|
By service Amount of use of service
Change/Barrier issues; e.g., Awareness Skills Ease of use Availability of support staff Demand |
By application/activity: Measures of the following nature as appropriate for activity Users Meets needs/Usefulness (satisfaction with outcome of activity) Affordability/value for money Accessibility Content Cost (relative to physical/traditional) Service Productivity/leverage Provider Extension of range and quality of services Market access Comms Return on investment Provider Extension of range and quality of services Market access Change/Barrier issues; e.g, Organisational acceptability - e.g., timetabling Personal acceptability - e.g., security, trust Technology - e.g., effectiveness of resource discovery tools Policy/regulatory - e.g., applicability of Medicare |