We have access to ever-increasing amounts of digital data. Yet we know little about how digital technologies are transforming employers’ work practices in British firms. This is because we lack representative, reliable data about these practices at the organisation-level. Existing data tend to:
draw from independent consultancy projects and reports;
be relatively narrow in scope or rely on potentially biased online surveys;
have a focus on larger organisations at the forefront of digital innovation (this gives little insight into the extent to which digital technologies are being used in smaller and medium sized businesses);
offer little information about human resources and the impacts on workers.
This commentary examines the limitations of existing employer-level datasets and the challenges for policymakers attempting to steer the digital transformation of work in the absence of a rigorous evidence base.
Digit’s new representative, employer survey of digital practices at work aims to fill this evidence gap. It will show the extent to which new digital technologies are being adopted, and how they are being used and adapted, by a wide range of UK employers.
1 A policy discourse dominated by incomplete, non-representative evidence
Due to a lack of robust, representative data, surveys by international consulting firms have become go-to data sources for policymakers interested in understanding the uptake of digital technology by employers. A recent McKinsey report, for example, argues that ‘COVID-19 has pushed companies over the technological tipping point – and transformed business forever’, leading to enhanced profitability for digital adopters. Such claims are widely cited in policy documents, reports of international institutions and academic discourse. To take one example, Chris Philp, MP, then UK Government Minister for Tech and the Digital Economy stated that:
The UK’s economic future, jobs, wage levels, prosperity, national security, cost of living, productivity, ability to compete globally and our geo-political standing in the world are all reliant on continued and growing success in digital technology. (Department for Digital, Culture, Media & Sport, 2022)
A key pillar of the UK Government’s focus on digital technologies is artificial intelligence (AI). AI is seen to ‘rewrite the rules of entire industries, drive substantial economic growth and transform all areas of life’. Academic debate, likewise, is increasingly concerned with understanding the effect of AI on work and work relationships. However, existing survey data either has methodological weaknesses and/or is unable to capture how emerging digital technologies and use of AI impact work and employment practices, from an employer’s perspective. This data gap hinders both efficient policymaking and impactful academic debate.
2 The limitations of current surveys
2.1 Established UK Secondary Employer Surveys: methodologically robust but digitally outdated
Comprehensive UK employer-based surveys related to work and employment relations include the 2011 Workplace Employment Relations Survey (WERS), the 2019 Employers’ Skills Survey (ESS) or the 2002 Change in Employer Practices Survey (CEPS-02). They are methodologically robust and rich in employment relations related questions but are not currently designed to capture developments in digital technology. These well-established employer surveys (e.g., WERS) lack detail in the differentiation of different types of digital technology, particularly AI or machine learning-enabled technologies (ESS).
2.2 Employer Surveys by International Institutions: issues of scope and small samples
Similarly, surveys by international bodies, such as the Eurofound, the Organisation for Economic Co-operation and Development (OECD) or the European Investment Bank, follow a robust sampling methodology, generally drawing on representative samples obtained using computer-assisted telephone interviews. But such surveys are limited in scope and depth of questions concerning digital technology or work and employment matters.
For example, the 2019 Eurofound Company Survey differentiated between computer use, the purchase of customised software, the use of robots, the use of data analytics and engagement in E-commerce. Yet, it still lacks nuance in relation to different types of AI-enabled equipment or software by only including a binary question on ‘use of robots’. Nor does the survey ask about the reasons for investing in digital technologies and whether (and how) it has been applied in corporate decision-making.
By contrast, the European Enterprise Survey and the European Investment Bank Investment Survey, conducted in 2020 and 2021, respectively, are much more detailed around AI-enabled technologies but only incorporated a limited number of questions on employment relations matters and employer work practices.
The international survey that comes closest to combining data/questions on work practices and the adoption of digital technologies is a new employer survey by the OECD on AI. Although this also follows a robust data collection methodology, the survey exclusively focuses on the adoption and use of AI technologies in the finance and manufacturing sector. This neither allows users to draw more general conclusions about the economy as a whole, nor to understand potential relationships between AI technologies and more established information and communication technologies.
Another issue common in all of the above-listed international surveys is the relatively small sample size per country. This makes these surveys less useful for within-country analyses and for conclusions about UK employers.
In summary, then, policy-makers and the academic community have predominantly relied on survey data that is incomplete or carries a significant risk of bias.
2.3 Independent surveys: more scope but methodologically flawed
Considering the lack of representative data on the proliferation of AI and machine learning-enabled technologies in UK organisations, it is not surprising that government departments and bodies have turned to reputable consulting firms. For example, the Department for Digital, Culture, Media & Sport commissioned both Ernst & Young and Capital Economics to conduct surveys around the proliferation of AI-enabled digital technologies in British businesses.
These and other independent surveys by international consulting firms served as an important initial step in exploring the use of digital and AI-powered digital technologies in general and specifically during the COVID-19 pandemic. Yet, it is important to acknowledge the limitations of these surveys when evaluating the reliability of their conclusions.
First, the samples used were not representative, that is, large firms with high capitalisation were over-sampled.
Second, they tend to omit the public sector. This might be perceived as less problematic given that private enterprises constitute the vast majority of the population of UK enterprises. But, with 5.7 million employees (ONS, 2022), the public sector is one of the largest UK employers and some public sector organisations are at the forefront of digital innovation.
Third, data in these as well as other well-cited reports were collected through online surveys. While the benefits of online surveys are well-known (e.g., easy access, low costs and in some instances lower bias due to reduced personal interaction (Marlar, 2018), they usually come with selection bias that is very hard to mitigate (Bethlehem, 2010). For example, digital-savvy managers are more likely to participate in an online survey than their less digitally competent counterparts. Most of these surveys are based on samples of convenience rather than more reliable probabilistic, stratified, or complex samples. Considering the need to establish a reliable picture across sectors and industries, estimates from online surveys should be taken with caution.
Last, independent surveys and reports were either solely focused on digital technologies (Capital Economics, 2022; Ernst and Young, 2021), or included very limited work aspects, mostly looking at business performance in terms of profit or productivity (McKinsey and Company, 2020a). Thus, the association between digital technologies and human resource practices, workers’ experiences, management control, training, skills and representation cannot be addressed.
As with existing representative data, independent surveys provide limited evidence concerning the relationships between novel digital technologies and work outcomes in the UK. These considerations were behind Digit’s initiative to design and conduct a large-scale, robust and representative survey at firm-level: the Employers’ Digital Practices at Work (DPaW) survey.
3 Digit’s new survey on employers’ digital practices at work
The main aim of the DPaW was to develop a representative understanding of the extent of the adoption of and investment in different types of digital technologies. The focus is mostly on AI-powered digital technologies, employers’ use of data analytics and associated human resource management practices. The survey is designed to inform key change-makers – policymakers and practitioners – of the potential benefits and challenges associated with the adoption and application of digital technologies at work.
To achieve these aims, we designed the survey with four principles in mind:
Capture a broad and robust sample, representative of the population of UK employers.
Cover a detailed range of different types of digital technologies, including various types of AI-enabled technology.
Include measures of a variety of work practices and employment matters including work organisation, employee autonomy, skills and training, pay, employment outlook and employee voice.
Facilitate possible data linkage with existing administrative databases.
3.1 Robust, representative, reliable data
One of the benefits of DPaW is its comprehensive survey design and sampling strategy. The sample of 2001 employers was drawn from the Dun and Bradstreet business directory – one of the largest and most recognised business databases that includes over 500 million unique organisations globally. To cover the appropriate range of organisations, we used a stratified sample that looked at the intersection of different categories of employer size (from 10-19 employees to 1000+), industry sectors (based on the Standard Industrial Classification) and regions (nine regions in England, Scotland, Wales and Northern Ireland). The sample corresponds to more than 260,000 firms that employ over 23.7 million workers across the United Kingdom. Senior managers responsible for human resources, IT or digital investment were contacted. To remove confounding selection biases that could emerge via use of online surveys to measure digital acceptance and adoption, employers were interviewed via the computer-assisted telephone interview (CATI) method. On average, interviews lasted for around 25 minutes.
We further split the sample in two parts: a larger nationally representative sample (1501 employers responded) and a smaller booster sample (500 employers responded). The objective of the booster sample was to capture a higher share of digital adopters to enable more precise estimates in our analysis of the impact of digital adoption on employment practices and organisational outcomes. The potential digital adopters for the booster sample were identified from previous studies on the uptake of AI, including some of the above-mentioned surveys, leading to a focus on larger employers with at least 100 employees from six different industrial sectors. This led to a higher share of adopters of new digital technologies in the sample, while the inclusion of sampling weights restored the overall representativeness of the data.
The final questionnaire consisted of five sections:
Employer demographics.
Investment and use of new, AI- or machine learning-powered digital technologies.
Employers’ use of data analytics.
Employers’ use of information and communication technologies.
Employment and work practices.
3.2 Coverage of an extensive range of digital technologies
The survey differentiated between three broad types of digital technologies. First, it included questions about well-established information and communication technologies (ICT), distinguishing between ICT hardware and software. Questions on ICT hardware focused on the share of employees by each employer that used: desktop computers; portable devices such as laptops and tablets; smart functions of smartphones; and wearables such as sensors and scanners in day-to-day tasks. In addition, we looked at different software types, including enterprise planning, video conferencing, customer relations, collaboration software and customised HR applications.
Second, the data analytics section aimed to develop a detailed picture by including questions on:
The extent of data analytics use by UK employers
Sources of digital data
Data management
Use of data analytics for human resource and strategic functions
The use of data analytics to inform decision making by managers or to create algorithms that make their own decisions
Challenges in using data analytics
To explore the extent of progress, we included retrospective questions on how the use of data analytics had changed over the past five years, since the beginning of the COVID-19 pandemic and how managers estimate it will change in the coming two years.
Lastly, the survey looked into new forms of digital technology defined as artificial intelligence and machine learning-enabled technology, referring to both equipment and software. We asked managers about specific types of AI-enabled digital technologies:
AI-enabled equipment including robots, drones, robot process automation, 3-D printing, biometric authentication.
AI-enabled applications and algorithms including content creation; natural language processes, image, text, voice recognition software; virtual reality; decision-making, marketing automation software.
Worker business support tools powered by AI such as diagnostic tools, visual assistants and chatbots.
Peer-to-peer networks such as blockchain and cryptocurrency.
Internet of Things, edge computing (devices connected though sensors or WIFI to store and share data).
Cloud computing (running communications remotely over the internet using commercial data centres).
To understand more about the adoption and application of these technologies, the survey incorporated questions behind the reasons for investing, the areas of organisational activity most affected, changes in use and challenges encountered. Importantly, we also asked those whom had not invested in these new digital technologies about their reasons for not doing so as well as about their plans for investing in the future.
In sum, the DPaW survey allows for both depth and breadth when examining the use of digital technologies in UK businesses; in some cases in more detail than previous surveys that only focused on the use of digital technologies.
3.3 Wide range of work practices
In their report The New Frontier: Artificial Intelligence at Work the All Party Parliamentary Group (APPG) for the Future of Work concludes that “AI offers invaluable opportunities to create new work and improve the quality of work if it is designed and deployed with this as an objective”. To establish more clarity on how digital technologies, and in particular AI, are currently interacting with work, a large part of the DPaW survey focused on different types of human resource management practices. This included:
Skills and training – specifically the extent to which employers are investing in digital skills upgrading or facing challenges in acquiring or retaining appropriate skills.
Work organisation, management control and employee autonomy – this allows for examination of the ways in employers may be adopting forms of algorithmic control, or the extent to which digitalisation may facilitate worker autonomy or forms of teamwork.
Employment and working time – this included retrospective questions on the creation or reduction in jobs in the past, in response to Covid-19 as well as plans for the future. Likewise, changes in working hours were considered, as well as questions on remote working and overtime.
Representation and voice – the survey asked a variety of questions around the extent to which employers involved their workforce in decision-making around key employment practices, including training, pay setting and investment in new technology.
Reward practices – finally, we asked about pay. How is it being set up, what are its determinants, how much are employees being paid? These questions will enable researchers to link investment in digital technology with pay distribution and performance related pay.
4 A reliable evidence-base to understand the proliferation and outcomes of digital technologies
The survey is based on a robust sampling strategy that will produce representative data on the extent to which UK employers are making use of types of digital technologies.
Combining this with data about a wide range of essential work practices that are important for and relate to high-involvement, high-quality work practices directly relates to open questions by political stakeholder such as the APPG Future of Work. Findings from the survey thus have the potential to enable change-makers to base their decisions on trustworthy data.
Dr Felix Schulz, Digit Research Fellow, Leeds University Business School.
Dr Danat Valizade, Digit Co-Investigator, Associate Professor in Quantitative Methods, Leeds University Business School.
Professor Mark Stuart, Digit Co-Director, Montague Burton Professor of Human Resource Management and Employment Relations, Leeds University Business School.
References
All-Party Parliamentary Group on the Future of Work (2021). The New Frontier: Artificial intelligence at work. Bethlehem, J. (2010). Selection Bias in Web Surveys. International Statistical Review, 78(2): 161-188.
Capital Economics (2022). AI Activity in UK Businesses: An assessment of the scale of AI activity in UK businesses and scenarios for growth over the next twenty years. January 2020.
Department for Digital, Culture, Media & Sport (2022). UK Digital Strategy. Policy Paper updated 4 October 2022.
Ernst and Young (2021). Data foundations and AI adoption in the UK private and third sector. July 2021.
HM Government (2021). National AI Strategy. Command Paper 525.
Podsakoff. P.M., MacKenzie, S.B., Lee, J.-Y. and Podsakoff, N.P. (2003). Common Method Biases in Behavioral Research: A Critical Review of Literature and Recommended Remedies. Journal of Applied Psychology, 88(5): 879-903.
Missing Data: Understanding UK firms’ investment in and use of new digital technologies at work
Felix Schulz, Danat Valizade and Mark Stuart
18 May 2023
Summary
We have access to ever-increasing amounts of digital data. Yet we know little about how digital technologies are transforming employers’ work practices in British firms. This is because we lack representative, reliable data about these practices at the organisation-level. Existing data tend to:
This commentary examines the limitations of existing employer-level datasets and the challenges for policymakers attempting to steer the digital transformation of work in the absence of a rigorous evidence base.
Digit’s new representative, employer survey of digital practices at work aims to fill this evidence gap. It will show the extent to which new digital technologies are being adopted, and how they are being used and adapted, by a wide range of UK employers.
1 A policy discourse dominated by incomplete, non-representative evidence
Due to a lack of robust, representative data, surveys by international consulting firms have become go-to data sources for policymakers interested in understanding the uptake of digital technology by employers. A recent McKinsey report, for example, argues that ‘COVID-19 has pushed companies over the technological tipping point – and transformed business forever’, leading to enhanced profitability for digital adopters. Such claims are widely cited in policy documents, reports of international institutions and academic discourse. To take one example, Chris Philp, MP, then UK Government Minister for Tech and the Digital Economy stated that:
A key pillar of the UK Government’s focus on digital technologies is artificial intelligence (AI). AI is seen to ‘rewrite the rules of entire industries, drive substantial economic growth and transform all areas of life’. Academic debate, likewise, is increasingly concerned with understanding the effect of AI on work and work relationships. However, existing survey data either has methodological weaknesses and/or is unable to capture how emerging digital technologies and use of AI impact work and employment practices, from an employer’s perspective. This data gap hinders both efficient policymaking and impactful academic debate.
2 The limitations of current surveys
2.1 Established UK Secondary Employer Surveys: methodologically robust but digitally outdated
Comprehensive UK employer-based surveys related to work and employment relations include the 2011 Workplace Employment Relations Survey (WERS), the 2019 Employers’ Skills Survey (ESS) or the 2002 Change in Employer Practices Survey (CEPS-02). They are methodologically robust and rich in employment relations related questions but are not currently designed to capture developments in digital technology. These well-established employer surveys (e.g., WERS) lack detail in the differentiation of different types of digital technology, particularly AI or machine learning-enabled technologies (ESS).
2.2 Employer Surveys by International Institutions: issues of scope and small samples
Similarly, surveys by international bodies, such as the Eurofound, the Organisation for Economic Co-operation and Development (OECD) or the European Investment Bank, follow a robust sampling methodology, generally drawing on representative samples obtained using computer-assisted telephone interviews. But such surveys are limited in scope and depth of questions concerning digital technology or work and employment matters.
For example, the 2019 Eurofound Company Survey differentiated between computer use, the purchase of customised software, the use of robots, the use of data analytics and engagement in E-commerce. Yet, it still lacks nuance in relation to different types of AI-enabled equipment or software by only including a binary question on ‘use of robots’. Nor does the survey ask about the reasons for investing in digital technologies and whether (and how) it has been applied in corporate decision-making.
By contrast, the European Enterprise Survey and the European Investment Bank Investment Survey, conducted in 2020 and 2021, respectively, are much more detailed around AI-enabled technologies but only incorporated a limited number of questions on employment relations matters and employer work practices.
The international survey that comes closest to combining data/questions on work practices and the adoption of digital technologies is a new employer survey by the OECD on AI. Although this also follows a robust data collection methodology, the survey exclusively focuses on the adoption and use of AI technologies in the finance and manufacturing sector. This neither allows users to draw more general conclusions about the economy as a whole, nor to understand potential relationships between AI technologies and more established information and communication technologies.
Another issue common in all of the above-listed international surveys is the relatively small sample size per country. This makes these surveys less useful for within-country analyses and for conclusions about UK employers.
In summary, then, policy-makers and the academic community have predominantly relied on survey data that is incomplete or carries a significant risk of bias.
2.3 Independent surveys: more scope but methodologically flawed
Considering the lack of representative data on the proliferation of AI and machine learning-enabled technologies in UK organisations, it is not surprising that government departments and bodies have turned to reputable consulting firms. For example, the Department for Digital, Culture, Media & Sport commissioned both Ernst & Young and Capital Economics to conduct surveys around the proliferation of AI-enabled digital technologies in British businesses.
These and other independent surveys by international consulting firms served as an important initial step in exploring the use of digital and AI-powered digital technologies in general and specifically during the COVID-19 pandemic. Yet, it is important to acknowledge the limitations of these surveys when evaluating the reliability of their conclusions.
As with existing representative data, independent surveys provide limited evidence concerning the relationships between novel digital technologies and work outcomes in the UK. These considerations were behind Digit’s initiative to design and conduct a large-scale, robust and representative survey at firm-level: the Employers’ Digital Practices at Work (DPaW) survey.
3 Digit’s new survey on employers’ digital practices at work
The main aim of the DPaW was to develop a representative understanding of the extent of the adoption of and investment in different types of digital technologies. The focus is mostly on AI-powered digital technologies, employers’ use of data analytics and associated human resource management practices. The survey is designed to inform key change-makers – policymakers and practitioners – of the potential benefits and challenges associated with the adoption and application of digital technologies at work.
To achieve these aims, we designed the survey with four principles in mind:
3.1 Robust, representative, reliable data
One of the benefits of DPaW is its comprehensive survey design and sampling strategy. The sample of 2001 employers was drawn from the Dun and Bradstreet business directory – one of the largest and most recognised business databases that includes over 500 million unique organisations globally. To cover the appropriate range of organisations, we used a stratified sample that looked at the intersection of different categories of employer size (from 10-19 employees to 1000+), industry sectors (based on the Standard Industrial Classification) and regions (nine regions in England, Scotland, Wales and Northern Ireland). The sample corresponds to more than 260,000 firms that employ over 23.7 million workers across the United Kingdom. Senior managers responsible for human resources, IT or digital investment were contacted. To remove confounding selection biases that could emerge via use of online surveys to measure digital acceptance and adoption, employers were interviewed via the computer-assisted telephone interview (CATI) method. On average, interviews lasted for around 25 minutes.
We further split the sample in two parts: a larger nationally representative sample (1501 employers responded) and a smaller booster sample (500 employers responded). The objective of the booster sample was to capture a higher share of digital adopters to enable more precise estimates in our analysis of the impact of digital adoption on employment practices and organisational outcomes. The potential digital adopters for the booster sample were identified from previous studies on the uptake of AI, including some of the above-mentioned surveys, leading to a focus on larger employers with at least 100 employees from six different industrial sectors. This led to a higher share of adopters of new digital technologies in the sample, while the inclusion of sampling weights restored the overall representativeness of the data.
The final questionnaire consisted of five sections:
3.2 Coverage of an extensive range of digital technologies
The survey differentiated between three broad types of digital technologies. First, it included questions about well-established information and communication technologies (ICT), distinguishing between ICT hardware and software. Questions on ICT hardware focused on the share of employees by each employer that used: desktop computers; portable devices such as laptops and tablets; smart functions of smartphones; and wearables such as sensors and scanners in day-to-day tasks. In addition, we looked at different software types, including enterprise planning, video conferencing, customer relations, collaboration software and customised HR applications.
Second, the data analytics section aimed to develop a detailed picture by including questions on:
To explore the extent of progress, we included retrospective questions on how the use of data analytics had changed over the past five years, since the beginning of the COVID-19 pandemic and how managers estimate it will change in the coming two years.
Lastly, the survey looked into new forms of digital technology defined as artificial intelligence and machine learning-enabled technology, referring to both equipment and software. We asked managers about specific types of AI-enabled digital technologies:
To understand more about the adoption and application of these technologies, the survey incorporated questions behind the reasons for investing, the areas of organisational activity most affected, changes in use and challenges encountered. Importantly, we also asked those whom had not invested in these new digital technologies about their reasons for not doing so as well as about their plans for investing in the future.
In sum, the DPaW survey allows for both depth and breadth when examining the use of digital technologies in UK businesses; in some cases in more detail than previous surveys that only focused on the use of digital technologies.
3.3 Wide range of work practices
In their report The New Frontier: Artificial Intelligence at Work the All Party Parliamentary Group (APPG) for the Future of Work concludes that “AI offers invaluable opportunities to create new work and improve the quality of work if it is designed and deployed with this as an objective”. To establish more clarity on how digital technologies, and in particular AI, are currently interacting with work, a large part of the DPaW survey focused on different types of human resource management practices. This included:
4 A reliable evidence-base to understand the proliferation and outcomes of digital technologies
The survey is based on a robust sampling strategy that will produce representative data on the extent to which UK employers are making use of types of digital technologies.
Combining this with data about a wide range of essential work practices that are important for and relate to high-involvement, high-quality work practices directly relates to open questions by political stakeholder such as the APPG Future of Work. Findings from the survey thus have the potential to enable change-makers to base their decisions on trustworthy data.
Initial findings from the Digit Employers’ Digital Practices at Work Survey will be published in June 2023. Register to be notified when the report is published.
Authors
References
All-Party Parliamentary Group on the Future of Work (2021). The New Frontier: Artificial intelligence at work. Bethlehem, J. (2010). Selection Bias in Web Surveys. International Statistical Review, 78(2): 161-188.
Capital Economics (2022). AI Activity in UK Businesses: An assessment of the scale of AI activity in UK businesses and scenarios for growth over the next twenty years. January 2020.
Department for Digital, Culture, Media & Sport (2022). UK Digital Strategy. Policy Paper updated 4 October 2022.
Ernst and Young (2021). Data foundations and AI adoption in the UK private and third sector. July 2021.
HM Government (2021). National AI Strategy. Command Paper 525.
Marlar, J. (2018). Why Phone and Web Survey Results Aren’t the Same. Gallup Methodology Blog. April 25 2018.
McKinsey & Company (2020a). How COVID-19 has pushed companies over the technology tipping point – and transformed business forever. October 2020.
McKinsey & Company (2020b). What 800 executives envision for the postpandemic workforce. September 2020.
Office for National Statistics (2022). Public sector employment, UK: June 2022.
Podsakoff. P.M., MacKenzie, S.B., Lee, J.-Y. and Podsakoff, N.P. (2003). Common Method Biases in Behavioral Research: A Critical Review of Literature and Recommended Remedies. Journal of Applied Psychology, 88(5): 879-903.
Share this: