Gen AI success starts with an effective pilot strategy

Generative artificial intelligence is all the rage, but how can enterprises actually harness the technology’s promise and implement it for value? What benefits can be expected and what challenges might arise?

Gen AI is a relatively new tool for organizations and individual users. And while many CIOs might have a fairly solid understanding of the technology, putting it into actual use at scale is something else entirely.

As with any other significant technology rollout, it’s generally better to walk before running. That means launching gen AI pilot programs first to get a feel for how it works and how it might benefit the business, and to help identify promising use cases across business functions.

Identifying worthwhile use cases

Hackajob, a company that provides a platform for organizations to find and recruit IT and developer talent, began piloting generative AI models in the second half of 2022 as part of an informal research and development initiative to explore emerging technology trends.

“The decision to launch this pilot phase was driven by a desire to stay ahead in the field, assess the potential applications of gen AI, and subsequently transition into targeted proof-of-concept projects,” says Vlad-George Iacob, vice president of engineering at Hackajob. “These initiatives helped us to understand the real-world applicability of gen AI in enhancing our efficiency and exploring new product development opportunities.”

One of the reasons the company is deploying gen AI is to help increase diversity and representation in technology teams, by removing bias from job descriptions. Employers can upload job descriptions to the Hackajob platform, where AI technology scans and assesses the content, creating suggestions for more inclusive language.

Employers can then implement suggested changes, ensuring that their recruitment materials are free from bias and appeal to a broad spectrum of potential candidates.

The initial engagement with gen AI was during a two-day hackathon with one of Hackajob’s product teams. “This phase included a company-wide ideation process, gathering insights from various parts of the business to prioritize and implement a selected idea,” Iacob says.

Following this initial success, gen AI technologies have been adopted by several product teams within the organization in various capacities, Iacob says.

“We have established a robust framework for leveraging gen AI across our product organization and the wider business,” he says, adding that, with a solid understanding of various technologies and use cases, the company now considers gen AI to be a viable solution for many challenges it faces.

Various Hackajob departments beyond its product teams have started adopting a variety of gen AI tools as well. “This demonstrates our company’s broader commitment to embracing and integrating this transformative technology, aligning it with our strategic goals, and enhancing our capabilities across various domains,” Iacob says.

Hackajob employs a multifaceted approach to identifying promising use cases for gen AI. One is an idea submission structure. “We utilize a system to capture product ideas from across the business, including specific gen AI ideas,” Iacob says. “By following a more targeted crowdsourcing approach for gen AI, we are frequently able to gather and consider fresh insights and applications.”

Another is education. “In the wider business, we’ve shared informal training and guidelines on using gen AI tools,” Iacob says. “This educational initiative empowers various teams with the knowledge and confidence to explore gen AI’s potential, and use it safely and appropriately. By suggesting a number of use cases, we further encourage experimentation and creative application.”

Fostering a culture of learning

Skillsoft, a provider of education services and technology, has been leveraging AI for some time to deliver personalized, dynamic, and interactive learning experiences for users.

“Most recently, our product teams have been piloting and successfully deploying new capabilities with gen AI in areas of content localization, content production, assessment creation, and assessment strategies, as well as within our search and recommendation engines,” says Orla Daly, CIO.

“Gen AI is an accelerator for Skillsoft to deliver capabilities to help our customer organizations and their people grow together,” Daly says.

Looking to achieve internal efficiency opportunities with gen AI, Skillsoft began with a Digital and IT Hackathon focused on leveraging ChatGPT to solve business problems or create new business value, Daly says. “But the real goal was to start to get people familiar with leveraging ChatGPT in a safe space, understand its potential and shortcomings in the context of real business use cases,” she says.

Since then, team members across product, engineering, content production, and the lines of business have been engaged in pilots and bringing new capabilities to market leveraging gen AI.

“In many examples, we have quickly moved from the pilot phase to new customer offerings or platform enhancements enabled by the responsible use of AI,” Daly says.

Skillsoft’s CAISY (Conversation AI Simulator), its gen AI-based tool designed to help professionals develop their business and leadership conversation skills, is an example of that.

The company will continue identifying opportunities to use gen AI to

deliver new capabilities. “We plan to leverage AI for driving adaptive learner journeys — meeting the learner where they are at — as well as to derive insights from usage data for [administrators] to understand how to best improve engagement and performance,” Daly says.

Skillsoft recently launched an Innovation Challenge, inviting all employees

to participate and get familiar with gen AI capabilities. “The winning solution from 18 teams will be showcased at the company town hall in September, highlighting the importance of embracing AI and making it part of our culture of innovation,” Daly says.

The company has also established an AI Advancement Council, with cross-functional representation. It’s charged with facilitating and advancing the integration of AI technology at Skillsoft, providing a collaborative forum and ensuring AI is effectively and responsibly used to drive innovation, efficiency, and growth, Daly says.

“We encourage all our team members to take advantage of our expansive content library, which offers a solid foundation on topics required for AI and AI-adjacent roles,” Daly says.

Democratizing AI education and ideation

Ceridian, a provider of human resources software, began its foray into gen AI by applying it within its Dayforce human capital management platform as a pilot program. Specifically, the company is using AI for self-service capabilities to enhance customer support.

The initiative was driven in the first quarter of 2023 by leaders within the customer support organization, says David Lloyd, chief data officer.

“Today, we are fine-tuning the model, using it internally first to achieve a strong correct answer rate,” Lloyd says. “Given our success in increasing the rate of answering directed questions to 85%, we are now working towards another 10% increase. In the future, we see the ability to use this approach for additional customer self-service as well as other resources.”

All of this is a byproduct of training the model against implementation guides, knowledge bases, and other internal documentation that is refined against customer questions and responses, Lloyd says.

Hundreds of internal customer support professionals are running all questions asked by customers and themselves through the platform, to determine the accuracy of the response from a reinforcement learning perspective, Lloyd says. “It will likely continue to be an internal support assistant as we consider when it can be provided to customers,” he says.

The key for Ceridian is democratizing how employees and partners think of AI. “We are focused on education, simple AI idea submission, and weekly ‘ask me anything’ sessions to discuss many topics,” Lloyd says.

The company’s Dayforce Lab provides an environment for testing and validating new AI technologies, as well as coaching and mentoring for business and product teams. “Dayforce Labs is complemented by our data science team, as there is no AI without data,” Lloyd says. “This includes carefully examining how data can be used to create better outcomes through AI.”

Every Ceridian employee can have a positive impact on the use of AI technologies, Lloyd says, whether it’s within internal business groups or via Dayforce. “In the end, we need to ensure that AI is being used to produce an outcome that has business value and supports our core principles,” he says.

Best practices for gen AI pilot success

For those enterprises looking to deploy gen AI starting with pilot programs, a few practices are worth considering.

Before introducing generative AI in the workplace, leaders should begin with a risk assessment to identify what risks are present to them as an organization in the use and potential misuse of the tool, Daly says. This could be misappropriation, plagiarism, or bias, to name a few.

“Establishing a sustainable, trustworthy, and transparent governance structure requires shared accountability from the company, its employees, and its providers,” Daly says.

Because the responsibility shouldn’t fall on one person or department, leaders might want to establish a cross-functional team to govern and help drive adoption, as well as create opportunities to identify use cases, Daly says. “Good use cases can come from anywhere in the organization, but need support to go from a good idea to a prototype or a production-ready solution,” she says.

In addition, Daly says, technology leaders should provide team members with the training and tools to get educated and stay current on the capabilities of gen AI. “Promote a culture of curiosity and learning, to keep pace with what is a fast-evolving space,” she says.

IT leaders should create a “safe space” for innovation, Iacob says. “It’s vital to encourage innovation without stifling creativity due to concerns around data protection risks or the use of proprietary data,” he says. “By providing clear training and guidelines, you can establish boundaries that protect essential assets while still allowing engineers and other team members the freedom to explore and experiment.”

This nurturing environment not only fosters innovation, but also builds confidence, ensuring that potential risks are well-managed without limiting the possibilities that gen AI offers, Iacob says.

It’s important to keep in mind that not all large language models are the same, Lloyd says. “Some have different attributes in the way they were trained,” he says. “This is important, as it may shape how you deploy the model. For example, do you want it to be more information than conversational? Do you want to use one that’s more fine-tuned for a specific industry or knowledge domain?”

Another good practice is to consider multiple models and stay up to date on technology developments.

“The gen AI space is evolving at a rapid pace, with new models and iterations frequently becoming available,” Iacob says. “It’s essential to maintain an open and explorative mindset, considering various models that fit your specific needs, and continuously staying abreast of the latest developments.”

By doing so, companies can position themselves to benefit from advancements in the field, tailoring solutions that are both innovative and aligned with the latest best practices, Iacob says.