Enterprise

Sleuth wants to use AI to measure software developer productivity

Comment

GettyImages 655147030
Image Credits: scyther5 / Getty Images

As knowledge workers, including software engineers, shifted to remote work during the pandemic, executives expressed a concern that productivity would suffer as a result. The evidence is mixed on this, but in the software industry particularly, remote work exacerbated many of the challenges that employees already faced. According to a 2021 Garden survey, the majority of developers found slow feedback loops during the software development process to be a source of frustration, second only to difficult communication between teams and functional groups. Seventy-five percent said the time they spend on specific tasks is time wasted, suggesting it could be put to more strategic use.

In search of a solution to bolster developer productivity, three former Atlassian employees — Dylan Etkin, Michael Knighten and Don Brown — co-founded Sleuth, a tool that integrates with existing software development toolchains to provide insights to measure efficiency. Sleuth today announced that it raised $22 million in Series A funding led by Felicis with participation from Menlo Ventures and CRV, which CEO Etkin says will be put toward product development and expanding Sleuth’s workforce (specifically the engineering and sales teams).

“With the avalanche of remote work brought on by the pandemic the need for developers, managers and executives to understand and communicate about engineering efficiency has increased sharply,” Etkin told TechCrunch via email. “Developers, no longer in the same room, need a way to coordinate around deploys and a quick way to discover when a deploy has gone wrong. Managers need an unobtrusive way to proactively learn about bottlenecks affecting their teams. Executives need an unobtrusive way to understand the impact of their organization-wide initiatives and investments. Sleuth takes the burden of understanding and communicating engineering efficiency off-line and makes it digestible by all.”

Etkin, Knighten and Brown were colleagues Atlassian, where they claim that they helped the company’s engineering organizations move from releasing software every nine months to releasing daily. Etkin was an architect on the Jira team before becoming the development manager at Bitbucket and StatusPage, while Knighten and Brown were a VP of product and an architect/team lead, respectively.

While at Atlassian, which grew from 50 to over 5,000 employees in the time that Sleuth’s co-founders worked there, Etkin says it became “crystal clear” that many engineering teams lack a quantitative way of measuring efficiency — and that this gap could hold them back from growing and improving.

“Measuring engineering efficiency is a known, large and growing problem that’s now become solvable. Because every company is investing more heavily into software engineering, the need for visibility into engineering efficiency has intensified,” Etkin said. “However, measuring efficiency has historically been very challenging for a multitude of reasons, namely tooling complexity, lack of access to data and use of dubious proxy metrics that bred micromanagement and distrust.”

Sleuth’s solution is DevOps Research and Assessment (DORA) metrics, an emerging standard used by developer teams to measure how long it takes to deploy code, the average time for a service to bounce back from failures and the how often a team’s fixes lead to issues post-deployment. DORA arose from an academic research team at Google, which between 2013 and 2017 surveyed over 31,000 engineers on DevOps practices to identify the key differentiators between “low performers” and “elite performers.”

Sleuth isn’t the only platform that uses DORA metrics to quantify productivity. LinearB, Jellyfish and Athenian are among the rival solutions that have adopted the DORA standard. But Etkin claims that its competitors don’t “fully or accurately” track these metrics.

“Sleuth is unique … because we employ deployment tracking to model how engineers are shipping their work from concept through to launch,” he explained. “Accurately modeling exactly how engineers ship across their pre-production and production environments and how they interact with issue trackers, CI/CD, error trackers and metrics allows Sleuth to build a fully automated … view of a team’s DORA metrics and their engineering efficiency.”

Sleuth uses AI to attempt to figure out a team’s baseline change failure rate (i.e. the percentage of changes that resulted in degraded services) and mean time to recovery — two of the four DORA metrics — from existing systems such as Datadog and Sentry. The platform can automatically determine when a metric is outside that baseline, Etkin says, and even automate steps in the development process to potentially improve on the metric.

From Sleuth’s project dashboard, individual teams can track their DORA metrics. An organization-wide dashboard reveals trends across different projects and teams.

“Customers just point Sleuth at … error data and Sleuth lets engineers know when they’ve pushed these metrics into a failure range. Using AI to determine these values means engineers can focus on their work without needing to understand every metric in their system or what ‘normal’ looks like for each.”

Sleuth
Tracking DORA metrics with Sleuth. Image Credits: Sleuth

DORA metrics aren’t the end-all be-all, of course. They can be a hindrance when an organization’s focus on them becomes all-consuming. As Sagar Bhujbal, VP of technology at Macmillan Learning, told InfoWorld: “Developer productivity should not be measured by the number of errors, delayed delivery or incidents. It causes unneeded angst with development teams that are always under pressure to deliver more capabilities faster and better.”

Etkin agrees, emphasizing that engineering managers need to avoid the temptation to micromanage.

“Engineering is a creative endeavor, and engineers are more similar to artists than assembly line workers,” Etkin said. Engineering managers need to … track the right metrics [and] track them accurately [but also] give engineers the tools they need to improve on the metrics.”

Sleuth customers vary from enterprises like Atlassian to startups, including LaunchDarkly, Puma, Matillion and Monte Carlo. Etkin says that the platform has tracked nearly a million deploys and undertaken over a million automated actions on behalf of developers. He declined to reveal revenue numbers when asked, but said that 12-employee Sleuth has grown 700% last year with a “very healthy” margin and cash flow.

More TechCrunch

The struggle isn’t universal, however.

Connected fitness is adrift post-pandemic

Featured Article

A comprehensive list of 2024 tech layoffs

The tech layoff wave is still going strong in 2024. Following significant workforce reductions in 2022 and 2023, this year has already seen 60,000 job cuts across 254 companies, according to independent layoffs tracker Layoffs.fyi. Companies like Tesla, Amazon, Google, TikTok, Snap and Microsoft have conducted sizable layoffs in the first months of 2024. Smaller-sized…

51 mins ago
A comprehensive list of 2024 tech layoffs

HoundDog actually looks at the code a developer is writing, using both traditional pattern matching and large language models to find potential issues.

HoundDog.ai helps developers prevent personal information from leaking

The changes are designed to enhance the consumer experience of using Google Pay and make it a more competitive option against other payment methods.

Google Pay will now display card perks, BNPL options and more

Few figures in the tech industry have earned the storied reputation of Vinod Khosla, founder and partner at Khosla Ventures. For over 40 years, he has been at the center…

Vinod Khosla is coming to Disrupt to discuss how AI might change the future

AI has already started replacing voice agents’ jobs. Now, companies are exploring ways to replace the existing computer-generated voice models with synthetic versions of human voices. Truecaller, the widely known…

Truecaller partners with Microsoft to let its AI respond to calls in your own voice

Meta is updating its Ray-Ban smart glasses with new hands-free functionality, the company announced on Wednesday. Most notably, users can now share an image from their smart glasses directly to…

Meta’s Ray-Ban smart glasses now let you share images directly to your Instagram Story

Spotify launched its own font, the company announced on Wednesday. The music streaming service hopes that its new typeface, “Spotify Mix,” will help Spotify distinguish its own unique visual identity. …

Why Spotify is launching its own font, Spotify Mix

In 2008, Marty Kagan, who’d previously worked at Cisco and Akamai, co-founded Cedexis, a (now-Cisco-owned) firm developing observability tech for content delivery networks. Fellow Cisco veteran Hasan Alayli joined Kagan…

Hydrolix seeks to make storing log data faster and cheaper

A dodgy email containing a link that looks “legit” but is actually malicious remains one of the most dangerous, yet successful, tricks in a cybercriminal’s handbook. Now, an AI startup…

Bolster, creator of the CheckPhish phishing tracker, raises $14M led by Microsoft’s M12

If you’ve been looking forward to seeing Boeing’s Starliner capsule carry two astronauts to the International Space Station for the first time, you’ll have to wait a bit longer. The…

Boeing, NASA indefinitely delay crewed Starliner launch

TikTok is the latest tech company to incorporate generative AI into its ads business, as the company announced on Tuesday that it’s launching a new “TikTok Symphony” AI suite for…

TikTok turns to generative AI to boost its ads business

Gone are the days when space and defense were considered fundamentally antithetical to venture investment. Now, the country’s largest venture capital firms are throwing larger portions of their money behind…

Space VC closes $20M Fund II to back frontier tech founders from day zero

These days every company is trying to figure out if their large language models are compliant with whichever rules they deem important, and with legal or regulatory requirements. If you’re…

Patronus AI is off to a magical start as LLM governance tool gains traction

Link-in-bio startup Linktree has crossed 50 million users and is rolling out the beta of its social commerce program.

Linktree surpasses 50M users, rolls out its social commerce program to more creators

For a $5.99 per month, immigrants have a bank account and debit card with fee-free international money transfers and discounted international calling.

Immigrant banking platform Majority secures $20M following 3x revenue growth

When developers have a particular job that AI can solve, it’s not typically as simple as just pointing an LLM at the data. There are other considerations such as cost,…

Unify helps developers find the best LLM for the job

Response time is Aerodome’s immediate value prop for potential clients.

Aerodome is sending drones to the scene of the crime

Granola takes a more collaborative approach to working with AI.

Granola debuts an AI notepad for meetings

DeepL, which builds automated text translation and writing tools, has raised a $300 million round led by Index Ventures.

AI language translation startup DeepL nabs $300M on a $2B valuation to focus on B2B growth

Praktika has secured a $35.5M Series A round to apply AI-powered avatars to language-learning apps.

Praktika raises $35.5M to use AI avatars to make learning languages feel more natural

Humane, the company behind the hyped Ai Pin that launched to less-than-glowing reviews last month, is reportedly on the hunt for a buyer.

Humane, the creator of the $700 Ai Pin, is reportedly seeking a buyer

India’s Oyo, once valued at $10 billion, has withdrawn its IPO application from the market regulator for the second time.

Oyo, once valued at $10 billion, shelves IPO plans for second time

Ore Energy emerged from stealth today with €10 million in seed funding. The company hopes to make grid-scale batteries that are cheaper and longer lasting.

Ore Energy emerges from stealth to build utility-scale batteries that last days, not hours

Paytm, a leading financial services firm in India, said its net loss widened in the fourth quarter as it grappled with a regulatory clampdown.

Paytm warns of job cuts as losses swell after RBI clampdown

Government officials and AI industry executives agreed on Tuesday to apply elementary safety measures in the fast-moving field and establish an international safety research network. Nearly six months after the…

In Seoul summit, heads of states and companies commit to AI safety

Copilot, Microsoft’s brand of generative AI, will soon be far more deeply integrated into the Windows 11 experience.

Microsoft wants to make Windows an AI operating system, launches Copilot+ PCs

Some startups choose to bootstrap from the beginning while others find themselves forced into self funding by a lack of investor interest or a business model that doesn’t fit traditional…

VCs wanted FarmboxRx to become a meal kit, the company bootstrapped instead

Uber and Lyft drivers in Minnesota will see higher pay thanks to a deal between the state and the country’s two largest ride-hailing companies. The upshot: a new law that…

Uber’s and Lyft’s ride-hailing deal with Minnesota comes at a cost

Andreessen Horowitz’s American Dynamism fund has established a new fellowship program aimed at introducing top engineers and technologists to venture investing, a move that could help the firm identify less…

a16z’s American Dynamism team launches program to introduce technical minds to VC