The problem
Airbnb adopted React Testing Library for all new React component tests in 2020, but thousands of older tests still used Enzyme, a framework designed for earlier versions of React (Airbnb Engineering). Enzyme’s deep access to component internals no longer matched modern testing practice, and the two frameworks were too different to swap one for the other.
Deleting the old tests was not an option: Airbnb’s analysis showed that would leave significant gaps in code coverage. The team needed to rewrite each file while keeping the intent of the original test and its coverage. Done by hand, the company estimated, that would take 1.5 years of engineering time.
What they did
The work started with a hackathon in mid-2023, when an Airbnb team showed that large language models could convert hundreds of Enzyme files to the new framework in a few days. In 2024, Airbnb built a pipeline to do the full migration (Airbnb Engineering).
The pipeline treated each file as a separate job moving through validation steps: refactor the test, make it pass, then pass the linter and the type checker. A file moved to the next step only when the previous check passed, and hundreds of files ran in parallel. When a check failed, the model got the errors and the latest version of the file and tried again.
For harder files, the team packed far more context into each prompt: the component’s source code, related tests from the same directory, examples of good tests and general migration guidance. By the end, prompts ran to 40,000–100,000 tokens and pulled in as many as 50 related files. The post says the main driver of success was choosing the right related files rather than perfecting the prompt wording. The post refers to “frontier models” without naming a vendor.
Results
According to Airbnb, the first bulk run migrated 75% of the target files in four hours (Airbnb Engineering). The team then spent four days on a loop it called “sample, tune, sweep”: pick 5 to 10 failing files with a common problem, fix the prompts or scripts, test the fix on the sample, and re-run all remaining files. That raised automated completion to 97% of the files.
Engineers fixed the last 3%, just under 100 files, by hand in about a week, starting from the failing automated drafts. The whole migration of nearly 3,500 test files took 6 weeks, and Airbnb says it kept the original test intent and its overall code coverage. InfoQ’s write-up of the post repeats the same figures (InfoQ).
These results are self-reported in Airbnb’s engineering blog. The company did not publish the cost, but said the total, including model usage and six weeks of engineering time, was far more efficient than the manual estimate.
Airbnb has since reported broader use of AI in software engineering. In its first-quarter 2026 shareholder letter, the company said nearly 60% of the code its engineers produce is now coauthored with AI, “roughly twice the industry average, by our estimate” (Airbnb, Inc.).
What didn’t go to plan
Prompt engineering, the obvious first lever, underdelivered. Airbnb says early experiments with prompt strategies did less than simply retrying each step with the error messages fed back, an approach the post calls “brute force” (Airbnb Engineering).
The long tail was stubborn. After the first run, nearly 900 files still failed validation. Even after tuning, the hardest files had been retried 50 to 100 times, and the team concluded it had hit “a ceiling of what we could fix via automation”. It stopped tuning and finished the last files by hand.
Large prompts and high retry counts carry a cost that Airbnb has not disclosed, so other companies can’t compare the model spend with their own estimates. The post also doesn’t publish coverage figures before and after the migration, only the statement that coverage was maintained.
Lessons
- Break the job into checkable steps. Per-file steps gated by tests, the linter and the type checker let Airbnb run hundreds of files at once and see exactly where each one stuck.
- Let validation drive retries. Feeding errors back to the model worked better than polishing prompts.
- Invest in context, not wording. Choosing the right related files mattered more than prompt phrasing, by Airbnb’s account.
- Tune on samples, then sweep. Fixing 5 to 10 representative failures at a time took automated completion from 75% to 97% in four days.
- Know when to stop automating. The last 3% went faster by hand, starting from the model’s drafts.
Adapt it in your organization
Is this for you?
- You have a large, repetitive code change with a clear target, such as a framework, library or language upgrade.
- An automated check can tell whether each file is done: tests, linters, type checkers or compilers.
- The work splits into independent files or modules that can run in parallel.
- You can estimate the manual effort, so the comparison is credible.
- Engineers are available to own the pipeline and finish the long tail.
A 90-day version
Pilot: pick one backlogged migration, build a per-file pipeline with automated validation and retry loops, and run it on a representative slice of a few hundred files. Owner: engineering director for the platform team. Success measure: at least 70% of the slice migrated automatically and passing all checks, with total effort below one-quarter of the manual estimate for that slice.
Metric to track
Share of files passing every automated check without human edits, alongside engineer hours per migrated file. Airbnb reached 75% on its first bulk run and 97% after four days of tuning; a first pilot should plan for less.
Failure modes to pre-empt
- Spending weeks on prompt wording before building the retry and validation loop.
- Migrating code whose tests are too weak to tell whether the change is correct.
- Chasing the last few percent with automation when engineers would finish faster.
- Not tracking model spend, so the business case can’t be checked afterwards.
Questions to take to your leadership team
- Which migrations or upgrades have we deferred because the manual estimate was too large?
- Do we have automated checks good enough to trust a model’s changes at scale?
Executive FAQ
How long did Airbnb’s AI code migration take?
Six weeks for nearly 3,500 test files, against an estimated 1.5 years by hand, according to Airbnb Engineering.
Did AI do the whole migration?
No. Automation migrated 97% of the files; engineers finished the rest by hand in about a week.
About the curator & AI disclosure
Curated by Luciano Ayres
Luciano Ayres is Director of Artificial Intelligence at ArcTouch, with more than 20 years of experience in software and technology leadership and specialized studies in deep learning. Connect with him on LinkedIn.
AI-generated briefing. This case was produced through an automated AI research and writing workflow curated by Luciano Ayres. Sources are linked for independent verification; summaries and analysis may contain errors and do not replace professional due diligence.