Airbnb Introduces CASTLE Framework to Train AI Natural Language Search
Airbnb researchers developed CASTLE, an AI system that generates synthetic search queries and relevance labels to improve natural language property search.
When a prospective guest sits down to book a short-term rental, search on major online channels has long relied on rigid filter checkboxes: setting arrival dates, selecting guest counts, and checking boxes for amenities like air conditioning or a private swimming pool. However, user search patterns across software platforms are rapidly transitioning toward natural language queries. Travelers increasingly enter open-ended conversational phrases such as “romantic cabin near skiing” or “pet-friendly beach house that sleeps eight” directly into search boxes, expecting platform algorithms to interpret context, style, and functional preferences without requiring manual filter adjustments.
For software engineering teams building natural language search engines, creating models capable of understanding these complex phrasing patterns introduces a significant technical challenge known as the cold-start problem. Training artificial intelligence models to match conversational search queries with relevant listings traditionally requires vast databases of historical user search queries paired with verified booking decisions. When a channel introduces natural language search capabilities or expands into new geographic markets, that historical search log does not yet exist in sufficient volume to train machine learning systems effectively.
To address this training bottleneck, artificial intelligence researchers at Airbnb published details of a synthetic data generation framework called CASTLE. The system uses de-identified guest booking sessions combined with large language models to construct synthetic search queries and match them with appropriate relevance labels automatically. The initiative highlights how major booking channels are working to upgrade search infrastructure, with direct implications for how property listings are indexed, evaluated, and presented to prospective guests.
What happened
According to a technical publication on InfoQ written by Wendy Ran Wei, Airbnb research team members introduced CASTLE to resolve the cold-start challenge inherent in conversational search. Natural language search allows travelers to express listing criteria using everyday conversational sentences rather than restricting their choices to predefined attribute checkboxes or strict numerical range filters.
Developing effective natural language search models requires solving two coupled operational tasks simultaneously: generating synthetic queries that reflect authentic human search syntax, and establishing accurate relevance labels that specify which property listings satisfy each generated query. Previous technical approaches in machine learning, such as frameworks known as InPars and Promptagator, treated query generation and relevance labeling as separate sequential processes. Those earlier models used large language models to generate synthetic queries from listing descriptions, then relied on secondary algorithmic models to evaluate and assign relevance ratings.
According to the Airbnb researchers, separating query generation from relevance evaluation introduces compounding error sources and creates structural mismatches between synthetic queries and real user search behavior. To eliminate that separation, CASTLE derives contrastive listing pairs directly from de-identified user booking sessions. When a guest views several candidate properties during a shopping session and completes a booking for one of them, the booked property serves as a positive listing example, while an unbooked property viewed during the same session serves as a negative control.
A large language model is then prompted to generate a search query that explicitly fits the positive listing while excluding the negative listing. Because the query is engineered specifically to distinguish the booked property from the rejected property, the relevance label is established directly through the pairing process rather than through a secondary judgment step.
The researchers explained that CASTLE combines three distinct query-generation prompt strategies: template-based prompts, few-shot prompts using approximately 500 seed queries collected through targeted user research, and attribute-grounded prompts focused on distinguishing physical and operational characteristics of listings. A built-in variety mechanism encourages variation in query length, phrasing style, and vocabulary structure across generated datasets.
For listings without sufficient past booking traffic—including newly posted properties—CASTLE utilizes an evaluator called a Virtual Judge to assign relevance labels to query-listing combinations. The researchers also developed CASTLE-R, a specialized variant designed for active production environments where real user search traffic accumulates alongside synthetic training data.
How natural language search works on travel channels
To understand why online channels are developing models like CASTLE, property managers must examine how search infrastructure on major booking websites has evolved over past decades. Traditional search engines on major travel websites functioned primarily through relational database queries. When a host filled out listing settings, the system stored binary data flags for features such as hot tubs, high-speed internet, or off-street parking. When a guest searched for listings, the platform matched selected check-box filters against those stored database fields.
Natural language search operates on vector-based semantic matching. Rather than checking for exact string matches in a database, semantic search algorithms convert listing descriptions, property titles, photo tags, and reviews into high-dimensional mathematical representations known as embeddings. When a guest enters a search phrase like “quiet mountain workspace for remote focus,” the search engine measures the semantic distance between the search vector and listing vectors across the channel platform.
However, training semantic search models on short-term rental platforms presents distinct operational hurdles compared to general web search engines. Rental platforms manage inventory that fluctuates constantly based on availability calendars, seasonal pricing rules, guest occupancy limits, and local regulatory restrictions. Furthermore, actual guest search logs contain typos, informal phrasing, local slang, and highly specific multi-attribute requests.
When travel platforms attempt to launch natural language search features without comprehensive training data, early machine learning models often return poor search results. A guest searching for a rustic cabin near hiking trails might receive results for downtown apartments if the model misinterprets architectural descriptions or spatial modifiers. Generating high-quality synthetic search queries allows platform engineers to train natural language search engines across millions of realistic scenarios before introducing the features to live site traffic.
Inside the synthetic data engine
The architectural mechanics of CASTLE illustrate how platform engineering teams simulate guest decision-making at scale. By grounding synthetic query generation in de-identified guest booking sessions, the framework anchors artificial intelligence prompt generation in observed consumer choices rather than unverified assumptions.
In a typical browsing session on a booking platform, a prospective guest might view five or six listing pages within a single destination area. The guest evaluates details such as interior design, bedding arrangements, kitchen amenities, balcony views, host house rules, and price points before confirming a reservation. Within the CASTLE pipeline, the selected listing becomes the positive example, while an alternative listing viewed during that exact session serves as the contrastive negative example.
The framework prompts a large language model to analyze the full attribute profiles of both properties. The model is asked to generate a search query that a guest would naturally type if they were searching for the positive listing rather than the negative listing. For instance, if the booked listing includes a fenced backyard and permits pets, while the unbooked listing prohibits animals, the synthetic query generator generates search terms focused on pet-friendly stays with outdoor spaces.
To ensure comprehensive coverage across diverse search behaviors, CASTLE integrates three distinct prompting methods:
- Template-based prompts: Structured sentence patterns that establish reliable baseline queries across standard property categories.
- Few-shot prompts: Models trained on approximately 500 seed queries gathered from targeted user research studies, teaching the synthetic generator how travelers phrase complex requests.
- Attribute-grounded prompts: Focused prompts that highlight specific distinguishing characteristics, such as specialized kitchen appliances, workspace configurations, or views.
To prevent synthetic datasets from becoming rigid or repetitive, CASTLE incorporates a mathematical variety mechanism. This system introduces variations in word count, grammatical structure, and vocabulary choice, simulating how travelers type into mobile devices or voice search interfaces.
Benchmarks and performance metrics
To evaluate how closely synthetic queries match actual guest behavior, the Airbnb research team measured statistical divergence between generated synthetic datasets and real-user search logs using Kullback–Leibler (KL) divergence. Lower scores on the KL divergence scale indicate that a synthetic data distribution closely mirrors real human search distributions.
In evaluations published in the paper, CASTLE achieved an overall KL divergence score of 1.01 against real-user query distributions. By comparison, Promptagator—a baseline model that generates queries and assigns relevance labels in separate steps—recorded a KL divergence score of 9.33. This comparison reflects a 9.2-fold reduction in statistical divergence, demonstrating that constructing synthetic queries from contrastive listing pairs produces search datasets substantially closer to live user behavior.
At the attribute-type level, CASTLE achieved a KL divergence score of 0.08. For comparison, manually curated seed queries collected through structured user survey studies recorded a KL divergence score of 0.09 on the same scale. The synthetic generation model matched the structural query patterns of manual survey data collected directly from travelers.
The research team also conducted a human evaluation study to verify the accuracy of automated relevance labels. Three independent subject matter experts reviewed 200 sampled query-listing triplets. Agreement between human experts and CASTLE relevance assignments ranged from 91% to 93% across the three evaluators, confirming high consistency in automated relevance tagging.
Peer review and data leakage questions
Following the publication of the research paper on arXiv, the evaluation methodology generated professional discussion among artificial intelligence researchers and software engineers. On LinkedIn, Mark Tsai, an AI engineer specializing in large language model fine-tuning and evaluation methodology, raised technical questions concerning dataset splitting protocols.
Tsai asked whether the Airbnb research team maintained strict separation between seed templates and listing identification numbers across training and evaluation datasets. He noted that if evaluation sets contain listing IDs or prompt templates seen during training, performance metrics might reflect model memorization of familiar properties or syntax patterns rather than true generalized learning of guest search intent.
Responding to the inquiry on LinkedIn, article author Wendy Ran Wei confirmed that the research team evaluated the model against untouched real-user queries and implemented data leakage guardrails when constructing evaluation datasets. Wei noted that while the published paper did not report a controlled comparison between standard random splits and stricter disjoint splits isolating seed templates and listing IDs, the engineering team recognized that split methodology as a potential direction for future research.
Constructing synthetic queries from contrastive listing pairs yields search datasets substantially closer to live user behavior.
The industry discussion highlights ongoing technical considerations as online travel channels implement generative artificial intelligence. For hosts and property managers, verifying that platform search algorithms generalize effectively to new listings and non-standard properties remains a critical benchmark for long-term distribution fairness.
The other view
While synthetic data models like CASTLE offer major technical advantages for platform development, some industry analysts and software engineers suggest caution regarding over-reliance on synthetic training frameworks.
One primary concern relates to potential algorithmic bias within underlying language models. When a synthetic data system uses an LLM to generate queries and act as a Virtual Judge for properties without booking histories, the outputs rely heavily on patterns learned during pre-training. If an architectural style, unique regional amenity, or uncommon lodging format is underrepresented in foundational AI models, the synthetic system may consistently miscalculate relevance scores or generate inaccurate search queries.
Additionally, using historical booking sessions to establish positive and negative listing pairs assumes that past guest booking decisions were primarily driven by textual search parameters. In real-world travel markets, guest choices are heavily influenced by factors such as pricing discounts, calendar availability, review counts, host response speed, and photography quality. When a guest views multiple listings and books one, the selection may stem from a reduced nightly rate or clear cancellation policy rather than specific textual attributes. Attributing the guest decision entirely to listing feature differences during synthetic query generation risks teaching search models patterns that do not fully reflect traveler motivations.
Finally, engineering experts point out that synthetic benchmark metrics do not eliminate the need for ongoing live market evaluation. High statistical performance on metrics like KL divergence demonstrates mathematical alignment with historical search logs, but guest travel preferences evolve rapidly due to shifting economic conditions, seasonal trends, and emerging travel patterns. Algorithmic search models must continue to undergo live traffic validation to prevent synthetic biases from distorting real-world search results.
What this means for property descriptions and amenities
The implementation of natural language search systems alters how short-term rental property owners should approach listing content creation. Traditional search engine strategies that relied on keyword repetition—such as typing words like "vacation home" or "cabin rental" repeatedly throughout a description—are ineffective under modern semantic search algorithms.
Semantic search models evaluate complete descriptive sentences, contextual detail, and specific property features. Listing descriptions should focus on clear, factual statements that specify exact amenities, equipment specs, and location context. For example, describing a property as offering "a dedicated workspace with an adjustable desk, ergonomic chair, 27-inch monitor, and high-speed fiber internet" provides clear semantic markers for language models indexing work-friendly accommodations.
Hosts should also maintain thorough detail within structured amenity checklists inside host dashboards. Although frameworks like CASTLE attempt to infer listing attributes from freeform text and photos, structured amenity fields remain primary reference points. Ensuring that every relevant amenity setting is accurately selected provides synthetic data generators and Virtual Judge components with clear baseline attributes when scoring contrastive listing pairs.
Furthermore, guest reviews carry increased weight under natural language search architectures. Because CASTLE bases synthetic query generation on de-identified booking sessions, real guest feedback serves to validate listing attributes over time. When guest reviews consistently highlight specific stay experiences—such as quiet evening hours, walkable neighborhood access, or complete kitchen cookware—those reviews reinforce the semantic relevance of the listing in natural language search models.
The broader channel landscape
The movement toward natural language search is expanding across the short-term rental industry in the United States. Major online distribution channels, including Vrbo, Booking.com, and Google Vacation Rentals, are actively deploying conversational search interfaces designed to process natural language guest requests. At the same time, direct-booking websites powered by booking engines or content management systems are interacting with third-party conversational AI search engines operated by tech companies.
For independent rental operators running direct-booking websites, understanding natural language indexing offers clear advantages for search engine optimization. As external search engines integrate AI search assistants, travelers will increasingly search for accommodation options using natural language prompts on external search engines. Independent websites structured with clear content headings, precise property details, accurate address references, and structured schema markup will be better positioned to appear in conversational search results alongside major channel listings.
Moreover, the adoption of synthetic evaluation tools like Virtual Judge modules may alter how newly posted listings gain early search visibility. Historically, new listings on distribution channels experienced a prolonged cold-start phase, during which a lack of historical clicks and bookings limited search placement. Systems that evaluate newly posted properties using synthetic relevance scoring based on detailed text and complete feature sets could allow high-quality new listings to gain search exposure more efficiently than under purely historical search algorithms.
What to do with this
As natural language search systems become standard across booking platforms, property managers should take practical steps to align listing content with modern semantic indexing.
- Review listing descriptions across all active distribution channels to ensure freeform text contains detailed, factual descriptions of property features, appliances, layout, and nearby amenities.
- Audit host dashboard amenity checklists on Airbnb, Vrbo, and major platforms to ensure every physical feature and house policy setting is fully and accurately selected.
- Replace repetitive keyword lists or vague promotional copy with clear, descriptive sentences that specify exact room setups, internet speeds, workspace amenities, and kitchen equipment.
- Encourage guests to write detailed reviews by providing exceptional hospitality, as natural language models parse review text to validate property characteristics and search relevance.
- Update direct-booking website pages with clear heading structures and structured property data so third-party conversational search tools can index independent listings effectively.
Because search algorithms, listing features, and platform ranking rules change regularly, property managers should periodically review channel documentation or consult a qualified digital marketing professional or software consultant to ensure listing optimization practices align with current channel guidelines.
The standards desk (June Ashby) went over this story: 28 details tested; 26 matched to the source material, with those sources listed below; 2 kept as general background; nothing cut.
Sources
- Bing News - reported November 16, 2026.
- lodgify.com - Google Vacation Rentals: Owners’ Guide (2026).
Read and analysed by the STR Gazette desk.
STR Gazette is general information for hosts in the United States. It is not legal, tax or financial advice; confirm rules and figures with your city, your state agency or a qualified professional.

What readers say