The problem you observe:
The specific case I have the issue with at the moment is content that was clicked on (therefore, not filtered, and seen by the customer) but not contained in our TR data.
One of the symptoms is that you can’t match 15-20% of TI redirects to a relevant TR message by a PQID value.
Why is there no match for those redirects?
The redirects have PQIDs returned from the updated (re-scraped) search results. These can’t be found in Travel Rankings search results messages.
Why is there no message in TR for the updated search results, which contain new PQIDs?
It’s not meant to be included in TR results by design as no new search session is created when all partner quotes are scraped from their Meta APIs. In instances when there’s a PQID mismatch, we can assert that a quote re-scrape happened.
Why is there a quote re-scrape and what is it?
Quote re-scrape is a scenario when a single partner’s API is called to get quotes for specific search criteria. This is not to be confused with a new search (session) when all partners’ API are requested.
Quote re-scrape is triggered in 2 scenarios:
- There’s a considerable delay between the original search and a user’s attempt to click (in the vast majority of cases)
- There’s an issue getting a quote from our quote cache
Here’s the flow example
- The User makes a Search 1 (all partners API requested, correlationID1 created, PQID1 returned from a Partner X, TR message sent containing all partner quotes)
- The User then stays idle (e.g. 30 min)
- Finally the User attempts to make a click on Partner X
- Quote cache expired? If no, proceed to the website, otherwise, make a re-scrape
- Skyscanner carries out a re-scrape by calling Partner X Meta API ONLY (i.e. no new session created, no other partners quotes retrieved)
- Skyscanner gets new quotes from Partner X (PQID2). TR message not sent
- Skyscanner logs a redirect (correlationID1 from Search 1, PQID2 from re-scraped search results)
So this explains why you observe calls to your Meta API and why you generate new PQIDs (PQID2 in the example above), however, no extra TR message is sent. This also explains why new PQID2 is assigned to the redirect.
What should a TR partner do in this case?
While PQID is one of the ways to match TI redirects to TR searches, you can also use another field called CorrelationID (and if required ItineraryID). The match rate between TI redirects > TR searches by correlationID is 99%.
So even if you decide to keep on using PQIDs to match data sets, you can use the following logic
- If PQIDs match between redirects and searches, then keep on using them
- Otherwise use correlationID to match TI redirects and TR searches
What about competition for the re-scraped search which you can’t get without an extra message in TR?
By definition, there won’t be any competitor quotes in re-scraped results as only Partner X Meta API is called (only your API is called in your case).
What if the price is updated during the re-scrape? How will I know it?
As there’s no extra TR message for the re-scrape search, the only way to identify the updated price is to take it from TI redirect value.
What are the plans to include a re-scrape search in TR?
At the moment, it’s not considered due to technical design limitations.