Remove duplicate LinkedIn search results with an Airtable URL lookup
People keep asking how to stop the same profile landing twice when they extract a LinkedIn search —
Sales Navigator or regular search. This sits on the
Growth-Hacking.io scrape flow from the previous video. I add one
Airtable lookup before anything new is written.
Search Airtable for the LinkedIn URL
Add an Airtable Search Records module in Integromat — now
Make.com. Point it at the same base that already holds the leads. The formula key is LinkedIn URL. That field is unique. Spell the field name exactly as it is in the base, including capitals. A mismatch throws an error.
The formula is LinkedIn URL equals the incoming profile URL.
Filter: ID does not exist
Run it on a profile that is already in the base. Airtable returns a record. If ID exists, they are already there. Do not continue. Put a filter on the path: ID does not exist. Only then create the row.
Test a processed search, then a new one
The extractor was set to pull about 2,000 contacts. I dropped it to ten so the test was fast. Same search, already processed. None of those ten wrote a new row.
Then I changed the search to profiles that were not in the base. All of them were added. That is the whole filter: look up the LinkedIn URL, skip if an ID comes back, write if it does not.
Schedule the scenario daily, or however often you re-run the scrape.
OFM Insider and
Automation Academy if you want the communities from this video. Tools and resources are
Bento. A one-off session is
Calendly. Four sessions is
4x Consultations.
Transcript
Hey guys so I get quite a lot of questions on people asking how do we stop duplicate search results coming in when you're extracting results from like LinkedIn sales navigator or just regular LinkedIn so I'm going to show you how we do that so this is the flow that we built in a previous video I'll link in the description so if you need to catch up to this point you can watch that video and then come back to this one so essentially what we would do is add in an air table module and we would do a search records and then you want to make sure that we're connected to the same base that you had the records stored in previously that we're checking for duplicates in right
and then under the formula all we're going to do is use the LinkedIn URL because it's always going to be unique so make sure this is spelled out exactly the same as how you have it in your database in terms of capitals and everything like that otherwise it will throw in an error so we'll do LinkedIn URL equals to and then we're going to take the we're going and if we just run this on one example that we already have in the database so it's going to return a duplicate and what we want to do is say that we're going to filter it out so if that if it returns a result for us if ID exists it means that they already exist so we want to make sure that we don't continue with it so we want to do does not exist and then it will continue down this path
so I'm just going to check this is currently set to extract 2,000 contacts so I'm just going to change this to just do like ten so it's going to be much faster Save Changes and then we'll just get it to run once so if this has been set up correctly none of these records if it finds will be added into this database because we've already processed all records in that search so it's making sure that no duplicates come in cool there we go so that has stopped the records from going through so then that what we'll do is we'll actually change up the search a little bit just to make sure that the logic is working so these records haven't been added into this database before so all of them should
cool so because they don't exist in the database they're all being added so that is just a simple filtering system search for the record based on the LinkedIn URL filter it out if it returns an ID it means it exists already so then we we don't want to continue with it and that way if it doesn't exist and it's going to be added so that is how to make sure that you don't keep adding duplicates into your database as always thank you very much for watching like you can schedule this to run as frequently if you want like everyday however often you want to do any questions yeah feel free to leave a comment and I'll get back to you