How to scrape an infinite scroll page into Excel or CSV
Social feeds, marketplaces, job boards and many store categories have no Next button. More items appear as you scroll down. That makes them hard to get into a spreadsheet.
Why the usual tools miss most of the list
- Copy and paste only gets what is on the page right now. Some sites also remove items from the top while you scroll, so even scrolling to the end first does not help.
- Google Sheets IMPORTHTML and Excel Data, From Web download the page address once. They see the first batch of items, or none at all when the list is built with JavaScript.
- Changing
?page=2in the address only works when the site has numbered pages behind the scenes. Infinite scroll pages often do not.
Manual workarounds
- If you are comfortable with developer tools, open the Network tab, scroll, and look for the request that returns the next batch as JSON. Some sites expose clean data that way. It takes time and breaks when the site changes.
- For a short list, scroll to the bottom, select everything and paste into a sheet, then clean up the columns by hand.
Collect the whole list with a browser add-on
List & Table Scraper is a Firefox add-on we make. It is free for up to 3 scroll rounds per run, and it reads the page as you see it, so it works on card layouts as well as tables.
- Open the page and click the add-on's toolbar button. It outlines the block of repeated items and shows a preview. If it picked the wrong block, click Try another.
- Set Delay (1 to 10 seconds, default 2). Use a longer delay on slow sites so each new batch has time to load.
- Set Max pages. For infinite scroll it limits the number of scrolls. The free version scrolls up to 3 times per run. With Pro the limit goes up to 500.
- Click Infinite scroll. The add-on scrolls to the bottom, waits, and adds the new rows to the preview. It stops when two scrolls in a row bring no new rows, when it reaches your limit, or when you click Stop.
- Rename or remove columns, then download CSV, Excel (.xlsx) or JSON, or copy and paste into Google Sheets.
Rows are added as they load and duplicates are dropped, so items the site later removes from the page are still kept in your export.
Good to know
- It only collects what your browser loads. If a site asks you to log in or stops loading after a few hundred items, the export stops there too.
- Lists with a Next link or button are better handled by Locate Next button and Crawl pages, see scraping a list from multiple pages.
- Free reads up to 3 scroll rounds per run, which suits short lists. Pro (from USD 4 a month) reads up to 500. CSV, Excel and JSON export are free either way.
- Scraped data stays in your browser. The free version needs no account, and the add-on has no server. Please respect each website's terms of use and keep the delay reasonable.