Hey ,
I'm thrilled to help you learn JavaScript. Unfortunately, you've landed on a page where you cannot access with your current purchase.
Please upgrade (use this link) access this content.
I'm super eager to help you learn more!
When you request for a list of items, many APIs respond with an incomplete list. For example, GitHub sends you 30 repositories even if the user has more than 30 repositories.
If you want to fetch all repositories, you can do two things:
APIs may let you change the number of items you get per request. For example, GitHub lets you get up to 100 items. To get 100 items, you change the per_page query parameter.
If the user has more than 100 repositories, GitHub will send you the first 100 repositories. If the user has less than 100 repositories, GitHub will send you every repository.
For this lesson, we’re going to fetch Sindre Sorhus’s repositories. Sindre creates a lot of open source projects and has over 900 repositories on GitHub.
fetch('https://api.github.com/users/sindresorhus/repos?per_page=100')
.then(r => r.json())
.then(repos => console.log(repos))
APIs will tell you how to make more requests. This is usually found in the Pagination section of the API. Make sure to read it.
How you make more requests depends on the first response.
If your first response tells you the total number of pages, you can fetch the rest of the information at once. If the response doesn’t tell you the total number of pages, you need to fetch page by page.
You’ll learn how to fetch many pages at once in this lesson, and how to fetch page by page in the next lesson.
Many APIs use a Link header to tell you the number of pages. GitHub is one example.
When you get a paginated response from GitHub, you’ll see the Link header in your dev tools.
fetch('https://api.github.com/users/sindresorhus/repos?per_page=100')
Your first step is to get this link header in JavaScript. We have already gone through how.
Here’s how you do it with XHR.
// Getting link header with XHR
request.addEventListener('load', e => {
const linkHeader = request.getRequestHeader('link')
})
Here’s how you do it with Fetch.
fetch('some-url')
.then(r => {
const headers = Array.from(r.headers.entries())
.reduce((accumulator, [header, value]) => {
return Object.assign(accumulator, {
[header]: value
})
}, {})
return r.json()
.then(body => ({ headers, body }))
})
.then(response => {
const linkHeader = response.headers.link
})
And here’s how you do it with zlFetch.
zlFetch('some-url')
.then(response => {
console.log(response.headers) // the headers
console.log(response.body) // the body
})
I’m going to show you how to use Fetch for the rest of the lesson. You can try this out with zlFetch later.
If you examine the link header carefully, you’ll see a pattern. There are two parts to this link header:
These two links are separated by a comma.
You can also see that links to the next page (and last page) share another pattern:
rel tells us which page the link leads to
The two patterns let us convert the link header into a JavaScript object that looks like this:
const parsedLinkHeader = {
next: 'a link',
last: 'another link'
}
To create this JavaScript object, we need to split the header into two parts. We can do this with the split method.
split lets you split a string into two or more parts. These parts will be stored as an array.
const array = string.split(separator, limit)
separator is the character(s) you want to use to split the string. The character(s) used will be removed from all parts.
limit lets you control the number of splits to make. You won’t use this much.
Here’s an example:
const string = 'My name is Zell'
const array = string.split(' ')
console.log(array)
// ['My', 'name', 'is', 'Zell']
We want to split the link header into two parts:
We can do this with split.
const parts = linkHeader.split(',')
console.log(parts)
// [
// <https://api.github.com/user/170270/repos?per_page=100&page=2>; rel="next",
// <https://api.github.com/user/170270/repos?per_page=100&page=10>; rel="last"
// ]
Next, we want to split each part with ; to get the link and rel for each part.
Here’s how splitting the first part would look like:
const firstPart = parts[0]
const [link, rel] = firstPart.split(';')
console.log(link) // '<https://api.github.com/user/170270/repos?per_page=100&page=2>'
console.log(rel) // ' rel="next"'
Next, we want to format link and rel so we get an object that uses rel as a property and link as the value, like this:
const parsedLinkHeader = {
next: 'a link',
last: 'another link'
}
We can do this with the replace method.
replace lets you replace parts of a string with other characters.
const replaced = string.replace(partToBeReplaced, replacer)
partToBeReplaced is the character(s) you want to replace.
replacer is the characters you want to replace with.
Here’s an example:
const string = 'My name is Zell'
const replaced = string.replace('Zell', 'Vincy')
console.log(replaced) // My name is Vincy
We’ll use an empty string as the replacer to remove the angled brackets. After we remove the angled brackets, we also want to use trim to remove any leading and trailing spaces.
const removedLeftBracket = link.replace('<', '')
const removedRightBracket = removedLeftBracket.replace('>', '')
const trimmed = removedRightBracket.trim()
These three steps can be combined into one.
const formattedLink = link.replace('<', '').replace('>', '').trim()
console.log(formattedLink) // 'https://api.github.com/user/170270/repos?per_page=100&page=2'
We want to format rel as well.
const formattedRel = rel.replace('rel="', '').replace('"', '').trim()
console.log(formattedRel) // 'next'
Once we’ve formatted link and rel, we can create an object that gives us information about the next link.
const parsedFirstPart = {
[formattedRel]: formattedLink
}
console.log(parsedFirstPart.next)
// { next: 'https://api.github.com/user/170270/repos?per_page=100&page=2' }
The process to parse and format the second part is the same as the process to parse and format the first part. I’ve created a function for you to speed things up.
const parseLinkHeader = linkHeader => {
const parts = linkHeader.split(',')
return parts.reduce((acc, part) => {
const [l, r] = part.split(';')
const link = l.replace('<', '').replace('>', '').trim()
const rel = r.replace('rel="', '').replace('"', '').trim()
return Object.assign({}, acc, { [rel]: link })
}, {})
}
// Usage
const parsedLink = parseLinkHeader(linkHeader)
Your parsed link header should look like this:
const parsedLinkHeader = {
'last': 'some-url?per_page=100&page=10',
'next': 'some-url?per_page=100&page=2'
}
We need to parse some-url?per_page&page=10 to get the final page. We can do this with URLSearchParams.
URLSearchParams help us extract parameters from query strings. To use it, you provide the query string you want to parse to new URLSearchParams.
const searchParams = new URLSearchParams('per_page=100&page=5')
You can search for the parameter you want with the get method.
const page = searchParams.get('page')
console.log(page) // 5
We need to pass the query string into URLSearchParams. To get the query string, we can split the link at ?.
const queryString = parsedLinkHeader.last.split('?')[1]
We can get the last page with the following code:
const searchParams = new URLSearchParams(queryString)
const lastPage = searchParams.get('page')
console.log(lastPage) // 10
We need to use Promise.all to fetch many requests at once. Since we know the last page is ten, we need to construct a Promise.all that looks like this:
Promise.all([
fetch('Page 2'),
fetch('Page 3'),
fetch('Page 4'),
fetch('Page 5'),
fetch('Page 6'),
fetch('Page 7'),
fetch('Page 8'),
fetch('Page 9'),
fetch('Page 10')
])
We can use a for loop to create this array or promises. This for loop can start with index 2 (for page 2) and end at index 10 (for page 10).
const link = 'https://api.github.com/users/sindresorhus/repos?per_page=100'
let array = []
for (let index = 2; index <= lastPage; index++) {
const request = fetch(`${link}&page=${index}`)
.then(r => r.json())
array = array.concat(request)
}
Promise.all(array)
.then(results => { /* Do something with results*/ })
Note: you’ll hit your rate limit quickly when you request for many pages at once. You might want to authenticate yourself so you have a larger limit.
Promise.all returns an array with results from each promise.
Promise.all(array)
.then(results => console.log(results))
We don’t want an array of arrays. We don’t care about the pages. We want one array that contains all repositories.
// What we want
const repos = [
repo1,
repo2,
repo3,
...,
lastRepo
]
We can use reduce to flatten the array of arrays into a single array. We covered how to do this in the lesson on reduce.
Promise.all(requests)
.then(responses => {
const flattened = responses.reduce(
(acc, repos) => [...acc, ...repos]
, [])
})
This gives us an array of repositories from Page 2 to the last page. We can include repositories from the first page if we set the first page as the initial value.
const page1 = response.body
Promise.all(requests)
.then(responses => {
const flattened = responses.reduce(
(acc, repos) => [...acc, ...repos], page1
)
})
You can return the value from Promise.all to work and work on the repos in the next then call.
.then(response => {
const page1 = response.body
return Promise.all(requests)
.then(responses => {
return responses.reduce(
(acc, repos) => [...acc, ...repos], page1
)
})
})
.then(repos => {
console.log(repos)
})
Here’s the code to execute everything we talked about.
const link = 'https://api.github.com/users/sindresorhus/repos?per_page=100'
fetch(link)
// Pass headers and body into the next then call
.then(r => {
const headers = Array.from(r.headers.entries())
.reduce((accumulator, [header, value]) => {
return Object.assign(accumulator, {
[header]: value
})
}, {})
return r.json()
.then(body => ({ headers, body }))
})
.then(res => {
// Parse link headers
const parsed = parseLinkHeader(res.headers.link)
// Gets last page
const queryString = parsed.last.split('?')[1]
const searchParams = new URLSearchParams(queryString)
const lastPage = searchParams.get('page')
// Make array of promises
let promises = []
for (let index = 2; index <= lastPage; index++) {
const request = fetch(`${link}&page=${index}`)
.then(r => r.json())
promises = promises.concat(request)
}
// Fetches all pages and flattens the results
return Promise.all(promises)
.then(results => {
return results.reduce(
(acc, result) => [...acc, ...result], res.body
)
})
})
.then(repos => {
console.log(repos)
})
We can make the array of promises with Array.from. To do so, we pass an object with a length property. Array.from will treat this object as an array-like object. It will create array items and fill them with undefined.
const promises = Array.from({length: lastPage - 1})
console.log(promises.length) // 9
console.log(promises) // [undefined, undefined, undefined, undefined, undefined, undefined, undefined, undefined, undefined,]
You can use map to create the array of fetch requests. Use the index value to fetch the correct page.
const promises = Array.from({length: lastPage - 1})
.map((value, index) => {
return fetch(`https://api.github.com/users/sindresorhus/repos?per_page=100&page=${index + 2}`)
.then(r => r.json())
})