Problem:
- Download 500 large files.
- Files follow the pattern data_1.zip, data_2.zip, … data_500.zip.
- All the files are in “https://examples.com/downloads”.
- Files have from kilobytes to gigabytes in size.
Solution:
- A naive Bash script to download the files using wget.
- It is set to use wget with -c (continue). This will skip files already downloaded and continue files that have been interrupted. This only works with FTP/HTTP servers that support the “Range” header.
#!/bin/bash
set -euxo pipefail
PREFIX="https://examples.com/downloads/data_"
for i in $(seq 1 500); do
wget -c "$PREFIX"_"$i".zip
done