Skip to content

Tag: shellscript

bash and wget to download a lot of files

Published by Silveira on 2025-04-21

Problem:

  • Download 500 large files.
  • Files follow the pattern data_1.zip, data_2.zip, … data_500.zip.
  • All the files are in “https://examples.com/downloads”.
  • Files have from kilobytes to gigabytes in size.

Solution:

  • A naive Bash script to download the files using wget.
  • It is set to use wget with -c (continue). This will skip files already downloaded and continue files that have been interrupted. This only works with FTP/HTTP servers that support the “Range” header.
#!/bin/bash
set -euxo pipefail

PREFIX="https://examples.com/downloads/data_"

for i in $(seq 1 500); do
  wget -c "$PREFIX"_"$i".zip
done

maybe

Published by Silveira on 2023-11-07 
while true; do if [[ $(($RANDOM % 2 )) = "1" ]] then; echo "y"; else echo "n"; fi done

Mac: changing screenshots location

Published by Silveira on 2023-06-22 
mkdir ~/Desktop/screenshots

defaults write com.apple.screencapture location ~/Desktop/screenshots

killall SystemUIServer
  1. Creates a directory in /Users/username/Desktop/screenshots.
  2. Sets the default screenshot location to that directory.
  3. Restarts the service responsible for screenshots.

New screenshots (?+Shift+3, 4 or 5) are now stored in ~/Desktop/screenshots.

Simple backup using S3 Sync

Published by Silveira on 2015-08-26

Given that you have the AWS CLI installed and configured, here are a few examples of syncing directories with S3.

Sync your home directory:
[bash]
aws s3 sync $HOME s3://homeof$USER
[/bash]

E.g. $USER is sil and $HOME is /home/sil, then this will copy all files from /home/sil to bucket homeofsil.

To download the files back to another directory:
[bash]
mkdir $HOME/oldhome
aws s3 sync s3://homeof$USER $HOME/oldhome
[/bash]

More details about S3 Sync in the official documentation.

ImageMagick, four point perspective distortion in a video

Published by Silveira on 2014-12-07

Step 1) Download the video from Youtube using JDownloader. The best one was the original Japanese upload, The Game Awards 2014 出展映像 ゼルダの伝説 最新作, which is 720p. I saved this file as zelda_720p.mp4.

Step 2) Transform the video in a series of PNGs.

[bash]mplayer -nosound -vo png:z=9 zelda_720p.mp4[/bash]

This generated image files ranging from 00000001.png to 00007563.png.

Step 3)
Using Gimp, find the 2 sets of 4 points for the perspective. The first in red is the source and the one in blue is the destination.

zelda video four point perspective

Step 4) Apply ImageMagick’s four point perspective distortion in each image using a ShellScript.

[bash]
#!/bin/sh
for image in *.png;
do
convert $image -matte -virtual-pixel transparent \
-distort Perspective \
‘60,90 0,0 50,415 0,720 582,418 1280,720 589,147 1280,0’ \
p_$image
done
[/bash]

Now I have images in the format p_00000001.png to p_00007563.png. Because PNG is a lossless format, the perspective lost less information in this step than it would if step 2 was outputting jpgs.

Step 5) Convert frames to jpgs using ImageMagick and ShellScript.

[bash]
#!/bin/sh
for image in p_*.png;
do
convert $image "${image/.png/}.jpg"
done
[/bash]

Where “${image/.png/}.jpg” removes the .png in the image string.

Ps: this step is not really necessary as you could use png as input to ffmpeg.

Step 6) Convert jpgs in a video.

[bash]
ffmpeg -start_number 0 -i ‘%08d.jpg’ -c:v libx264 output.mp4
[/bash]

‘%08d.jpg’ means a 8 digits filled with zeros in the left followed by .jpg, in this case 00000001.jpg to 00007563.jpg.

With this, I have the output.mp4 video ready to upload.

The whole process took several hours and a total 13GiB, although the final video has only 96 MiB. This could be optimized using pipelines and parallelism if it was needed to repeat the process.

HackerNews discussion: https://news.ycombinator.com/item?id=8713070

Bash Brace Expansion

Published by Silveira on 2014-07-31

photo by whiskeyandtears at https://www.flickr.com/photos/whiskeyandtears/2140154564

[bash]$ echo {0..9}
0 1 2 3 4 5 6 7 8 9[/bash]

[bash]$ echo b{a,e,i,o,u}
ba be bi bo bu
[/bash]

[bash]$ echo x{0..9}y
x0y x1y x2y x3y x4y x5y x6y x7y x8y x9y[/bash]

[bash]$ echo {a..z}
a b c d e f g h i j k l m n o p q r s t u v w x y z[/bash]

[bash]$ echo {1..3} {A..C}
1 2 3 A B C[/bash]

[bash]$ echo {1..3}{A..C}
1A 1B 1C 2A 2B 2C 3A 3B 3C[/bash]

[bash]echo {a,b{1,2,3},c}
a b1 b2 b3 c[/bash]

[bash]$ mkdir -p {project1,project2}/{src,tst,bin,lib}/
$ find .
.
./project1
./project1/tst
./project1/bin
./project1/lib
./project1/src
./project2
./project2/tst
./project2/bin
./project2/lib
./project2/src
[/bash]

[bash]$ echo {{A..Z},{a..z}}
A B C D E F G H I J K L M N O P Q R S T U V W X Y Z a b c d e f g h i j k l m n o p q r s t u v w x y z[/bash]

[bash]$ for i in {a..f} 1 2 {3..5} ; do echo $i;done
a
b
c
d
e
f
1
2
3
4
5[/bash]

The examples below requires Bash version 4.0 or greater.

[bash]$echo {001..9}
001 002 003 004 005 006 007 008 009
[/bash]

[bash]$ echo {1..10..2}
1 3 5 7 9
[/bash]

find a process running a deleted library

Published by Silveira on 2014-06-09

sudo find /proc -maxdepth 2 -name maps -exec grep -HE ‘\(deleted\)’ {} \; | cut -d/ -f3 | sort -u | xargs –no-run-if-empty ps

Looking for libssl in specific:

sudo find /proc -maxdepth 2 -name maps -exec grep -HE ‘/libssl\.so.* \(deleted\)’ {} \; | cut -d/ -f3 | sort -u | xargs –no-run-if-empty ps

Killing all process using a deleted version of libssl:

sudo find /proc -maxdepth 2 -name maps -exec grep -HE ‘/libssl\.so.* \(deleted\)’ {} \;| cut -d/ -f3 | sort -u | xargs –no-run-if-empty sudo kill