URL Checker

Check URLs and save headers

Read the input from url_new, run up to 50 jobs in parallel, treat each line as {}, spawn a new bash shell for each line, and write the result to log_file.

xargs -a url_new -P50 -I{} bash -c '
  tmp=$(mktemp /tmp/urlchk.XXXXXX) || exit 1
  status=$(curl --head --silent --show-error --write-out "HTTP/%{http_version} %{http_code}" --output /dev/null -- "$1")
  printf "%s   %s\n" "$1" "$status" > "$tmp"
  curl --head --silent --show-error -- "$1" | tail -n +2 >> "$tmp"
  echo >> "$tmp"
  cat "$tmp" >> /home/r1nge/Downloads/test/log_file
  rm -f "$tmp"
' _ {}

What it does: reads each URL from url_new, runs 50 in parallel, uses a temp file, keeps the response headers, skips the duplicate status line, appends a blank line, then writes everything to the log.

• --output /dev/null sends any response body to /dev/null.
• -- "$1" passes the URL safely as $1.
• tail -n +2 skips the initial status line from the second curl.
• _ {} makes the first argument after the command string become $0, and the URL become $1.

It’s best to use full paths, but ./log_file should work too.

Input file: url_new

https://ftp.halifax.rwth-aachen.de/gentoo/distfiles/gobject-introspection-1.86.0.tar.xz
https://mirror.hs-esslingen.de/Mirrors/gentoo/distfiles/gobject-introspection-1.86.0.tar.xz
https://linux.rub.de/gentoo-mirror/distfiles/gobject-introspection-1.86.0.tar.xz
https://mirror.metanet.ch/gentoo/distfiles/gobject-introspection-1.86.0.tar.xz
https://download.gnome.org/sources/gobject-introspection/1.86/gobject-introspection-1.86.0.tar.xz

Example output: log_file

https://mirror.hs-esslingen.de/Mirrors/gentoo/distfiles/gobject-introspection-1.86.0.tar.xz   HTTP/2 404
content-type: text/html; charset=iso-8859-1
date: Wed, 06 May 2026 12:17:34 GMT
server: Apache/2.4.66 (Fedora Linux)


https://ftp.halifax.rwth-aachen.de/gentoo/distfiles/gobject-introspection-1.86.0.tar.xz   HTTP/2 404
server: nginx/1.26.3
date: Wed, 06 May 2026 12:17:34 GMT
content-type: text/html; charset=iso-8859-1


https://linux.rub.de/gentoo-mirror/distfiles/gobject-introspection-1.86.0.tar.xz   HTTP/1.1 404
Date: Wed, 06 May 2026 12:17:34 GMT
Server: Apache/2.4.6 (Red Hat Enterprise Linux) OpenSSL/1.0.2k-fips
Content-Type: text/html; charset=iso-8859-1


https://download.gnome.org/sources/gobject-introspection/1.86/gobject-introspection-1.86.0.tar.xz   HTTP/2 200
date: Wed, 06 May 2026 12:17:35 GMT
content-type: application/octet-stream
content-length: 1083172
last-modified: Sat, 13 Sep 2025 11:20:41 GMT
etag: "68c55389-108724"
expires: Sun, 24 Jan 2027 11:48:42 GMT
cache-control: max-age=31536000
cache-control: public
x-77-nzt: kkJBx6W11KG0ROl7Vg33WPSCfUJHK2Ac64FD9gfzviX/s4DLjEnvsuY
x-77-nzt-ray: ae9a04090e09b47a5f31fb695ee9b414
x-77-cache: HIT
x-77-age: 4642
server: CDN77-Turbo
x-77-pop: bucharestRO
accept-ranges: bytes


https://mirror.metanet.ch/gentoo/distfiles/gobject-introspection-1.86.0.tar.xz   HTTP/2 404
server: nginx
date: Wed, 06 May 2026 12:17:35 GMT
content-type: text/html; charset=utf-8
vary: Accept-Encoding

Just the URL + HTTP response

xargs -a url_new -P50 -I{} bash -c '
  tmp=$(mktemp /tmp/urlchk.XXXXXX) || exit 1
  status=$(curl --head --silent --show-error --write-out "HTTP/%{http_version} %{http_code}" --output /dev/null -- "$1")
  printf "%s   %s\n" "$1" "$status" > "$tmp"
  cat "$tmp" >> /home/r1nge/Downloads/test/log_file
  rm -f "$tmp"
' _ {}

Basically the same as the first one, but one curl less, since no need for headers.

Example output: log_file

https://mirror.hs-esslingen.de/Mirrors/gentoo/distfiles/gobject-introspection-1.86.0.tar.xz   HTTP/2 404
https://mirror.metanet.ch/gentoo/distfiles/gobject-introspection-1.86.0.tar.xz   HTTP/2 404
https://ftp.halifax.rwth-aachen.de/gentoo/distfiles/gobject-introspection-1.86.0.tar.xz   HTTP/2 404
https://linux.rub.de/gentoo-mirror/distfiles/gobject-introspection-1.86.0.tar.xz   HTTP/1.1 404
https://download.gnome.org/sources/gobject-introspection/1.86/gobject-introspection-1.86.0.tar.xz   HTTP/2 200

Remove non-200 and HTTP responses

xargs -a url_new -P50 -I{} bash -c '
  tmp=$(mktemp /tmp/urlok.XXXXXX) || exit 1
  code=$(curl --head --silent --show-error --write-out "%{http_code}" --output /dev/null -- "$1")
  if [ "$code" = "200" ]; then
    printf "%s\n" "$1" > "$tmp"
    cat "$tmp" >> /home/r1nge/Downloads/test/log_file
  fi
  rm -f "$tmp"
' _ {}

The same as the second one, but checks the status and doesn’t output the HTTP code.

Example output: log_file

https://download.gnome.org/sources/gobject-introspection/1.86/gobject-introspection-1.86.0.tar.xz

Read the first line if it exists

read -r first_line < /home/r1nge/Downloads/test/log_file && [ -n "$first_line" ] && echo "$first_line" || echo "missing/empty"