Build download links
Choose data criteria and generate a link to fetch a zip file of data. Open it and your browser downloads a zip file. This is the quickest route for most people, and everything else on this page is a variation on it.
- Type the three-letter code of an observatory, for example
eskfor Eskdalemuir,lerfor Lerwick orhadfor Hartland. Separate several codes with commas. - Choose the dates. The “up to” date is the boundary of the period, so use 1 January 2021 to get all of 2020.
- Open the link, then unzip the file. It contains your data files and a short summary of your request.
Other ways to get data
If a downloaded zip isn't what you need, other services are also available:
Browse and plot first The Observatory Data Portal lets you look at the data before you download anything. Automate it with curl or Python Repeat downloads for many dates or observatories, or feed the results into your own analysis. Read data straight into your code The text endpoints return CSV or JSON directly, with no zip to unpack. Connect HAPI-aware software Use our HAPI server if your tools already speak the Heliophysics API standard. Try requests in the interactive docs The full reference for every endpoint and option, where you can fill in a form and run a request.Query parameters
esk. Lower case is safest in links.second, minute, hour, day, month or year, plus k for the three-hourly local K-index.2020-01-01, optionally with a time (2020-01-01T00:00:00). All times are UTC. start_dt and end_dt mark the boundaries of the period, so 2020-01-01 to 2021-01-01 requests the year 2020.XYZF gives north (X), east (Y), vertical (Z) and total field (F). HDZF gives horizontal intensity (H), declination (D), vertical (Z) and total field (F).
original returns the data as they are stored. Yearly data use XYZ and HDZ.
Zipped downloads: full details
The link generator above writes links for zipped downloads, with the template for any download shown below. One request returns one zip file, and you can ask for several observatories at once. Zipped downloads are also the only route that offers the WDC format (minute and hourly means) and, for most cadences, IAGA-2002.
https://wdcapi.bgs.ac.uk/data/zipped-data/{cadence}/{observatories}?start_dt={start}&end_dt={end}&file_format={format}Example - three observatories in WDC format
Hourly means for Eskdalemuir, Lerwick and Hartland for 2020:
https://wdcapi.bgs.ac.uk/data/zipped-data/hour/esk,ler,had?start_dt=2020-01-01&end_dt=2021-01-01&file_format=wdc&orientation=originalYour browser may show the commas as %2C. They mean the same thing.
Particular months instead of dates
Minute, hourly and daily data can also be selected with years and months. Use one method or the other in a request, not both.
https://wdcapi.bgs.ac.uk/data/zipped-data/hour/esk?years=2020&months=1,2,3&file_format=csv
This requests January, February and March 2020. Leave months out to get whole years.
For several years, separate them with dashes and give one group of months per year,
for example years=2020-2021&months=1,2,3-1,10,11,12. Monthly and yearly cadences accept years only.
Which formats does each cadence offer?
Show all parameters
cadencesecond, minute, hour, day, month, year or k.observatoriesmonth and year you can enter ALL for every observatory.start_dt, end_dtyears, monthsfile_formatorientationoriginal (default), XYZF or HDZF; XYZ or HDZ for yearly data. Not offered for K-index or one-second data.group_byday (default, as in IAGA-2002) or month for one file per month.excludeALL.Large requests
The size of a single download is limited. If a request is too large the server replies with 413 Request Entity Too Large. The fix is to split the period into smaller pieces and download them one after another. For ten years of minute data from multiple observatories, one year at a time works well.
https://wdcapi.bgs.ac.uk/data/zipped-data/minute/esk,ler,had?start_dt=2010-01-01&end_dt=2011-01-01&file_format=wdc
https://wdcapi.bgs.ac.uk/data/zipped-data/minute/esk,ler,had?start_dt=2011-01-01&end_dt=2012-01-01&file_format=wdc
https://wdcapi.bgs.ac.uk/data/zipped-data/minute/esk,ler,had?start_dt=2012-01-01&end_dt=2013-01-01&file_format=wdcKeep a note of what you requested and check that each request finished. A successful reply doesn't guarantee that every date holds observations, because coverage varies between observatories and periods. The API documentation has more details on limits; for example, minute means are limited to 10 total years of data per download. For many observatories or long periods, the bulk download script does the splitting for you.
Bulk downloads
Need data from all available observatories for a year or period? Visit the data portal to explore the available data. If you want programmatic access instead, then these scripts check which observatories actually have data, splits your request into pieces that fit the size limits, and downloads them one after another. Each zip is named after its time slice.
Python, curl (bash) and wget (bash) scripts are available. Scripts adapted from code produced by Claude AI.- It asks the availability endpoint once for your whole period.
- Builds time slices that use only the observatories with data, sized to fit the limit.
- Downloads each slice as a zip. Files you already have are skipped.
Largest request per download (observatories x years)
Monthly and yearly means don't need this: use ALL in a single zipped download.
Choosing a file format
The scripts pick a default for you: WDC for minute and hourly means, IAGA-2002 for one-second data and CSV for everything else.
To get a different format, set file_format in Python or use -f in the bash scripts. The options are csv, json, iaga, wdc, xml and cov-json.
Not every cadence offers every format, so check the list of formats for each cadence.
Python 3.7 or later, with no installs. Use it as a .py file or paste it into a Jupyter cell.
Preview script
"""
Bulk download from the WDC Geomagnetism API (https://wdcapi.bgs.ac.uk).
Works out which observatories have data, splits the request into pieces that
fit the size limits, and saves each piece as a zip file in outdir.
Be polite: keep the pause between requests and run one copy at a time.
Python 3.7+, standard library only. Works as a .py file or in a Jupyter cell.
"""
import json, time, urllib.request, urllib.parse, urllib.error
from collections import defaultdict
from datetime import date
from pathlib import Path
BASE = "https://wdcapi.bgs.ac.uk/data"
# Largest request allowed: observatories x years (one-second data: x months)
LIMITS = {"second": 4 / 12, "minute": 10, "hour": 50, "day": 100, "k": 50}
DEFAULT_FORMAT = {"second": "iaga", "minute": "wdc", "hour": "wdc"} # others: csv
def get(url, params, retries=4):
"""Fetch bytes. Client errors (HTTP 300-422) stop at once and show the server's message."""
url += "?" + urllib.parse.urlencode(params)
req = urllib.request.Request(url, headers={"User-Agent": "wdc-bulk-download-script"})
for attempt in range(retries):
try:
return urllib.request.urlopen(req, timeout=300).read()
except urllib.error.HTTPError as e:
if 300 <= e.code <= 422:
msg = e.read().decode(errors="replace")[:500]
raise RuntimeError(f"HTTP {e.code}: {msg}\n{url}") from None
err = e
except urllib.error.URLError as e:
err = e
time.sleep(2 ** attempt)
raise RuntimeError(f"Failed after {retries} tries: {err}\n{url}")
def download_wdc(cadence="hour", start="2000-01-01", end="2020-01-01", codes=None,
file_format=None, orientation=None, outdir="wdc_downloads",
dry_run=True, pause=1.0):
"""end is exclusive. codes: None (all with data), "esk,ler" or ["esk", "ler"]."""
if cadence not in LIMITS:
raise ValueError(f"cadence must be one of {list(LIMITS)}")
start, end = date.fromisoformat(start), date.fromisoformat(end)
if start >= end:
raise ValueError("start must be before end")
by_month = LIMITS[cadence] < 1 # one-second data steps month by month
budget = round(LIMITS[cadence] * 12) if by_month else int(LIMITS[cadence])
keep = {c.strip().lower() for c in (codes.split(",") if isinstance(codes, str) else codes or [])}
file_format = file_format or DEFAULT_FORMAT.get(cadence, "csv")
to_date = lambda i: date(i // 12, i % 12 + 1, 1) if by_month else date(i, 1, 1)
# 1. One availability call: which observatories have data in each step (year or month)
avail = json.loads(get(f"{BASE}/data-availability/{cadence}",
{"start_dt": start, "end_dt": end, "formatting": "simple-json"}))
first = start.year * 12 + start.month - 1
last = end.year * 12 + end.month - 1 + (1 if end.day > 1 else 0)
steps = defaultdict(set)
for code, years in avail.items():
if keep and code not in keep:
continue
for year, info in years.items():
for month in info["months"]:
i = int(year) * 12 + month - 1
if first <= i < last:
steps[i if by_month else int(year)].add(code)
# 2. Grow each time slice until observatories x steps would exceed the limit
keys, plan, i = sorted(steps), [], 0
while i < len(keys):
group, j = set(steps[keys[i]]), i
if len(group) > budget: # too many observatories in one step: split them up
names = sorted(group)
plan += [(names[k:k + budget], keys[i], keys[i]) for k in range(0, len(names), budget)]
i += 1
continue
while j + 1 < len(keys) and keys[j + 1] == keys[j] + 1: # contiguous steps only
bigger = group | steps[keys[j + 1]]
if len(bigger) * (j + 2 - i) > budget:
break
group, j = bigger, j + 1
plan.append((sorted(group), keys[i], keys[j]))
i = j + 1
# 3. Name each file after its time slice (plus _partN if the slice had to be split)
n_slices, part, jobs = defaultdict(int), defaultdict(int), []
for _, a, b in plan:
n_slices[a, b] += 1
for names, a, b in plan:
s, e = max(to_date(a), start), min(to_date(b + 1), end)
part[a, b] += 1
tag = f"_part{part[a, b]}" if n_slices[a, b] > 1 else ""
jobs.append((names, s, e, f"{cadence}_{s:%Y%m%d}_{e:%Y%m%d}{tag}.zip"))
if not jobs:
print("No data found for that period.")
return
print(f"{len(jobs)} request(s), at most {budget} observatory-{'months' if by_month else 'years'} each")
for names, s, e, name in jobs:
print(f" {name}: {len(names)} observatories ({', '.join(names)})")
if dry_run:
print("Dry run only. Add dry_run=False to download.")
return
# 4. Download, and keep a manifest of what is in each file
out = Path(outdir)
out.mkdir(parents=True, exist_ok=True)
for k, (names, s, e, name) in enumerate(jobs, 1):
if (out / name).exists():
print(f"[{k}/{len(jobs)}] already have {name}")
continue
params = {"start_dt": s, "end_dt": e, "file_format": file_format}
if orientation:
params["orientation"] = orientation
time.sleep(pause)
try:
data = get(f"{BASE}/zipped-data/{cadence}/{','.join(names)}", params)
except RuntimeError as err:
print(f"[{k}/{len(jobs)}] FAILED {name}\n {err}")
continue
if data[:2] != b"PK": # a "no data" (202) reply is text, not a zip
print(f"[{k}/{len(jobs)}] no data: {name}")
else:
(out / name).write_bytes(data)
print(f"[{k}/{len(jobs)}] saved {name} ({len(data) / 1e6:.1f} MB)")
Then call it. It defaults to a dry run that only prints the plan:
download_wdc("hour", "1950-01-01", "1960-01-01") # dry run: prints the plan
download_wdc("hour", "1950-01-01", "1960-01-01", dry_run=False) # download (WDC format by default)
download_wdc("hour", "1950-01-01", "1960-01-01", file_format="iaga", dry_run=False) # IAGA-2002 instead
download_wdc("day", "1995-01-01", "2010-01-01", codes=["esk", "ler"], dry_run=False)
A failed request prints the server's message and the script carries on with the next one. Re-run it afterwards to fetch only what is missing.
A 413 means a request was still too large for the server. Lower LIMITS for that cadence and re-run.
For Linux and macOS terminals (and Windows with WSL or Git Bash). Needs bash and curl, with no installs.
Preview script
#!/usr/bin/env bash
# wdc_bulk_download_curl.sh
# Bulk download from the WDC Geomagnetism API (https://wdcapi.bgs.ac.uk) using curl.
#
# Works out which observatories have data, splits your request into pieces that
# fit the size limits, and saves each piece as a zip file in the output folder.
# Needs bash, curl, awk, grep and tr. No other installs.
#
# Usage: ./wdc_bulk_download_curl.sh -c hour -s 1950-01-01 -e 1960-01-01 -g
# -c cadence second, minute, hour, day or k (default hour)
# -s start first date, YYYY-MM-DD (default 2000-01-01)
# -e end end date, YYYY-MM-DD, not included (default 2020-01-01)
# -k codes observatory codes, e.g. esk,ler,had (default all with data)
# -f format csv, json, iaga, wdc, xml, cov-json (default depends on cadence)
# -r orient orientation, e.g. XYZF or HDZF (default as stored)
# -d folder output folder (default wdc_downloads)
# -p seconds pause between requests (default 1)
# -g go: really download (without -g it only prints the plan)
set -euo pipefail
BASE="https://wdcapi.bgs.ac.uk/data"
UA="wdc-bulk-download-script"
CADENCE=hour; START=2000-01-01; END=2020-01-01; CODES=""
FORMAT=""; ORIENT=""; OUTDIR=wdc_downloads; PAUSE=1; GO=0
die() { echo "Error: $*" >&2; exit 1; }
usage() { sed -n '/^# Usage/,/^$/p' "$0" | sed 's/^# \{0,1\}//'; }
while getopts "c:s:e:k:f:r:d:p:gh" opt; do
case $opt in
c) CADENCE=$OPTARG ;; s) START=$OPTARG ;; e) END=$OPTARG ;;
k) CODES=$OPTARG ;; f) FORMAT=$OPTARG ;; r) ORIENT=$OPTARG ;;
d) OUTDIR=$OPTARG ;; p) PAUSE=$OPTARG ;; g) GO=1 ;;
h) usage; exit 0 ;; *) usage; exit 1 ;;
esac
done
TMP=$(mktemp -d)
trap 'rm -rf "$TMP"' EXIT
# Fetch a URL into a file. Client errors (HTTP 300-422) stop at once and show the
# server's message; anything else is retried a few times.
fetch() {
local out=$1 url=$2 status=000 attempt=0
while [ "$attempt" -lt 4 ]; do
status=$(curl -sL -A "$UA" --max-time 300 -o "$out" -w '%{http_code}' "$url" </dev/null 2>/dev/null) || status=000
if [ "$status" = 200 ] || [ "$status" = 202 ]; then return 0; fi
if [ "$status" -ge 300 ] && [ "$status" -le 422 ]; then
echo "HTTP $status: $(head -c 500 "$out")" >&2
echo "$url" >&2
return 2
fi
attempt=$(( attempt + 1 )); sleep $(( 2 ** attempt ))
done
echo "Failed after 4 tries (last status $status): $url" >&2
return 1
}
# Largest request allowed: observatories x years (one-second data: x months)
case $CADENCE in
second) BUDGET=4; BY_MONTH=1; : "${FORMAT:=iaga}" ;;
minute) BUDGET=10; BY_MONTH=0; : "${FORMAT:=wdc}" ;;
hour) BUDGET=50; BY_MONTH=0; : "${FORMAT:=wdc}" ;;
day) BUDGET=100; BY_MONTH=0; : "${FORMAT:=csv}" ;;
k) BUDGET=50; BY_MONTH=0; : "${FORMAT:=csv}" ;;
*) die "cadence must be second, minute, hour, day or k" ;;
esac
datepat='^[0-9]{4}-[0-9]{2}-[0-9]{2}$'
[[ $START =~ $datepat && $END =~ $datepat ]] || die "dates must look like 2020-01-01"
[[ $START < $END ]] || die "start must be before end"
CODES=$(printf '%s' "$CODES" | tr 'A-Z' 'a-z' | tr -d ' ')
[[ -z $CODES || $CODES =~ ^[a-z]{3}(,[a-z]{3})*$ ]] || die "codes must be three letters, separated by commas"
# Months since year 0, so start and end can be compared as numbers
FIRST=$(( 10#${START:0:4} * 12 + 10#${START:5:2} - 1 ))
LAST=$(( 10#${END:0:4} * 12 + 10#${END:5:2} - 1 ))
if [ $(( 10#${END:8:2} )) -gt 1 ]; then LAST=$(( LAST + 1 )); fi
# 1. One availability call for the whole period
echo "Checking availability..."
fetch "$TMP/avail.json" \
"$BASE/data-availability/$CADENCE?start_dt=$START&end_dt=$END&formatting=simple-json" || exit 1
# 2. Turn the JSON into lines of "code year months", then plan the requests.
# Each time slice grows until observatories x steps would exceed the limit,
# and uses only the observatories that have data inside it.
{ tr -d ' \t\r\n' < "$TMP/avail.json" \
| { grep -oE '"[A-Za-z0-9_]+":\{("months":\[[0-9,]*\])?' || true; } \
| awk -F'"' '/"months"/ { s = $0; sub(/.*\[/, "", s); sub(/\].*/, "", s); print code, $2, s; next }
{ code = $2 }' \
| awk -v budget="$BUDGET" -v bymonth="$BY_MONTH" -v first="$FIRST" -v last="$LAST" \
-v start="$START" -v end="$END" -v cad="$CADENCE" -v keep="$CODES" '
function sortn(a, m, i, j, t) {
for (i = 2; i <= m; i++) { t = a[i]; for (j = i - 1; j >= 1 && a[j] > t; j--) a[j+1] = a[j]; a[j+1] = t }
}
function dt(idx) {
return (bymonth == 1) ? sprintf("%04d-%02d-01", int(idx / 12), idx % 12 + 1) : sprintf("%04d-01-01", idx)
}
function job(a, b, names) { nj++; ja[nj] = a; jb[nj] = b; jn[nj] = names; total[a, b]++ }
BEGIN {
n = split(keep, kk, ",")
for (i = 1; i <= n; i++) want[kk[i]] = 1
}
{
if (keep != "" && !($1 in want)) next
n = split($3, ms, ",")
for (k = 1; k <= n; k++) {
i = $2 * 12 + ms[k] - 1
if (i < first || i >= last) continue
key = (bymonth == 1) ? i : $2 + 0
if (!((key, $1) in seen)) { seen[key, $1] = 1; cnt[key]++; list[key, cnt[key]] = $1 }
if (!got || key < lo) lo = key
if (!got || key > hi) hi = key
got = 1
}
}
END {
if (!got) exit
i = lo
while (i <= hi) {
if (!(i in cnt)) { i++; continue }
delete grp; ng = 0
for (k = 1; k <= cnt[i]; k++) { grp[list[i, k]] = 1; ng++ }
if (ng > budget) { # too many observatories in one step: split them up
m = 0; for (k = 1; k <= cnt[i]; k++) names[++m] = list[i, k]
sortn(names, m)
for (k = 1; k <= m; k += budget) {
s = ""; for (q = k; q < k + budget && q <= m; q++) s = s (s == "" ? "" : ",") names[q]
job(i, i, s)
}
i++; continue
}
j = i
while (j + 1 <= hi && ((j + 1) in cnt)) { # contiguous steps only
nn = ng
for (k = 1; k <= cnt[j + 1]; k++) if (!(list[j + 1, k] in grp)) nn++
if (nn * (j + 2 - i) > budget) break
for (k = 1; k <= cnt[j + 1]; k++) grp[list[j + 1, k]] = 1
ng = nn; j++
}
m = 0; for (c in grp) names[++m] = c
sortn(names, m)
s = ""; for (q = 1; q <= m; q++) s = s (q > 1 ? "," : "") names[q]
job(i, j, s)
i = j + 1
}
for (x = 1; x <= nj; x++) { # name each file after its time slice (+ _partN)
s = dt(ja[x]); e = dt(jb[x] + 1)
if (s < start) s = start
if (e > end) e = end
sn = s; en = e; gsub(/-/, "", sn); gsub(/-/, "", en)
part[ja[x], jb[x]]++
tag = (total[ja[x], jb[x]] > 1) ? "_part" part[ja[x], jb[x]] : ""
printf "%s_%s_%s%s.zip\t%s\t%s\t%s\n", cad, sn, en, tag, s, e, jn[x]
}
}'
} > "$TMP/jobs.tsv"
[ -s "$TMP/jobs.tsv" ] || { echo "No data found for that period."; exit 0; }
N=$(wc -l < "$TMP/jobs.tsv" | tr -d ' ')
echo "$N request(s), at most $BUDGET observatory-$([ "$BY_MONTH" = 1 ] && echo months || echo years) each"
while IFS=$'\t' read -r name s e codes; do
echo " $name: $(awk -F, '{ print NF }' <<< "$codes") observatories (${codes//,/, })"
done < "$TMP/jobs.tsv"
if [ "$GO" != 1 ]; then
echo "Dry run only. Add -g to download."
exit 0
fi
# 3. Download
mkdir -p "$OUTDIR"
k=0; failed=0
while IFS=$'\t' read -r name s e codes <&3; do
k=$(( k + 1 ))
if [ -e "$OUTDIR/$name" ]; then echo "[$k/$N] already have $name"; continue; fi
url="$BASE/zipped-data/$CADENCE/$codes?start_dt=$s&end_dt=$e&file_format=$FORMAT"
if [ -n "$ORIENT" ]; then url="$url&orientation=$ORIENT"; fi
sleep "$PAUSE"
if ! fetch "$TMP/part" "$url"; then
echo "[$k/$N] FAILED $name"; failed=$(( failed + 1 )); continue
fi
if [ "$(head -c 2 "$TMP/part")" = "PK" ]; then # a "no data" (202) reply is text, not a zip
mv "$TMP/part" "$OUTDIR/$name"
echo "[$k/$N] saved $name ($(du -h "$OUTDIR/$name" | cut -f1))"
else
echo "[$k/$N] no data: $name"
fi
done 3< "$TMP/jobs.tsv"
if [ "$failed" -gt 0 ]; then
echo "$failed request(s) failed. Run the same command again to fetch only what is missing."
exit 1
fi
Make it runnable, then do a dry run. It only prints the plan unless you add -g:
chmod +x wdc_bulk_download_curl.sh
./wdc_bulk_download_curl.sh -c hour -s 1950-01-01 -e 1960-01-01 # dry run: prints the plan
./wdc_bulk_download_curl.sh -c hour -s 1950-01-01 -e 1960-01-01 -g # download (WDC format by default)
./wdc_bulk_download_curl.sh -c hour -s 1950-01-01 -e 1960-01-01 -f iaga -g # IAGA-2002 instead
./wdc_bulk_download_curl.sh -c day -s 1995-01-01 -e 2010-01-01 -k esk,ler -g
A failed request prints the server's message and the script carries on with the next one. Run the same command again to fetch only what is missing. Use -h to list all options.
The same script for systems that have wget instead of curl. Needs bash and GNU wget 1.19.2 or newer, with no other installs.
Preview script
#!/usr/bin/env bash
# wdc_bulk_download_wget.sh
# Bulk download from the WDC Geomagnetism API (https://wdcapi.bgs.ac.uk) using wget.
#
# Works out which observatories have data, splits your request into pieces that
# fit the size limits, and saves each piece as a zip file in the output folder.
# Needs bash, GNU wget 1.19.2 or newer, awk, grep and tr. No other installs.
#
# Usage: ./wdc_bulk_download_wget.sh -c hour -s 1950-01-01 -e 1960-01-01 -g
# -c cadence second, minute, hour, day or k (default hour)
# -s start first date, YYYY-MM-DD (default 2000-01-01)
# -e end end date, YYYY-MM-DD, not included (default 2020-01-01)
# -k codes observatory codes, e.g. esk,ler,had (default all with data)
# -f format csv, json, iaga, wdc, xml, cov-json (default depends on cadence)
# -r orient orientation, e.g. XYZF or HDZF (default as stored)
# -d folder output folder (default wdc_downloads)
# -p seconds pause between requests (default 1)
# -g go: really download (without -g it only prints the plan)
set -euo pipefail
BASE="https://wdcapi.bgs.ac.uk/data"
UA="wdc-bulk-download-script"
CADENCE=hour; START=2000-01-01; END=2020-01-01; CODES=""
FORMAT=""; ORIENT=""; OUTDIR=wdc_downloads; PAUSE=1; GO=0
die() { echo "Error: $*" >&2; exit 1; }
usage() { sed -n '/^# Usage/,/^$/p' "$0" | sed 's/^# \{0,1\}//'; }
while getopts "c:s:e:k:f:r:d:p:gh" opt; do
case $opt in
c) CADENCE=$OPTARG ;; s) START=$OPTARG ;; e) END=$OPTARG ;;
k) CODES=$OPTARG ;; f) FORMAT=$OPTARG ;; r) ORIENT=$OPTARG ;;
d) OUTDIR=$OPTARG ;; p) PAUSE=$OPTARG ;; g) GO=1 ;;
h) usage; exit 0 ;; *) usage; exit 1 ;;
esac
done
TMP=$(mktemp -d)
trap 'rm -rf "$TMP"' EXIT
# Fetch a URL into a file. Client errors (HTTP 300-422) stop at once and show the
# server's message; anything else is retried a few times.
fetch() {
local out=$1 url=$2 status=000 attempt=0
while [ "$attempt" -lt 4 ]; do
wget -nv -S --content-on-error --tries=1 --timeout=300 -U "$UA" -O "$out" "$url" 2>"$TMP/headers" </dev/null || true
status=$(grep -o 'HTTP/[0-9.]* [0-9][0-9][0-9]' "$TMP/headers" | tail -1 | awk '{ print $2 }' || true)
status=${status:-000}
if [ "$status" = 200 ] || [ "$status" = 202 ]; then return 0; fi
if [ "$status" -ge 300 ] && [ "$status" -le 422 ]; then
echo "HTTP $status: $(head -c 500 "$out")" >&2
echo "$url" >&2
return 2
fi
attempt=$(( attempt + 1 )); sleep $(( 2 ** attempt ))
done
echo "Failed after 4 tries (last status $status): $url" >&2
return 1
}
# Largest request allowed: observatories x years (one-second data: x months)
case $CADENCE in
second) BUDGET=4; BY_MONTH=1; : "${FORMAT:=iaga}" ;;
minute) BUDGET=10; BY_MONTH=0; : "${FORMAT:=wdc}" ;;
hour) BUDGET=50; BY_MONTH=0; : "${FORMAT:=wdc}" ;;
day) BUDGET=100; BY_MONTH=0; : "${FORMAT:=csv}" ;;
k) BUDGET=50; BY_MONTH=0; : "${FORMAT:=csv}" ;;
*) die "cadence must be second, minute, hour, day or k" ;;
esac
datepat='^[0-9]{4}-[0-9]{2}-[0-9]{2}$'
[[ $START =~ $datepat && $END =~ $datepat ]] || die "dates must look like 2020-01-01"
[[ $START < $END ]] || die "start must be before end"
CODES=$(printf '%s' "$CODES" | tr 'A-Z' 'a-z' | tr -d ' ')
[[ -z $CODES || $CODES =~ ^[a-z]{3}(,[a-z]{3})*$ ]] || die "codes must be three letters, separated by commas"
# Months since year 0, so start and end can be compared as numbers
FIRST=$(( 10#${START:0:4} * 12 + 10#${START:5:2} - 1 ))
LAST=$(( 10#${END:0:4} * 12 + 10#${END:5:2} - 1 ))
if [ $(( 10#${END:8:2} )) -gt 1 ]; then LAST=$(( LAST + 1 )); fi
# 1. One availability call for the whole period
echo "Checking availability..."
fetch "$TMP/avail.json" \
"$BASE/data-availability/$CADENCE?start_dt=$START&end_dt=$END&formatting=simple-json" || exit 1
# 2. Turn the JSON into lines of "code year months", then plan the requests.
# Each time slice grows until observatories x steps would exceed the limit,
# and uses only the observatories that have data inside it.
{ tr -d ' \t\r\n' < "$TMP/avail.json" \
| { grep -oE '"[A-Za-z0-9_]+":\{("months":\[[0-9,]*\])?' || true; } \
| awk -F'"' '/"months"/ { s = $0; sub(/.*\[/, "", s); sub(/\].*/, "", s); print code, $2, s; next }
{ code = $2 }' \
| awk -v budget="$BUDGET" -v bymonth="$BY_MONTH" -v first="$FIRST" -v last="$LAST" \
-v start="$START" -v end="$END" -v cad="$CADENCE" -v keep="$CODES" '
function sortn(a, m, i, j, t) {
for (i = 2; i <= m; i++) { t = a[i]; for (j = i - 1; j >= 1 && a[j] > t; j--) a[j+1] = a[j]; a[j+1] = t }
}
function dt(idx) {
return (bymonth == 1) ? sprintf("%04d-%02d-01", int(idx / 12), idx % 12 + 1) : sprintf("%04d-01-01", idx)
}
function job(a, b, names) { nj++; ja[nj] = a; jb[nj] = b; jn[nj] = names; total[a, b]++ }
BEGIN {
n = split(keep, kk, ",")
for (i = 1; i <= n; i++) want[kk[i]] = 1
}
{
if (keep != "" && !($1 in want)) next
n = split($3, ms, ",")
for (k = 1; k <= n; k++) {
i = $2 * 12 + ms[k] - 1
if (i < first || i >= last) continue
key = (bymonth == 1) ? i : $2 + 0
if (!((key, $1) in seen)) { seen[key, $1] = 1; cnt[key]++; list[key, cnt[key]] = $1 }
if (!got || key < lo) lo = key
if (!got || key > hi) hi = key
got = 1
}
}
END {
if (!got) exit
i = lo
while (i <= hi) {
if (!(i in cnt)) { i++; continue }
delete grp; ng = 0
for (k = 1; k <= cnt[i]; k++) { grp[list[i, k]] = 1; ng++ }
if (ng > budget) { # too many observatories in one step: split them up
m = 0; for (k = 1; k <= cnt[i]; k++) names[++m] = list[i, k]
sortn(names, m)
for (k = 1; k <= m; k += budget) {
s = ""; for (q = k; q < k + budget && q <= m; q++) s = s (s == "" ? "" : ",") names[q]
job(i, i, s)
}
i++; continue
}
j = i
while (j + 1 <= hi && ((j + 1) in cnt)) { # contiguous steps only
nn = ng
for (k = 1; k <= cnt[j + 1]; k++) if (!(list[j + 1, k] in grp)) nn++
if (nn * (j + 2 - i) > budget) break
for (k = 1; k <= cnt[j + 1]; k++) grp[list[j + 1, k]] = 1
ng = nn; j++
}
m = 0; for (c in grp) names[++m] = c
sortn(names, m)
s = ""; for (q = 1; q <= m; q++) s = s (q > 1 ? "," : "") names[q]
job(i, j, s)
i = j + 1
}
for (x = 1; x <= nj; x++) { # name each file after its time slice (+ _partN)
s = dt(ja[x]); e = dt(jb[x] + 1)
if (s < start) s = start
if (e > end) e = end
sn = s; en = e; gsub(/-/, "", sn); gsub(/-/, "", en)
part[ja[x], jb[x]]++
tag = (total[ja[x], jb[x]] > 1) ? "_part" part[ja[x], jb[x]] : ""
printf "%s_%s_%s%s.zip\t%s\t%s\t%s\n", cad, sn, en, tag, s, e, jn[x]
}
}'
} > "$TMP/jobs.tsv"
[ -s "$TMP/jobs.tsv" ] || { echo "No data found for that period."; exit 0; }
N=$(wc -l < "$TMP/jobs.tsv" | tr -d ' ')
echo "$N request(s), at most $BUDGET observatory-$([ "$BY_MONTH" = 1 ] && echo months || echo years) each"
while IFS=$'\t' read -r name s e codes; do
echo " $name: $(awk -F, '{ print NF }' <<< "$codes") observatories (${codes//,/, })"
done < "$TMP/jobs.tsv"
if [ "$GO" != 1 ]; then
echo "Dry run only. Add -g to download."
exit 0
fi
# 3. Download
mkdir -p "$OUTDIR"
k=0; failed=0
while IFS=$'\t' read -r name s e codes <&3; do
k=$(( k + 1 ))
if [ -e "$OUTDIR/$name" ]; then echo "[$k/$N] already have $name"; continue; fi
url="$BASE/zipped-data/$CADENCE/$codes?start_dt=$s&end_dt=$e&file_format=$FORMAT"
if [ -n "$ORIENT" ]; then url="$url&orientation=$ORIENT"; fi
sleep "$PAUSE"
if ! fetch "$TMP/part" "$url"; then
echo "[$k/$N] FAILED $name"; failed=$(( failed + 1 )); continue
fi
if [ "$(head -c 2 "$TMP/part")" = "PK" ]; then # a "no data" (202) reply is text, not a zip
mv "$TMP/part" "$OUTDIR/$name"
echo "[$k/$N] saved $name ($(du -h "$OUTDIR/$name" | cut -f1))"
else
echo "[$k/$N] no data: $name"
fi
done 3< "$TMP/jobs.tsv"
if [ "$failed" -gt 0 ]; then
echo "$failed request(s) failed. Run the same command again to fetch only what is missing."
exit 1
fi
Make it runnable, then do a dry run. It only prints the plan unless you add -g:
chmod +x wdc_bulk_download_wget.sh
./wdc_bulk_download_wget.sh -c hour -s 1950-01-01 -e 1960-01-01 # dry run: prints the plan
./wdc_bulk_download_wget.sh -c hour -s 1950-01-01 -e 1960-01-01 -g # download (WDC format by default)
./wdc_bulk_download_wget.sh -c hour -s 1950-01-01 -e 1960-01-01 -f iaga -g # IAGA-2002 instead
./wdc_bulk_download_wget.sh -c day -s 1995-01-01 -e 2010-01-01 -k esk,ler -g
It takes the same options as the curl version and gives the same file names. Use -h to list all options.
Check what data exist
Before a big download, you can see which months have data. The availability endpoint lists, for each observatory and year, the months we hold for a chosen cadence.
https://wdcapi.bgs.ac.uk/data/data-availability/hour?obs_code=esk&start_dt=2020-01-01&end_dt=2021-01-01&formatting=simple-json
Swap hour for second, minute, day, month, year or k. obs_code takes one or more codes;
leave it out, along with the dates, and you get every observatory for all time,
The formatting option is simple-json (default) or nested-json.
The response describes availability only and contains no magnetic data.
A trimmed example of a response:
{
"bsl": {
"2021": { "months": [1, 2, 3, 4, 5, 6, 7, 8, 10, 11, 12] },
"2025": { "months": [] }
}
}An empty list means no data for that year. Here September 2021 is missing.
Text endpoints
These return data directly in the response as CSV, JSON or COV-JSON instead of a zip, so a program can read them immediately. They cover one observatory at a time, and split large requests into pages, and are not designed for bulk downloads across many observatories, but are useful for scraping large volumes of targetted data. They don't offer WDC, IAGA-2002 or XML; use zipped downloads for those.
https://wdcapi.bgs.ac.uk/data/text-data/cadence/observatory?start_dt=start&end_dt=end&file_format=format&months_per_page=n&page=pTry one month of minute data
https://wdcapi.bgs.ac.uk/data/text-data/minute/esk?start_dt=2010-01-01&end_dt=2010-02-01&file_format=json&orientation=original&page=1&months_per_page=1How paging works
Every response includes page and total pages, plus a data list. Ask for ten years (2010-01-01 to 2020-01-01) with months_per_page=1 and total pages is 120, so you collect everything by requesting page=1, page=2, and so on up to page=120.
Always read total pages from the response rather than assuming it.
For day, hour, minute and K-index requests you must include months_per_page. The most each cadence allows:
Monthly and yearly requests don't use months_per_page. If there's nothing for your dates you get a 202 reply with an empty data list.
Python: collect every page into a table
import requests
import pandas as pd
url = "https://wdcapi.bgs.ac.uk/data/text-data/minute/esk"
params = {
"start_dt": "2010-01-01",
"end_dt": "2020-01-01",
"file_format": "json",
"orientation": "original",
"months_per_page": 1,
"page": 1,
}
rows = []
while True:
r = requests.get(url, params=params, timeout=120)
r.raise_for_status()
body = r.json()
rows.extend(body["data"])
if params["page"] >= body["total pages"]:
break
params["page"] += 1
df = pd.DataFrame(rows)
print(df.head())What a response looks like
{
"iaga-code": "...",
"latitude": ...,
"longitude": ...,
"data-quality": "...",
"converted from": "...",
"page": 1,
"total pages": 120,
"data": [
{ "X": ..., "Y": ..., "Z": ..., "datetime": "..." }
]
}The names of the data columns depend on the orientation you ask for.
One-second data
One-second data are available as JSON for a single observatory, with pages measured in days (days_per_page, 1 or 2, default 2):
https://wdcapi.bgs.ac.uk/data/text-data/second/kir?start_dt=2024-01-01&end_dt=2024-01-03&page=1The “Larger requests via API” section of the FAQs on the World Data Centre data portal also has a ready-made scraping script.
Download with curl or Python
Scripts are the way to repeat downloads, loop over years, or fetch many observatories. Each example below saves a zip file.
curl is a command-line tool for web requests, available on most Linux and macOS systems and installable on Windows. Put the link in quotes so your shell treats it as one argument.
curl -fL -o esk-hourly-2020.zip "https://wdcapi.bgs.ac.uk/data/zipped-data/hour/esk?start_dt=2020-01-01&end_dt=2021-01-01&file_format=wdc&orientation=original"
-f reports HTTP errors as failures, -L follows redirects and -o names the file to save.
A 202 “no data” reply is not an error to curl, so if the saved file is empty, this is why.
This uses only Python's standard library.
import zipfile
from urllib.request import urlretrieve
url = (
"https://wdcapi.bgs.ac.uk/data/zipped-data/hour/esk"
"?start_dt=2020-01-01"
"&end_dt=2021-01-01"
"&file_format=wdc"
"&orientation=original"
)
filename = "esk-hourly-2020.zip"
urlretrieve(url, filename)
if zipfile.is_zipfile(filename):
print("Downloaded", filename)
else:
# No data for these criteria: the server sent a short message instead of a zip.
print(open(filename).read())This needs the requests package (pip install requests) and fetches two years of minute data, one year per request.
import requests
from pathlib import Path
from zipfile import is_zipfile
from io import BytesIO
url = "https://wdcapi.bgs.ac.uk/data/zipped-data/minute/esk"
# Save files in a dedicated folder
output_dir = Path.cwd() / "esk_data"
output_dir.mkdir(parents=True, exist_ok=True)
print("Saving files to:", output_dir.resolve())
for year in range(2010, 2012):
try:
r = requests.get(
url,
params={"years": year, "file_format": "csv"},
timeout=600
)
print(f"\nYear: {year}")
print("Status:", r.status_code)
print("Content type:", r.headers.get("Content-Type"))
print("Downloaded bytes:", len(r.content))
if r.status_code == 200:
if is_zipfile(BytesIO(r.content)):
file_path = output_dir / f"esk_minute_{year}.zip"
file_path.write_bytes(r.content)
print("Saved:", file_path.resolve())
else:
print("Response is not a valid ZIP file.")
print(r.text)
elif r.status_code == 202:
print("Request accepted, but data is not yet available in this response.")
print(r.text)
else:
print("Request failed:", r.text)
except requests.RequestException as e:
print(f"Error downloading {year}: {e}")
For scripts you rely on, add retries and check that each zip opens before you use it.
HAPI server
We also have a HAPI server. Our server follows version 3.2 of the
Heliophysics API (HAPI) specification,
a standard way for software to discover time series, inspect their parameters and fetch data for a time range.
Tools that read HAPI servers, such as Autoplot or the hapiclient packages for Python, IDL and MATLAB, can be pointed at https://wdcapi.bgs.ac.uk/hapi.
/hapi/capabilitiescsv, json)./hapi/about/hapi/catalog/hapi/info/hapi/dataDataset names
Each dataset is named observatory/cadence/orientation, so this is Eskdalemuir minute means in XYZ:
esk/PT1M/xyzCadence
PT1M minute · PT1H hourly · P1D daily · P1M monthly · P1Y yearly · PT3H K-index
Orientation
xyz · hdz · original (as stored) · k (three-hourly K-index only)
Inspect a dataset
https://wdcapi.bgs.ac.uk/hapi/info?dataset=esk/PT1M/originalOther examples: esk/PT1M/xyz, esk/PT1M/hdz and esk/PT3H/k. Check the catalogue for the datasets that exist.
Fetch data
One day of Eskdalemuir minute data as CSV, then the same as JSON:
https://wdcapi.bgs.ac.uk/hapi/data?dataset=esk/PT1M/original&format=csv&start=2020-01-01T00:00:00Z&stop=2020-01-02T00:00:00Zhttps://wdcapi.bgs.ac.uk/hapi/data?dataset=esk/PT1M/original&format=json&start=2020-01-01T00:00:00Z&stop=2020-01-02T00:00:00Z
format is csv (default) or json. Add include=header to receive the dataset description with the data, or parameters= to choose columns. Times are UTC and the stop time marks the end of the period.
Request limits
A single HAPI data request can cover at most:
The maxRequestDuration value in a dataset's /hapi/info response is the definitive limit for that dataset. For longer periods, make several requests.
Use it from Python
import pandas as pd
url = ("https://wdcapi.bgs.ac.uk/hapi/data"
"?dataset=esk/PT1M/xyz&format=csv"
"&start=2020-01-01T00:00:00Z&stop=2020-01-02T00:00:00Z")
df = pd.read_csv(url, header=None,
names=["time", "x", "y", "z", "original_format"],
parse_dates=["time"])
print(df.head())# pip install hapiclient
from hapiclient import hapi
server = "https://wdcapi.bgs.ac.uk/hapi"
dataset = "esk/PT1M/xyz"
start, stop = "2020-01-01T00:00:00Z", "2020-01-02T00:00:00Z"
data, meta = hapi(server, dataset, "", start, stop) # "" means all parametersOther data and metadata
The same API serves several other datasets. Each of these is a link you can paste into your browser.
Monthly means in BGS format
Our database of observatory monthly means, compiled from definitive hourly data and near-definitive minute data, joined into one series per observatory in a format designed after IAGA-2002.
https://wdcapi.bgs.ac.uk/monthly-means?obs_code=eskLeave out obs_code to download every observatory (this may take a few minutes), or add include_plot=true for PNG plots comparing each observatory with field models.
curl -X 'GET' \
'https://wdcapi.bgs.ac.uk/monthly-means' \
-H 'accept: application/json'There's also a browsable version on the Observatory Monthly Means page.
Annual means and jump information
https://wdcapi.bgs.ac.uk/data/annual-means-cataloguehttps://wdcapi.bgs.ac.uk/data/annual-means-discontinuities?obs_code=eskJumps come from changes of site or instrument. Values are old minus new, and no corrections have been applied. To make earlier annual means consistent with later ones, for example when calculating secular variation, subtract the jump from the earlier values.
Global geomagnetic survey data
https://wdcapi.bgs.ac.uk/global-geomagnetic-survey?start_dt=2020-01-01&end_dt=2021-01-01&file_format=csvFormats are json, csv and geo-json. Without dates you get the last five years. Limit the area with north-latitude, south-latitude, west-longitude and east-longitude, and choose measurement types with include-one-off, include-repeat-station, include-observatory, include-aeromag, include-marine-vector, include-marine-scalar and include-satellite. By default only one-off and repeat-station results are included.
Observatory metadata and locations
https://wdcapi.bgs.ac.uk/metadata/observatory-metadata/eskInstruments, locations, institutes and contacts for one observatory. Leave off the code to get every observatory, which is a much larger response.
https://wdcapi.bgs.ac.uk/metadata/location?observatory=eskLatitude, longitude, elevation and opening and closing dates, including past moves.
GeoJSON for maps
https://wdcapi.bgs.ac.uk/metadata/wdc-geojson/Observatory locations for mapping software. Note that this version lists coordinates as [latitude, longitude], whereas the EPOS version (/metadata/epos-geojson/) uses [longitude, latitude].
Versions that add IGRF-14 field values (/metadata/wdc-geojson-igrf) or quasi-dipole coordinates (/metadata/wdc-geojson-qd-coords) need a datetime parameter.
Yearbooks
https://wdcapi.bgs.ac.uk/metadata/yearbooks?observatory=eskLinks to digitised yearbooks. INTERMAGNET only version is also available at /metadata/intermagnet-yearbooks.
If something goes wrong
The server tells you what happened with a status code.
Common questions
Which format should I choose?
What's the difference between zipped/text endpoints and the HAPI server?
Can I download several observatories at once?
esk,ler,had. For monthly and yearly means you can enter ALL. Text endpoints and HAPI are one at a time.Where can I find every parameter?
Still stuck? Email wdcgeomag@bgs.ac.uk and include the link you used.
Acknowledging the data
Observatory data are the work of the teams that run each observatory. If you use them in a publication, please acknowledge them. Ready-made acknowledgement text and contact details for any observatory are available at:
https://wdcapi.bgs.ac.uk/acknowledgements/esk