Update correlation sources

Remove the stale metar.tbl and zonecatalog.curr.tar, which the USA
NWS hasn't been updating for many years, and add the public domain
airports.csv file from the amazing ourairports.com community. Also
update to latest (2019) USA Census Bureau location data, March 2020
WX zone information, cooperative sites list from 2018 (latest), and
regenerated active station and zone lists. Loss of the zonecatalog
necessitates directly applying various forecast and alert URL
patterns, though some which appeared unused by NWS for many years
were not included.

Clear out all old overrides, since the vast majority are obsoleted
by refreshed data, and build fresh correlation sets from the above
sources. Basically all sites have switched from HTTP to HTTPS, so
update URLs for this too.
This commit is contained in:
Jeremy Stanley
2020-05-23 18:09:39 +00:00
parent 1ec2848c20
commit 8a37eddc06
13 changed files with 186295 additions and 203172 deletions

3
.gitignore vendored
View File

@@ -1,9 +1,8 @@
*_old
*Gaz_*_national.zip
COOP-ACT.TXT
airports.csv
bp??????.dbx
coop-stations.txt
metar.tbl
nsd_cccc.txt
*.pyc
zonecatalog.curr.tar

View File

@@ -21,7 +21,7 @@ which follow.
Copyright Notice
----------------
Copyright (c) 2006-2012 Jeremy Stanley <fungi@yuggoth.org>
Copyright (c) 2006-2020 Jeremy Stanley <fungi@yuggoth.org>
Permission Notice
-----------------

11600
airports

File diff suppressed because it is too large Load Diff

File diff suppressed because it is too large Load Diff

File diff suppressed because it is too large Load Diff

150739
places

File diff suppressed because it is too large Load Diff

811
qa.log
View File

@@ -1,22 +1,761 @@
cahr: no description
cahr: no location
cerm: no description
cerm: no location
cmfm: no description
cmfm: no location
cmgb: no description
cmgb: no location
cpbt: no description
cpbt: no location
cpeh: no description
cpeh: no location
cpfi: no description
cpfi: no location
cpir: no description
cpir: no location
cpro: no description
cpro: no location
cpry: no description
cpry: no location
cpst: no description
cpst: no location
cpsv: no description
cpsv: no location
cpxl: no description
cpxl: no location
ctck: no description
ctck: no location
ctnk: no description
ctnk: no location
ctra: no description
ctra: no location
cwbu: no description
cwbu: no location
cwtt: no description
cwtt: no location
cxaf: no description
cxaf: no location
cxag: no description
cxag: no location
cxaj: no description
cxaj: no location
cxak: no description
cxak: no location
cxat: no description
cxat: no location
cxba: no description
cxba: no location
cxbi: no description
cxbi: no location
cxbr: no description
cxbr: no location
cxbw: no description
cxbw: no location
cxca: no description
cxca: no location
cxcd: no description
cxcd: no location
cxcp: no description
cxcp: no location
cxcs: no description
cxcs: no location
cxdb: no description
cxdb: no location
cxdk: no description
cxdk: no location
cxdp: no description
cxdp: no location
cxea: no description
cxea: no location
cxet: no description
cxet: no location
cxfm: no description
cxfm: no location
cxfr: no description
cxfr: no location
cxha: no description
cxha: no location
cxhd: no description
cxhd: no location
cxhm: no description
cxhm: no location
cxhp: no description
cxhp: no location
cxhr: no description
cxhr: no location
cxib: no description
cxib: no location
cxka: no description
cxka: no location
cxke: no description
cxke: no location
cxki: no description
cxki: no location
cxkm: no description
cxkm: no location
cxlb: no description
cxlb: no location
cxmg: no description
cxmg: no location
cxmo: no description
cxmo: no location
cxnp: no description
cxnp: no location
cxol: no description
cxol: no location
cxoy: no description
cxoy: no location
cxpa: no description
cxpa: no location
cxpc: no description
cxpc: no location
cxpl: no description
cxpl: no location
cxrb: no description
cxrb: no location
cxrl: no description
cxrl: no location
cxsc: no description
cxsc: no location
cxse: no description
cxse: no location
cxsl: no description
cxsl: no location
cxsp: no description
cxsp: no location
cxtd: no description
cxtd: no location
cxth: no description
cxth: no location
cxto: no description
cxto: no location
cxvm: no description
cxvm: no location
cxvn: no description
cxvn: no location
cxvw: no description
cxvw: no location
cxwb: no description
cxwb: no location
cxwm: no description
cxwm: no location
cxyh: no description
cxyh: no location
cxzc: no description
cxzc: no location
cxzv: no description
cxzv: no location
czcr: no description
czcr: no location
czdb: no description
czdb: no location
czel: no description
czel: no location
czev: no description
czev: no location
czfs: no description
czfs: no location
czhy: no description
czhy: no location
czkd: no description
czkd: no location
czmj: no description
czmj: no location
czmu: no description
czmu: no location
czol: no description
czol: no location
czps: no description
czps: no location
czsm: no description
czsm: no location
czsp: no description
czsp: no location
cztb: no description
cztb: no location
czzj: no description
czzj: no location
dnad: no description
dnad: no location
ehak: no description
ehak: no location
ehdv: no description
ehdv: no location
ehfd: no description
ehfd: no location
ehfs: no description
ehfs: no location
ehfz: no description
ehfz: no location
ehja: no description
ehja: no location
ehkv: no description
ehkv: no location
ehma: no description
ehma: no location
ehmg: no description
ehmg: no location
ehpg: no description
ehpg: no location
ehqe: no description
ehqe: no location
enhm: no description
enhm: no location
enle: no description
enle: no location
enne: no description
enne: no location
enqr: no description
enqr: no location
ense: no description
ense: no location
ensf: no description
ensf: no location
enug: no description
enug: no location
enun: no description
enun: no location
enwv: no description
enwv: no location
eqbk: no description
eqbk: no location
eqph: no description
eqph: no location
eqya: no description
eqya: no location
eqyb: no description
eqyb: no location
eqyc: no description
eqyc: no location
eqyd: no description
eqyd: no location
eqye: no description
eqye: no location
eqyf: no description
eqyf: no location
eqyk: no description
eqyk: no location
eqyl: no description
eqyl: no location
eqyt: no description
eqyt: no location
eqyv: no description
eqyv: no location
etml: no description
etml: no location
fajb: no description
fajb: no location
fdbb: no description
fdbb: no location
fdmy: no description
fdmy: no location
fdot: no description
fdot: no location
fdpp: no description
fdpp: no location
fqcl: no description
fqcl: no location
gakd: no description
gakd: no location
gmmp: no description
gmmp: no location
heis: no description
heis: no location
hesw: no description
hesw: no location
htha: no description
htha: no location
htmh: no description
htmh: no location
htsi: no description
htsi: no location
k00u: no description
k00u: no location
k04w: no description
k04w: no location
k05u: no description
k05u: no location
k06c: no description
k06c: no location
k0a0: no description
k0a0: no location
k0co: no description
k0co: no location
k0e0: no description
k0e0: no location
k18h: no description
k18h: no location
k19s: no description
k19s: no location
k1am: no description
k1am: no location
k1an: no description
k1an: no location
k1bm: no description
k1bm: no location
k1bn: no description
k1bn: no location
k1bw: no description
k1bw: no location
k1cm: no description
k1cm: no location
k1cw: no description
k1cw: no location
k1dm: no description
k1dm: no location
k1dn: no description
k1dn: no location
k1dw: no description
k1dw: no location
k1em: no description
k1em: no location
k1en: no description
k1en: no location
k1ew: no description
k1ew: no location
k1fm: no description
k1fm: no location
k1fw: no description
k1fw: no location
k1gm: no description
k1gm: no location
k1gn: no description
k1gn: no location
k1gw: no description
k1gw: no location
k1hm: no description
k1hm: no location
k1hn: no description
k1hn: no location
k1hw: no description
k1hw: no location
k1im: no description
k1im: no location
k1iw: no description
k1iw: no location
k1jm: no description
k1jm: no location
k1jn: no description
k1jn: no location
k1k1: no description
k1k1: no location
k1km: no description
k1km: no location
k1lm: no description
k1lm: no location
k1mm: no description
k1mm: no location
k1mw: no description
k1mw: no location
k1nm: no description
k1nm: no location
k1nn: no description
k1nn: no location
k1nw: no description
k1nw: no location
k1om: no description
k1om: no location
k1on: no description
k1on: no location
k1ow: no description
k1ow: no location
k1v6: no description
k1v6: no location
k27k: no description
k27k: no location
k2g4: no description
k2g4: no location
k2j9: no description
k2j9: no location
k2ls: no description
k2ls: no location
k2p2: no description
k2p2: no location
k2r9: no description
k2r9: no location
k2v6: no description
k2v6: no location
k2w6: no description
k2w6: no location
k33n: no description
k33n: no location
k3d2: no description
k3d2: no location
k3n8: no description
k3n8: no location
k3s8: no description
k3s8: no location
k41u: no description
k41u: no location
k42j: no description
k42j: no location
k48w: no description
k48w: no location
k4v0: no description
k4v0: no location
k54a: no description
k54a: no location
k59k: no description
k59k: no location
k5a6: no description
k5a6: no location
k5t9: no description
k5t9: no location
k65s: no description
k65s: no location
k66r: no description
k66r: no location
k6b0: no description
k6b0: no location
k6l4: no description
k6l4: no location
k6p9: no description
k6p9: no location
k6s2: no description
k6s2: no location
k74v: no description
k74v: no location
k7n0: no description
k7n0: no location
k7w4: no description
k7w4: no location
k82v: no description
k82v: no location
k8b0: no description
k8b0: no location
k9mn: no description
k9mn: no location
ka08: no description
ka08: no location
ka39: no description
ka39: no location
kabh: no description
kabh: no location
kalk: no description
kalk: no location
kbpc: no description
kbpc: no location
kc07: no description
kc07: no location
kc65: no description
kc65: no location
kc99: no description
kc99: no location
kcpf: no description
kcpf: no location
kcyd: no description
kcyd: no location
kd73: no description
kd73: no location
kd95: no description
kd95: no location
kdsf: no description
kdsf: no location
ke11: no description
ke11: no location
ke16: no description
ke16: no location
ke41: no description
ke41: no location
keod: no description
keod: no location
kezp: no description
kezp: no location
kf00: no description
kf00: no location
kf44: no description
kf44: no location
kf46: no description
kf46: no location
kf70: no description
kf70: no location
kfpy: no description
kfpy: no location
kfwz: no description
kfwz: no location
kgyf: no description
kgyf: no location
ki19: no description
ki19: no location
ki68: no description
ki68: no location
kjsl: no description
kjsl: no location
kky8: no description
kky8: no location
kl08: no description
kl08: no location
kl18: no description
kl18: no location
kl35: no description
kl35: no location
kl52: no description
kl52: no location
km25: no description
km25: no location
km40: no description
km40: no location
km75: no description
km75: no location
kmp1: no description
kmp1: no location
kmtk: no description
kmtk: no location
kmvh: no description
kmvh: no location
knoz: no description
knoz: no location
knxf: no description
knxf: no location
ko22: no description
ko22: no location
ko69: no description
ko69: no location
kpo1: no description
kpo1: no location
kpzz: no description
kpzz: no location
kqa5: no description
kqa5: no location
kqa8: no description
kqa8: no location
kqaj: no description
kqaj: no location
kqal: no description
kqal: no location
kqay: no description
kqay: no location
kqaz: no description
kqaz: no location
kqb4: no description
kqb4: no location
kqbd: no description
kqbd: no location
kqbg: no description
kqbg: no location
kqbl: no description
kqbl: no location
kqbr: no description
kqbr: no location
kqbt: no description
kqbt: no location
kqc7: no description
kqc7: no location
kqc8: no description
kqc8: no location
kqcl: no description
kqcl: no location
kqcp: no description
kqcp: no location
kqd3: no description
kqd3: no location
kqdg: no description
kqdg: no location
kqei: no description
kqei: no location
kqej: no description
kqej: no location
kqek: no description
kqek: no location
kqel: no description
kqel: no location
kqen: no description
kqen: no location
kqeo: no description
kqeo: no location
kqep: no description
kqep: no location
kqeq: no description
kqeq: no location
kqer: no description
kqer: no location
kqes: no description
kqes: no location
kqev: no description
kqev: no location
kqew: no description
kqew: no location
kqex: no description
kqex: no location
kqey: no description
kqey: no location
kqfb: no description
kqfb: no location
kqfl: no description
kqfl: no location
kqfs: no description
kqfs: no location
kqft: no description
kqft: no location
kqfu: no description
kqfu: no location
kqfv: no description
kqfv: no location
kqfw: no description
kqfw: no location
kqfx: no description
kqfx: no location
kqge: no description
kqge: no location
kqgx: no description
kqgx: no location
kqhb: no description
kqhb: no location
kqhc: no description
kqhc: no location
kqhk: no description
kqhk: no location
kqhy: no description
kqhy: no location
kqm2: no description
kqm2: no location
kqmk: no description
kqmk: no location
kqn3: no description
kqn3: no location
kqnd: no description
kqnd: no location
kqrd: no description
kqrd: no location
kqrh: no description
kqrh: no location
kqri: no description
kqri: no location
kqs3: no description
kqs3: no location
kqsf: no description
kqsf: no location
kqsi: no description
kqsi: no location
kqsn: no description
kqsn: no location
kqti: no description
kqti: no location
kqut: no description
kqut: no location
kqwm: no description
kqwm: no location
kqwx: no description
kqwx: no location
kqx2: no description
kqx2: no location
kqxh: no description
kqxh: no location
kqxt: no description
kqxt: no location
kqyv: no description
kqyv: no location
krga: no description
krga: no location
ks39: no description
ks39: no location
ktdr: no description
ktdr: no location
ku55: no description
ku55: no location
kvky: no description
kvky: no location
kvoa: no description
kvoa: no location
kvqt: no description
kvqt: no location
kw13: no description
kw13: no location
kw29: no description
kw29: no location
kw75: no description
kw75: no location
kw96: no description
kw96: no location
kwbf: no description
kwbf: no location
kxer: no description
kxer: no location
kxif: no description
kxif: no location
kxpy: no description
kxpy: no location
ky31: no description
ky31: no location
ky49: no description
ky49: no location
lebr: no description
lebr: no location
letu: no description
letu: no location
lirx: no description
lirx: no location
mgmt: no description
mgmt: no location
mgtu: no description
mgtu: no location
mhoc: no description
mhoc: no location
mhtr: no description
mhtr: no location
mjks: no description
mjks: no location
mslu: no description
mslu: no location
ncrk: no description
ncrk: no location
nfnf: no description
nfnf: no location
nzfx: no description
nzfx: no location
oear: no description
oear: no location
oekj: no description
oekj: no location
oicm: no description
oicm: no location
oimq: no description
oimq: no location
perr: no description
perr: no location
rrli: no description
rrli: no location
sbsy: no description
sbsy: no location
seca: no description
seca: no location
sgsj: no description
sgsj: no location
sgvr: no description
sgvr: no location
slgm: no description
slgm: no location
smae: no description
smae: no location
ugam: no description
ugam: no location
urff: no description
urff: no location
uuii: no description
uuii: no location
uwsg: no description
uwsg: no location
veko: no description
veko: no location
vepy: no description
vepy: no location
vond: no description
vond: no location
wahq: no description
wahq: no location
alz262: within one km of alz266
alz263: within one km of alz265
alz265: within one km of alz263
alz266: within one km of alz262
caz032: no centroid
caz101: within one km of caz103
caz102: within one km of caz101
caz103: within one km of caz101
caz104: within one km of caz101
caz105: within one km of caz101
caz106: within one km of caz101
caz107: within one km of caz110
caz108: within one km of caz110
caz110: within one km of caz108
caz111: within one km of caz112
caz112: within one km of caz113
caz113: within one km of caz112
caz549: no centroid
caz550: no centroid
caz107: within one km of caz108
caz108: within one km of caz107
flz024: within one km of flz124
flz025: within one km of flz125
flz124: within one km of flz024
@@ -27,78 +766,35 @@ flz148: within one km of flz248
flz149: within one km of flz249
flz155: within one km of flz255
flz160: within one km of flz260
flz162: within one km of flz262
flz239: within one km of flz139
flz242: within one km of flz142
flz248: within one km of flz148
flz249: within one km of flz149
flz255: within one km of flz155
flz260: within one km of flz160
flz262: within one km of flz162
idz001: within one km of wyz001
laz069: no centroid
laz073: within one km of laz074
laz074: within one km of laz073
mtz001: within one km of wyz001
nmz401: no centroid
nmz403: no centroid
nmz404: no centroid
nmz405: no centroid
nmz406: no centroid
nmz407: no centroid
nmz408: no centroid
nmz409: no centroid
nmz410: no centroid
nmz411: no centroid
nmz412: no centroid
nmz413: no centroid
nmz414: no centroid
nmz415: no centroid
nmz416: no centroid
nmz417: no centroid
orz015: no centroid
orz016: no centroid
orz510: no centroid
orz511: no centroid
paz101: within one km of paz102
paz102: within one km of paz101
paz103: within one km of paz104
paz104: within one km of paz103
paz105: within one km of paz106
paz106: within one km of paz105
txz418: no centroid
txz419: no centroid
txz420: no centroid
txz421: no centroid
txz422: no centroid
txz423: no centroid
txz424: no centroid
vaz509: no centroid
vaz510: no centroid
vaz511: no centroid
vaz512: no centroid
vaz513: within one km of vaz514
vaz514: within one km of vaz513
vaz515: within one km of vaz516
vaz516: within one km of vaz515
vaz517: within one km of vaz518
vaz518: within one km of vaz517
vaz519: within one km of vaz520
vaz520: within one km of vaz519
vaz523: no centroid
vtz004: no centroid
waz045: no centroid
waz046: no centroid
waz503: within one km of waz567
waz504: within one km of waz569
waz507: within one km of waz555
waz507: within one km of waz568
waz511: within one km of waz559
waz555: within one km of waz507
waz556: within one km of waz568
waz555: within one km of waz568
waz556: within one km of waz569
waz559: within one km of waz511
waz567: within one km of waz503
waz568: within one km of waz556
waz569: within one km of waz504
waz568: within one km of waz507
waz569: within one km of waz556
wvz506: within one km of wvz507
wvz507: within one km of wvz506
wvz509: within one km of wvz510
@@ -107,4 +803,3 @@ wvz511: within one km of wvz512
wvz512: within one km of wvz511
wvz513: within one km of wvz514
wvz514: within one km of wvz513
wyz001: within one km of mtz001

911
slist

File diff suppressed because it is too large Load Diff

20115
stations

File diff suppressed because it is too large Load Diff

View File

@@ -1,12 +1,12 @@
"""Contains various object definitions needed by the weather utility."""
weather_copyright = """\
# Copyright (c) 2006-2016 Jeremy Stanley <fungi@yuggoth.org>. Permission to
# Copyright (c) 2006-2020 Jeremy Stanley <fungi@yuggoth.org>. Permission to
# use, copy, modify, and distribute this software is granted under terms
# provided in the LICENSE file distributed with this software.
#"""
weather_version = "2.3"
weather_version = "2.4"
radian_to_km = 6372.795484
radian_to_mi = 3959.871528
@@ -1209,25 +1209,26 @@ def gecos(formatted):
return tuple(coordinates)
def correlate():
import codecs, datetime, hashlib, os, re, sys, tarfile, time, zipfile
import codecs, csv, datetime, hashlib, os, re, sys, tarfile, time, zipfile
if pyversion("3"): import configparser
else: import ConfigParser as configparser
gcounties_an = "2015_Gaz_counties_national.zip"
gcounties_fn = "2015_Gaz_counties_national.txt"
gcousubs_an = "2015_Gaz_cousubs_national.zip"
gcousubs_fn = "2015_Gaz_cousubs_national.txt"
gplace_an = "2015_Gaz_place_national.zip"
gplace_fn = "2015_Gaz_place_national.txt"
gzcta_an = "2015_Gaz_zcta_national.zip"
gzcta_fn = "2015_Gaz_zcta_national.txt"
for filename in os.listdir("."):
if re.match("bp[0-9][0-9][a-z][a-z][0-9][0-9].dbx$", filename):
if re.match("[0-9]{4}_Gaz_counties_national.zip$", filename):
gcounties_an = filename
gcounties_fn = filename[:-4] + ".txt"
elif re.match("[0-9]{4}_Gaz_cousubs_national.zip$", filename):
gcousubs_an = filename
gcousubs_fn = filename[:-4] + ".txt"
elif re.match("[0-9]{4}_Gaz_place_national.zip$", filename):
gplace_an = filename
gplace_fn = filename[:-4] + ".txt"
elif re.match("[0-9]{4}_Gaz_zcta_national.zip$", filename):
gzcta_an = filename
gzcta_fn = filename[:-4] + ".txt"
elif re.match("bp[0-9]{2}[a-z]{2}[0-9]{2}.dbx$", filename):
cpfzcf_fn = filename
break
nsdcccc_fn = "nsd_cccc.txt"
zcatalog_an = "zonecatalog.curr.tar"
metartbl_fn = "metar.tbl"
coopstn_fn = "coop-stations.txt"
ourairports_fn = "airports.csv"
overrides_fn = "overrides.conf"
overrideslog_fn = "overrides.log"
slist_fn = "slist"
@@ -1242,25 +1243,19 @@ def correlate():
%s
# generated by %s on %s from these public domain sources:
#
# http://www.census.gov/geo/maps-data/data/gazetteer2015.html
# https://www.census.gov/geographies/reference-files/time-series/geo/gazetteer-files.html
# %s %s %s
# %s %s %s
# %s %s %s
# %s %s %s
#
# http://www.weather.gov/geodata/catalog/wsom/html/cntyzone.htm
# https://www.weather.gov/gis/ZoneCounty/
# %s %s %s
#
# http://tgftp.nws.noaa.gov/data/nsd_cccc.txt
# https://tgftp.nws.noaa.gov/data/
# %s %s %s
#
# http://tgftp.nws.noaa.gov/data/zonecatalog.curr.tar
# %s %s %s
#
# http://www.nco.ncep.noaa.gov/pmb/codes/nwprod/dictionaries/metar.tbl
# %s %s %s
#
# http://www.ncdc.noaa.gov/homr/reports
# https://ourairports.com/data/
# %s %s %s
#
# ...and these manually-generated or hand-compiled adjustments:
@@ -1303,21 +1298,11 @@ def correlate():
datetime.datetime.fromtimestamp( os.path.getmtime(nsdcccc_fn) )
),
nsdcccc_fn,
hashlib.md5( open(zcatalog_an, "rb").read() ).hexdigest(),
hashlib.md5( open(ourairports_fn, "rb").read() ).hexdigest(),
datetime.date.isoformat(
datetime.datetime.fromtimestamp( os.path.getmtime(zcatalog_an) )
datetime.datetime.fromtimestamp( os.path.getmtime(ourairports_fn) )
),
zcatalog_an,
hashlib.md5( open(metartbl_fn, "rb").read() ).hexdigest(),
datetime.date.isoformat(
datetime.datetime.fromtimestamp( os.path.getmtime(metartbl_fn) )
),
metartbl_fn,
hashlib.md5( open(coopstn_fn, "rb").read() ).hexdigest(),
datetime.date.isoformat(
datetime.datetime.fromtimestamp( os.path.getmtime(coopstn_fn) )
),
coopstn_fn,
ourairports_fn,
hashlib.md5( open(overrides_fn, "rb").read() ).hexdigest(),
datetime.date.isoformat(
datetime.datetime.fromtimestamp( os.path.getmtime(overrides_fn) )
@@ -1344,9 +1329,9 @@ def correlate():
sys.stdout.flush()
count = 0
gcounties = zipfile.ZipFile(gcounties_an).open(gcounties_fn, "rU")
columns = gcounties.readline().decode("latin1").strip().split("\t")
columns = gcounties.readline().decode("utf-8").strip().split("\t")
for line in gcounties:
fields = line.decode("latin1").strip().split("\t")
fields = line.decode("utf-8").strip().split("\t")
f_geoid = fields[ columns.index("GEOID") ].strip()
f_name = fields[ columns.index("NAME") ].strip()
f_usps = fields[ columns.index("USPS") ].strip()
@@ -1367,9 +1352,9 @@ def correlate():
sys.stdout.flush()
count = 0
gcousubs = zipfile.ZipFile(gcousubs_an).open(gcousubs_fn, "rU")
columns = gcousubs.readline().decode("latin1").strip().split("\t")
columns = gcousubs.readline().decode("utf-8").strip().split("\t")
for line in gcousubs:
fields = line.decode("latin1").strip().split("\t")
fields = line.decode("utf-8").strip().split("\t")
f_geoid = fields[ columns.index("GEOID") ].strip()
f_name = fields[ columns.index("NAME") ].strip()
f_usps = fields[ columns.index("USPS") ].strip()
@@ -1390,9 +1375,9 @@ def correlate():
sys.stdout.flush()
count = 0
gplace = zipfile.ZipFile(gplace_an).open(gplace_fn, "rU")
columns = gplace.readline().decode("latin1").strip().split("\t")
columns = gplace.readline().decode("utf-8").strip().split("\t")
for line in gplace:
fields = line.decode("latin1").strip().split("\t")
fields = line.decode("utf-8").strip().split("\t")
f_geoid = fields[ columns.index("GEOID") ].strip()
f_name = fields[ columns.index("NAME") ].strip()
f_usps = fields[ columns.index("USPS") ].strip()
@@ -1412,51 +1397,22 @@ def correlate():
sys.stdout.write(message)
sys.stdout.flush()
count = 0
slist = codecs.open(slist_fn, "rU")
slist = codecs.open(slist_fn, "rU", "utf-8")
for line in slist:
icao = line.split("#")[0].strip()
if icao:
stations[icao] = {
"metar": "http://tgftp.nws.noaa.gov/data/observations/"\
"metar": "https://tgftp.nws.noaa.gov/data/observations/"\
+ "metar/decoded/%s.TXT" % icao.upper()
}
count += 1
slist.close()
print("done (%s lines)." % count)
message = "Reading %s..." % metartbl_fn
sys.stdout.write(message)
sys.stdout.flush()
count = 0
metartbl = codecs.open(metartbl_fn, "rU")
for line in metartbl:
icao = line[:4].strip().lower()
if icao in stations:
description = []
name = " ".join(
line[16:48].replace("_", " ").strip().title().split()
)
if name: description.append(name)
st = line[49:51].strip()
if st: description.append(st)
cn = line[52:54].strip()
if cn: description.append(cn)
if description:
stations[icao]["description"] = ", ".join(description)
lat = line[55:60].strip()
if lat:
lat = int(lat)/100.0
lon = line[61:67].strip()
if lon:
lon = int(lon)/100.0
stations[icao]["location"] = gecos( "%s,%s" % (lat, lon) )
count += 1
metartbl.close()
print("done (%s lines)." % count)
message = "Reading %s..." % nsdcccc_fn
sys.stdout.write(message)
sys.stdout.flush()
count = 0
nsdcccc = codecs.open(nsdcccc_fn, "rU", "latin1")
nsdcccc = codecs.open(nsdcccc_fn, "rU", "utf-8")
for line in nsdcccc:
line = str(line)
fields = line.split(";")
@@ -1481,44 +1437,49 @@ def correlate():
count += 1
nsdcccc.close()
print("done (%s lines)." % count)
message = "Reading %s..." % coopstn_fn
message = "Reading %s..." % ourairports_fn
sys.stdout.write(message)
sys.stdout.flush()
count = 0
coopstn = open(coopstn_fn)
for line in coopstn:
icao = line[33:37].strip().lower()
ourairports = open(ourairports_fn, "rU")
for row in csv.reader(ourairports):
icao = row[12].decode('utf-8').lower()
if icao in stations:
iata = line[22:26].strip().lower()
iata = row[13].decode('utf-8').lower()
if len(iata) == 3: airports[iata] = { "station": icao }
if "description" not in stations[icao]:
description = []
name = " ".join( line[99:129].strip().title().split() )
name = row[3].decode('utf-8')
if name: description.append(name)
st = line[59:61].strip()
if st: description.append(st)
country = " ".join( line[38:58].strip().title().split() )
if country: description.append(country)
municipality = row[10].decode('utf-8')
if municipality: description.append(municipality)
region = row[9].decode('utf-8')
country = row[8].decode('utf-8')
if region:
if "-" in region:
c,r = region.split("-", 1)
if c == country: region = r
description.append(region)
if country:
description.append(country)
if description:
stations[icao]["description"] = ", ".join(description)
if "location" not in stations[icao]:
lat = line[130:139].strip()
lat = row[4].decode('utf-8')
if lat:
lat = lat.replace(" ", "-")
lon = line[140:150].strip()
lon = row[5].decode('utf-8')
if lon:
lon = lon.replace(" ", "-")
stations[icao]["location"] = gecos(
"%s,%s" % (lat, lon)
)
count += 1
coopstn.close()
ourairports.close()
print("done (%s lines)." % count)
message = "Reading %s..." % zlist_fn
sys.stdout.write(message)
sys.stdout.flush()
count = 0
zlist = codecs.open(zlist_fn, "rU")
zlist = codecs.open(zlist_fn, "rU", "utf-8")
for line in zlist:
line = line.split("#")[0].strip()
if line:
@@ -1526,69 +1487,76 @@ def correlate():
count += 1
zlist.close()
print("done (%s lines)." % count)
message = "Reading %s:*..." % zcatalog_an
sys.stdout.write(message)
sys.stdout.flush()
count = 0
zcatalog = tarfile.open(zcatalog_an)
for entry in zcatalog.getmembers():
if entry.isfile():
fnmatch = re.match(
r"([a-z]+z[0-9]+)\.txt$",
os.path.basename(entry.name)
)
if fnmatch:
zone = fnmatch.group(1)
if zone in zones:
data = zcatalog.extractfile(entry).readlines()
description = data[0].decode("ascii").strip()
zones[zone]["description"] = description
for line in data[1:]:
line = line.decode("latin1").strip()
urimatch = re.match("/webdocs/pub/(.+):(.+) for ",
line)
if urimatch:
uritype = urimatch.group(2).lower().replace(" ","_")
zones[zone][uritype] = (
"http://tgftp.nws.noaa.gov/%s"
% urimatch.group(1))
count += 1
zcatalog.close()
print("done (%s files)." % count)
message = "Reading %s..." % cpfzcf_fn
sys.stdout.write(message)
sys.stdout.flush()
count = 0
cpfz = {}
cpfzcf = open(cpfzcf_fn)
cpfzcf = codecs.open(cpfzcf_fn, "rU", "utf-8")
for line in cpfzcf:
fields = line.strip().split("|")
if len(fields) == 11 \
and fields[0] and fields[1] and fields[9] and fields[10]:
zone = "z".join( fields[:2] ).lower()
if zone in zones:
zones[zone]["centroid"] = gecos( ",".join( fields[9:11] ) )
elif fields[6]:
state = fields[0]
description = fields[3]
county = fields[5]
if state:
zones[zone]["coastal_flood_statement"] = (
"https://tgftp.nws.noaa.gov/data/watches_warnings/"
"flood/coastal/%s/%s.txt" % (state.lower(), zone))
zones[zone]["flash_flood_statement"] = (
"https://tgftp.nws.noaa.gov/data/watches_warnings/"
"flash_flood/statement/%s/%s.txt"
% (state.lower(), zone))
zones[zone]["flash_flood_warning"] = (
"https://tgftp.nws.noaa.gov/data/watches_warnings/"
"flash_flood/warning/%s/%s.txt"
% (state.lower(), zone))
zones[zone]["flash_flood_watch"] = (
"https://tgftp.nws.noaa.gov/data/watches_warnings/"
"flash_flood/watch/%s/%s.txt" % (state.lower(), zone))
zones[zone]["flood_statement"] = (
"https://tgftp.nws.noaa.gov/data/watches_warnings/"
"flood/statement/%s/%s.txt" % (state.lower(), zone))
zones[zone]["flood_warning"] = (
"https://tgftp.nws.noaa.gov/data/watches_warnings/"
"flood/warning/%s/%s.txt" % (state.lower(), zone))
zones[zone]["severe_thunderstorm_warning"] = (
"https://tgftp.nws.noaa.gov/data/watches_warnings/"
"thunderstorm/%s/%s.txt" % (state.lower(), zone))
zones[zone]["severe_weather_statement"] = (
"https://tgftp.nws.noaa.gov/data/watches_warnings/"
"severe_weather_stmt/%s/%s.txt"
% (state.lower(), zone))
zones[zone]["short_term_forecast"] = (
"https://tgftp.nws.noaa.gov/data/forecasts/nowcast/"
"%s/%s.txt" % (state.lower(), zone))
zones[zone]["special_weather_statement"] = (
"https://tgftp.nws.noaa.gov/data/watches_warnings/"
"special_weather_stmt/%s/%s.txt"
% (state.lower(), zone))
zones[zone]["state_forecast"] = (
"https://tgftp.nws.noaa.gov/data/forecasts/state/"
"%s/%s.txt" % (state.lower(), zone))
zones[zone]["urgent_weather_message"] = (
"https://tgftp.nws.noaa.gov/data/watches_warnings/"
"non_precip/%s/%s.txt" % (state.lower(), zone))
zones[zone]["zone_forecast"] = (
"https://tgftp.nws.noaa.gov/data/forecasts/zone/"
"%s/%s.txt" % (state.lower(), zone))
description = fields[3].strip()
fips = "fips%s"%fields[6]
possible = [
"%s, %s" % (county, state),
"%s County, %s" % (county, state),
]
if description.endswith(" Counties"):
description = description[:-9]
for addition in description.split(" and "):
possible.append( "%s, %s" % (addition, state) )
possible.append( "%s County, %s" % (addition, state) )
if fips in places and "centroid" in places[fips]:
for candidate in zones:
if "centroid" not in zones[candidate] and \
"description" in zones[candidate] and \
zones[candidate]["description"] in possible:
zones[candidate]["centroid"] = \
places[fips]["centroid"]
county = fields[5]
if county:
if description.endswith(county):
description += " County"
else:
description += ", %s County" % county
description += ", %s, US" % state
zones[zone]["description"] = description
zones[zone]["centroid"] = gecos( ",".join( fields[9:11] ) )
if fips in places and not zones[zone]["centroid"]:
zones[zone]["centroid"] = places[fips]["centroid"]
count += 1
cpfzcf.close()
print("done (%s lines)." % count)
@@ -1597,9 +1565,9 @@ def correlate():
sys.stdout.flush()
count = 0
gzcta = zipfile.ZipFile(gzcta_an).open(gzcta_fn, "rU")
columns = gzcta.readline().decode("latin1").strip().split("\t")
columns = gzcta.readline().decode("utf-8").strip().split("\t")
for line in gzcta:
fields = line.decode("latin1").strip().split("\t")
fields = line.decode("utf-8").strip().split("\t")
f_geoid = fields[ columns.index("GEOID") ].strip()
f_intptlat = fields[ columns.index("INTPTLAT") ].strip()
f_intptlong = fields[ columns.index("INTPTLONG") ].strip()
@@ -1967,7 +1935,7 @@ def correlate():
count = 0
if os.path.exists(stations_fn):
os.rename(stations_fn, "%s_old"%stations_fn)
stations_fd = codecs.open(stations_fn, "w", "utf8")
stations_fd = codecs.open(stations_fn, "w", "utf-8")
stations_fd.write(header)
for station in sorted( stations.keys() ):
stations_fd.write("\n\n[%s]" % station)

66682
zctas

File diff suppressed because it is too large Load Diff

358
zlist
View File

@@ -1,15 +1,15 @@
# Copyright (c) 2006-2016 Jeremy Stanley <fungi@yuggoth.org>. Permission to
# Copyright (c) 2006-2020 Jeremy Stanley <fungi@yuggoth.org>. Permission to
# use, copy, modify, and distribute this software is granted under terms
# provided in the LICENSE file distributed with this software.
#
# generated on 2016-10-04 by running:
# generated on 2020-05-24 by running:
#
# for area in $(
# wget -qO- http://tgftp.nws.noaa.gov/data/forecasts/zone/ \
# wget -qO- https://tgftp.nws.noaa.gov/data/forecasts/zone/ \
# | grep '<a href="[^/]*/">' \
# | sed 's,.*<a href="\([^/]*\)/">.*,\1,'
# ) ; do
# wget -qO- http://tgftp.nws.noaa.gov/data/forecasts/zone/$area \
# wget -qO- https://tgftp.nws.noaa.gov/data/forecasts/zone/$area \
# | grep '<a href="[^\.]*\.txt">.*-'$( date +%b-%Y )' ' \
# | sed 's,.*<a href="\([^\.]*\)\.txt">.*,\1,'
# sleep 1
@@ -239,15 +239,6 @@ azz015
azz016
azz017
azz018
azz020
azz021
azz022
azz023
azz024
azz025
azz026
azz027
azz028
azz036
azz037
azz038
@@ -268,6 +259,40 @@ azz512
azz513
azz514
azz515
azz530
azz531
azz532
azz533
azz534
azz535
azz536
azz537
azz538
azz539
azz540
azz541
azz542
azz543
azz544
azz545
azz546
azz547
azz548
azz549
azz550
azz551
azz552
azz553
azz554
azz555
azz556
azz557
azz558
azz559
azz560
azz561
azz562
azz563
caz006
caz013
caz014
@@ -276,10 +301,6 @@ caz016
caz017
caz018
caz019
caz030
caz031
caz032
caz033
caz034
caz035
caz036
@@ -307,7 +328,6 @@ caz060
caz061
caz062
caz063
caz064
caz065
caz066
caz067
@@ -325,17 +345,6 @@ caz084
caz085
caz087
caz088
caz089
caz090
caz091
caz092
caz093
caz094
caz095
caz096
caz097
caz098
caz099
caz101
caz102
caz103
@@ -349,6 +358,29 @@ caz110
caz111
caz112
caz113
caz114
caz115
caz179
caz180
caz181
caz182
caz183
caz184
caz185
caz186
caz187
caz188
caz189
caz190
caz191
caz192
caz193
caz194
caz195
caz196
caz197
caz198
caz199
caz505
caz506
caz507
@@ -379,6 +411,17 @@ caz549
caz550
caz552
caz554
caz560
caz561
caz562
caz563
caz564
caz565
caz566
caz567
caz568
caz569
caz570
coz001
coz002
coz003
@@ -497,7 +540,6 @@ flz018
flz019
flz020
flz021
flz022
flz023
flz024
flz025
@@ -511,10 +553,8 @@ flz032
flz033
flz034
flz035
flz036
flz037
flz038
flz040
flz041
flz043
flz044
@@ -550,12 +590,17 @@ flz112
flz114
flz115
flz118
flz122
flz124
flz125
flz127
flz128
flz133
flz134
flz136
flz138
flz139
flz140
flz141
flz142
flz144
@@ -577,7 +622,10 @@ flz203
flz204
flz205
flz206
flz222
flz236
flz239
flz240
flz242
flz248
flz249
@@ -586,6 +634,7 @@ flz255
flz260
flz262
flz265
flz340
gaz001
gaz002
gaz003
@@ -735,7 +784,6 @@ gaz146
gaz147
gaz148
gaz149
gaz150
gaz151
gaz152
gaz153
@@ -749,13 +797,12 @@ gaz160
gaz161
gaz162
gaz163
gaz164
gaz165
gaz166
guz001
guz002
guz003
guz004
gaz250
gaz264
gaz350
gaz364
hiz001
hiz002
hiz003
@@ -899,23 +946,37 @@ idz013
idz014
idz015
idz016
idz017
idz018
idz019
idz020
idz021
idz022
idz023
idz024
idz025
idz026
idz027
idz028
idz029
idz030
idz031
idz032
idz033
idz051
idz052
idz053
idz054
idz055
idz056
idz057
idz058
idz059
idz060
idz061
idz062
idz063
idz064
idz065
idz066
idz067
idz068
idz069
idz070
idz071
idz072
idz073
idz074
idz075
ilz001
ilz002
ilz003
@@ -929,7 +990,6 @@ ilz010
ilz011
ilz012
ilz013
ilz014
ilz015
ilz016
ilz017
@@ -937,7 +997,6 @@ ilz018
ilz019
ilz020
ilz021
ilz022
ilz023
ilz024
ilz025
@@ -1018,6 +1077,12 @@ ilz099
ilz100
ilz101
ilz102
ilz103
ilz104
ilz105
ilz106
ilz107
ilz108
inz001
inz002
inz003
@@ -1494,6 +1559,7 @@ mez029
mez030
mez031
mez032
mez033
miz001
miz002
miz003
@@ -1501,18 +1567,15 @@ miz004
miz005
miz006
miz007
miz008
miz009
miz010
miz011
miz012
miz013
miz014
miz015
miz016
miz017
miz018
miz019
miz020
miz021
miz022
@@ -1579,6 +1642,14 @@ miz082
miz083
miz084
miz085
miz086
miz087
miz088
miz095
miz096
miz097
miz098
miz099
mnz001
mnz002
mnz003
@@ -1901,20 +1972,15 @@ mtz024
mtz025
mtz026
mtz027
mtz028
mtz029
mtz030
mtz031
mtz032
mtz033
mtz034
mtz035
mtz036
mtz037
mtz038
mtz039
mtz040
mtz041
mtz042
mtz043
mtz044
@@ -1942,6 +2008,16 @@ mtz065
mtz066
mtz067
mtz068
mtz138
mtz139
mtz141
mtz169
mtz170
mtz171
mtz172
mtz173
mtz228
mtz235
ncz001
ncz002
ncz003
@@ -2029,21 +2105,25 @@ ncz089
ncz090
ncz091
ncz092
ncz093
ncz094
ncz095
ncz096
ncz098
ncz099
ncz102
ncz103
ncz104
ncz105
ncz106
ncz107
ncz108
ncz109
ncz110
ncz193
ncz194
ncz195
ncz196
ncz198
ncz199
ncz203
ncz204
ncz205
ncz501
ncz502
ncz503
@@ -2253,6 +2333,47 @@ nmz028
nmz029
nmz033
nmz034
nmz201
nmz202
nmz203
nmz204
nmz205
nmz206
nmz207
nmz208
nmz209
nmz210
nmz211
nmz212
nmz213
nmz214
nmz215
nmz216
nmz217
nmz218
nmz219
nmz220
nmz221
nmz222
nmz223
nmz224
nmz225
nmz226
nmz227
nmz228
nmz229
nmz230
nmz231
nmz232
nmz233
nmz234
nmz235
nmz236
nmz237
nmz238
nmz239
nmz240
nmz241
nmz401
nmz402
nmz403
@@ -2270,46 +2391,6 @@ nmz414
nmz415
nmz416
nmz417
nmz501
nmz502
nmz503
nmz504
nmz505
nmz506
nmz507
nmz508
nmz509
nmz510
nmz511
nmz512
nmz513
nmz514
nmz515
nmz516
nmz517
nmz518
nmz519
nmz520
nmz521
nmz522
nmz523
nmz524
nmz525
nmz526
nmz527
nmz528
nmz529
nmz530
nmz531
nmz532
nmz533
nmz534
nmz535
nmz536
nmz537
nmz538
nmz539
nmz540
nvz001
nvz002
nvz003
@@ -2744,7 +2825,6 @@ scz011
scz012
scz013
scz014
scz015
scz016
scz017
scz018
@@ -2780,10 +2860,13 @@ scz049
scz050
scz051
scz052
scz053
scz054
scz055
scz056
scz058
scz059
scz115
scz116
sdz001
sdz002
sdz003
@@ -3014,8 +3097,6 @@ txz051
txz052
txz053
txz054
txz057
txz058
txz059
txz060
txz061
@@ -3031,14 +3112,10 @@ txz070
txz071
txz072
txz073
txz074
txz075
txz076
txz077
txz078
txz079
txz080
txz081
txz082
txz083
txz084
@@ -3213,11 +3290,38 @@ txz254
txz255
txz256
txz257
txz258
txz259
txz260
txz261
txz262
txz270
txz271
txz272
txz273
txz274
txz275
txz276
txz277
txz278
txz279
txz280
txz281
txz282
txz300
txz313
txz317
txz335
txz336
txz337
txz338
txz342
txz343
txz344
txz345
txz346
txz347
txz351
txz353
txz418
txz419
txz420
@@ -3225,6 +3329,12 @@ txz421
txz422
txz423
txz424
txz436
txz437
txz438
txz442
txz443
txz447
utz001
utz002
utz003
@@ -3542,17 +3652,11 @@ wvz031
wvz032
wvz033
wvz034
wvz035
wvz036
wvz037
wvz038
wvz039
wvz040
wvz042
wvz043
wvz044
wvz046
wvz047
wvz050
wvz051
wvz052
@@ -3572,6 +3676,18 @@ wvz511
wvz512
wvz513
wvz514
wvz515
wvz516
wvz517
wvz518
wvz519
wvz520
wvz521
wvz522
wvz523
wvz524
wvz525
wvz526
wyz001
wyz002
wyz003
@@ -3608,8 +3724,6 @@ wyz056
wyz057
wyz058
wyz071
wyz098
wyz099
wyz101
wyz102
wyz103
@@ -3629,3 +3743,5 @@ wyz116
wyz117
wyz118
wyz119
wyz198
wyz199

132340
zones

File diff suppressed because it is too large Load Diff