Some NWS products, such as forecasts and other alerts, have
expiration times marked relative to the issuing authority's local
time zone. In preparation for being able to calculate expirations
accurately relative to the user's local time, embed IANA TZDB
compliant identifiers in all WX weather zone definitions, translated
from their NWS time zone codes.
Correlation files are all updated because of amending overrides
accordingly.
NWS switched to using FIPS based county designations for flood
warnings around 2016, so fix the URLs for them. Also flash flood
statements and warnings (but nor watches which, for some reason,
still use WX weather zones). Severe weather statements too (but not
special weather statements nor urgent weather messages). Oh, and
thunderstorms.
Add a new tornado alert type with relevant URLs, and fix
configuration examples which reference the older tornado_warning
field which hasn't been available for years.
While at it, remove the separate flood statement alert which seems
to be entirely unused by NWS.
Since NWS alerts and forecasts list their expiration times relative
to the issuing office's local timezone, filtering for expired
documents relative to the users timezone can lead to them being
filtered early when they're not both coincidentally the same.
Introduce a one day (86400 second) offset buffer as a simple
workaround for now, since the user's and issuing authority's
timezones shouldn't ever differ by more than that. This has a
downside of showing forecasts or alerts which have expired and not
been replaced, but that was possible already if timezones differed
in the other direction, and is preferable to the alternative.
The NWS DBX schema for WX weather zones does include a field for a
timezone code, so a future change may introduce more accurate
calculations in order to identify the relative offset between the
user and issuer, but this will require extending our own zones
format to add a new value for it.
Switch to the 2023 US Census Bureau data, March 2024 NWS WX zones,
latest OurAirports open data set, refresh active forecast and
station lists, and clean up obsolete overrides. Regenerate all
correlation sets based on these updated sources.
Clean up the overrides.conf a bit and regenerate correlations now
that the NWS has retired some WX weather zones. This addresses some
incorrect correlations to now dead forecasts in the Los Angeles
area (thanks to Chime Hart for reporting), as well as likely others.
Switch to the 2022 US Census Bureau data, March 2023 NWS WX zones,
latest OurAirports open data set, and refreshed active forecast and
station lists. Regenerate all correlation sets based on these
updated sources.
This patch was helpfully submitted by Bas Couwenberg to drop use of
the universal newline flag, since Python 3.11 no longer supports it.
It probably breaks the ability to build new correlation files under
Python 2.7 and earlier, but since it shouldn't affect operation of
the utility with prebuilt correlations (the way it's typically
used), this isn't yet considered to drop Python 2.7 support
altogether.
The correlate() function stopped needing tarfile a couple of years
ago (version 2.4), but it was overlooked that the script continued
to unnecessarily import it. Clean this up.
Apparently, Python on Windows defaults to assuming CP1252 encoding
unless otherwise specified, as opposed to the UTF-8 assumption made
on POSIX platforms. Since our configuration and data files are
expected to always use UTF-8 encoding, be clear in the
ConfigParser.read() calls about that. We only do this under Python
3.x, as that method doesn't have an encoding parameter in 2.7.
Thanks to Lance Bermudez for reporting this.
Just a basic correlation update based on more recent active METAR
station and WX zone lists. Also update the copyright year for files
which have been edited so far in 2021 as well as in the LICENSE
file.
The selections proxy class, which mashes together command-line
arguments and configuration options, contained a longstanding and
fatal flaw with its handling of boolean values. In particular,
falsey values were consistently treated as truthy due to naively
recasting str to bool (which will always yield True unless empty).
This went unnoticed for so long because the majority of these
settings default to False, meaning the only reason most users had to
set them was to override them to True.
Many thanks to Jordan Russell for bringing this bug to my attention,
and for supplying an initial patch on which this fix is heavily
based.
Co-Authored-By: Jordan Russell
Perform a fresh build of data sets from current sources, and add a
few additional overrides for previously unknown stations. Also
update from 2019 to 2021 US Census data, from March 2020 to
September 2021 CountyZone maps, and from 2020-08-29 to 2021-08-29
airport IDs and active METAR stations/WX zones.
Julien Palard pointed out that the way URLError exceptions were
being manually cobbled into the stderr stream wasn't quite working
(thanks!), but it was also unnecessarily complicated for reasons I
don't recall now. Rip most of it out and just go with a basic
catch/error/re-raise there instead.
As a more complete fix and future-proofing for the earlier mismatch
between default_atypes and the alert URLs generated for WX zones
during correlation, stop aborting and simply add a warning if a
requested alert type has no corresponding URL.
Kevin Monceaux reported a regression with the 2.4 release. Running
with the -a/--alert option and no limited --atypes or atypes
override in weatherrc resulted in a message about undefined URLs
and no normal output. This problem crept in when hard-coding alert
types in the correlator after ditching the woefully unmaintained
zonecatalog.curr.tar data source (commit 8a37edd).
Update default_atypes so that it covers all relevant non-forecast
URLs the correlate routine embeds.
While auditing Debian's packages, Chris Lamb reported[*] that
weather-util's correlation set generation is not reproducible
because it embeds timestamps without a means to override them and
also varies by system timezone. Allow SOURCE_DATE_EPOCH from the
calling environment and assume UTC rather than relying on locale
settings when no timezones are specified.
[*] https://bugs.debian.org/964721
Update a bunch of the parsing for various correlation source files
to work in both Python 2.7 and 3.5+, mostly where str vs bytes and
UTF-8 encoding/decoding are concerned. This can be cleaned up
significantly once support for 2.7 is finally dropped.
Add a copyright header to the .gitignore file with start and end
years determined from its commit history. Add copyright headers for
the current year to overrides.log and qa.log, and also add
functionality to correlate() which adds these headers from now on.
Update the copyright year on overrides.conf, which was missed in
8a37edd and later commits. All files tracked in this repository now
declare a copyright and refer to the main LICENSE file for licensing
terms.
Solve a SyntaxWarning under Python 3.8 and later for use of the "is"
identity operator when comparing literals, by replacing with the
"==" equality operator.
When mangling URLs of fetched data to store in the local cache, only
split on the first colon so that URLs with port numbers in them are
properly differentiated. Previously, all URLs for the same domain
name landed in a single file if a port number was included, causing
incorrect results to be returned from the cache.
Fix a cache corruption issue by using a new "cached" field to hold
the timestamp for cached correlation search results. Previously the
"description" field was being overloaded, but this could cause the
cache to no longer load because of duplicate fields.
Python 2.7 is likely the only Python 2 anyone is using any longer
(even that's well past EOL upstream now), and reasonably recent
versions of 2.7 it need the same decode hack as Python 3 anyway when
dealing with some retrieved content. Just get rid of the version
detection and do it under any version.
Thanks to Bill Agee for suggesting the Hong Kong Observatory's
weather forecast page. A custom filter is implemented to strip the
forecast text from the HTML page in which it is embedded (if anyone
finds a plaintext version published at an alternate URL, let me know
and I'll rip out the extra routine).
Override a number of active weather stations where searches of
various online sites return names and locations for them which are
not provided by the included sources.
Remove the stale metar.tbl and zonecatalog.curr.tar, which the USA
NWS hasn't been updating for many years, and add the public domain
airports.csv file from the amazing ourairports.com community. Also
update to latest (2019) USA Census Bureau location data, March 2020
WX zone information, cooperative sites list from 2018 (latest), and
regenerated active station and zone lists. Loss of the zonecatalog
necessitates directly applying various forecast and alert URL
patterns, though some which appeared unused by NWS for many years
were not included.
Clear out all old overrides, since the vast majority are obsoleted
by refreshed data, and build fresh correlation sets from the above
sources. Basically all sites have switched from HTTP to HTTPS, so
update URLs for this too.
One piecemeal use of the retired weather.noaa.gov/pub URL was missed
in the correlate() function, causing it to be reintroduced for
zone-based reports (such as forecasts) in a subsequent correlation
dataset update. Correct the invalid URLs in the zones file, and
update the correlation routine to embed the correct and working
tgftp.nws.noaa.gov hostname instead.
* overrides.conf: Latest source data corrections from
script-assisted research.
These remaining files are generated data. Normally they're not
something I feel good about committing into version control, but in
this case it allows for logging and tracking deltas in the data over
time...
* airports: Removed 18 airports corresponding to nonexistent
stations.
* stations: Removed 326 stations with no recent conditions, added
429.
* zones: Removed 45 zones with no recent forecasts, added 104.
* places, zctas: Based on latest Census Bureau data corrections,
updated with new correlations.
* overrides.log: Record of correlation set build.
* slist, zlist: State of active stations and weather zones at the
time of generation.
* overrides.conf: Latest source data corrections from
script-assisted research.
These remaining files are generated data. Normally they're not
something I feel good about committing into version control, but in
this case it allows for logging and tracking deltas in the data over
time...
* airports: Removed 527 airports corresponding to nonexistent
stations.
* stations: Removed 176 stations with no recent conditions, added
196.
* zones: Removed 5 zones with no recent forecasts.
* places, zctas: Based on latest Census Bureau data corrections,
updated with new correlations.
* overrides.log: Record of correlation set build.
* slist, zlist: State of active stations and weather zones at the
time of generation.
* INSTALL: Add a new section documenting the way in which newer
correlation data sets can be rebuilt and substituted for officially
distributed copies.
