Skip to content

Navigation Menu

Sign in
Appearance settings

Search code, repositories, users, issues, pull requests...

Provide feedback

We read every piece of feedback, and take your input very seriously.

Saved searches

Use saved searches to filter your results more quickly
Sign up
Appearance settings

Commit ba9653a

Browse files
clamprngehrsitzLairdStreakandersoncarlosfs
authored
Version 1.7.0 (#184)
* Change numpy.NaN to numpy.nan for compatibility with numpy>2.0.0 (#160) * Version 1.7.0: Initial commit * Adds developer quality of life dev enhancements (#138) * Adds developer quality of life dev enhancements 1: Adds a make file for running tasks 2: Add make for black formatting and pylint + flake8 3: Create a requirements_dev file for dev setup 4: Use pip-tools to generate appropriate requirements file. TAG: #HACKTOBERFEST2023 * Update: Update python file formatting via black * Update: Fixup some flake8 and pylint issues Added Ignore for E203 space before : Added Ignore for W503 newline before binary operator * Update: Update README and add diagrams Update to readme with developer instructions Adds Class and package diagrams. [pyreverse] * Adding proxy configuration (#143) * adding proxy configuration * reshaping the code * adding imports * reshaping the code * adding support for proxies * renaming variables * decompressing the response * renaming variable * renaming variable * fixing errors --------- Co-authored-by: Christian Lamprecht <christian.lamprecht@aol.de> * initial commit for hourly ts * feat: rename diverging columns * feat: virtual columns * feat: enable daily/monthly interfaces for new data access * feat: retain normals behavior * fix: broken inventory mask * fix: series normalization * chore: update requirements * fix: formatting * fix: linting * fix: ignore import errors when linting * fix: install pytest * fix: filter cols before interpolation * fix: pass fill_value * fix: convert to numpy float64 for interpolation * feat: cleanup * feat: update README * feat: review * fix: formatting * feat: add default start & end date for daily data * fix: formatting * fix: use correct freq * feat: order cols, use joined flags * fix: formatting * fix: silence future warnings * fix: formatting --------- Co-authored-by: ngehrsitz <45375059+ngehrsitz@users.noreply.github.com> Co-authored-by: Laird Streak <laird.streak@bentley.com> Co-authored-by: Anderson Carlos Ferreira da Silva <andersoncarlosfs@outlook.com>
1 parent 3556cca commit ba9653a

File tree

26 files changed

+456
-321
lines changed

26 files changed

+456
-321
lines changed

‎.pylintrc‎

Lines changed: 3 additions & 7 deletions
Original file line numberDiff line numberDiff line change
@@ -1,16 +1,12 @@
11
[MESSAGES CONTROL]
22

3-
disable=print-statement,
4-
singleton-comparison,
5-
no-member,
3+
disable=no-member,
64
too-few-public-methods,
75
protected-access,
8-
inconsistent-return-statements,
96
import-outside-toplevel,
107
duplicate-code,
11-
import-error,
12-
nan-comparison,
13-
consider-using-set-comprehension
8+
too-many-positional-arguments,
9+
import-error
1410

1511
[BASIC]
1612

‎README.md‎

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -2,7 +2,7 @@
22

33
The Meteostat Python library provides a simple API for accessing open weather and climate data. The historical observations and statistics are collected by [Meteostat](https://meteostat.net) from different public interfaces, most of which are governmental.
44

5-
Among the data sources are national weather services like the National Oceanic and Atmospheric Administration (NOAA) and Germany's national meteorological service (DWD).
5+
Among the data sources are national weather services like the National Oceanic and Atmospheric Administration (NOAA) and Germany's national weather service (DWD).
66

77
Are you looking for a **hosted solution**? Try our [JSON API](https://rapidapi.com/meteostat/api/meteostat/).
88

‎examples/daily/compare_aggregate.py‎

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -28,7 +28,7 @@
2828
data = Daily(stations, start, end)
2929

3030
# Aggregate annually
31-
data = data.aggregate(freq="1Y").fetch()
31+
data = data.aggregate(freq="1YE").fetch()
3232

3333
# Plot chart
3434
fig, ax = plt.subplots(figsize=(8, 6))

‎examples/monthly/aggregate.py‎

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -19,7 +19,7 @@
1919
# Get monthly data
2020
# Then, aggregate annually
2121
data = Monthly("72202", start, end)
22-
data = data.normalize().aggregate(freq="1Y").fetch()
22+
data = data.normalize().aggregate(freq="1YE").fetch()
2323

2424
# Plot chart
2525
data.plot(y="tavg")

‎meteostat/__init__.py‎

Lines changed: 12 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -12,7 +12,7 @@
1212
"""
1313

1414
__appname__ = "meteostat"
15-
__version__ = "1.6.8"
15+
__version__ = "1.7.0"
1616

1717
from .interface.base import Base
1818
from .interface.timeseries import TimeSeries
@@ -22,3 +22,14 @@
2222
from .interface.daily import Daily
2323
from .interface.monthly import Monthly
2424
from .interface.normals import Normals
25+
26+
__all__ = [
27+
"Base",
28+
"TimeSeries",
29+
"Stations",
30+
"Point",
31+
"Hourly",
32+
"Daily",
33+
"Monthly",
34+
"Normals",
35+
]

‎meteostat/core/cache.py‎

Lines changed: 0 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -53,7 +53,6 @@ def clear_cache(cls, max_age: int = None) -> None:
5353
"""
5454

5555
if os.path.exists(cls.cache_dir + os.sep + cls.cache_subdir):
56-
5756
# Set max_age
5857
if max_age is None:
5958
max_age = cls.max_age
@@ -63,7 +62,6 @@ def clear_cache(cls, max_age: int = None) -> None:
6362

6463
# Go through all files
6564
for file in os.listdir(cls.cache_dir + os.sep + cls.cache_subdir):
66-
6765
# Get full path
6866
path = os.path.join(cls.cache_dir + os.sep + cls.cache_subdir, file)
6967

‎meteostat/core/loader.py‎

Lines changed: 26 additions & 28 deletions
Original file line numberDiff line numberDiff line change
@@ -8,16 +8,19 @@
88
The code is licensed under the MIT license.
99
"""
1010

11+
from io import BytesIO
12+
from gzip import GzipFile
13+
from urllib.request import Request, ProxyHandler, build_opener
1114
from urllib.error import HTTPError
1215
from multiprocessing import Pool
1316
from multiprocessing.pool import ThreadPool
14-
from typing import Callable, Union
17+
from typing import Callable, List, Optional
1518
import pandas as pd
1619
from meteostat.core.warn import warn
1720

1821

1922
def processing_handler(
20-
datasets: list, load: Callable[[dict], None], cores: int, threads: int
23+
datasets: List, load: Callable[[dict], None], cores: int, threads: int
2124
) -> None:
2225
"""
2326
Load multiple datasets (simultaneously)
@@ -28,10 +31,8 @@ def processing_handler(
2831

2932
# Multi-core processing
3033
if cores > 1 and len(datasets) > 1:
31-
3234
# Create process pool
3335
with Pool(cores) as pool:
34-
3536
# Process datasets in pool
3637
output = pool.starmap(load, datasets)
3738

@@ -41,10 +42,8 @@ def processing_handler(
4142

4243
# Multi-thread processing
4344
elif threads > 1 and len(datasets) > 1:
44-
4545
# Create process pool
4646
with ThreadPool(threads) as pool:
47-
4847
# Process datasets in pool
4948
output = pool.starmap(load, datasets)
5049

@@ -54,49 +53,48 @@ def processing_handler(
5453

5554
# Single-thread processing
5655
else:
57-
5856
for dataset in datasets:
5957
output.append(load(*dataset))
6058

6159
# Remove empty DataFrames
62-
filtered = list(filter(lambda df: df.index.size>0, output))
60+
filtered = list(filter(lambda df: notdf.empty, output))
6361

6462
return pd.concat(filtered) if len(filtered) > 0 else output[0]
6563

6664

6765
def load_handler(
6866
endpoint: str,
6967
path: str,
70-
columns: list,
71-
types: Union[dict, None],
72-
parse_dates: list,
73-
coerce_dates: bool = False,
68+
proxy: Optional[str] = None,
69+
names: Optional[List] = None,
70+
dtype: Optional[dict] = None,
71+
parse_dates: Optional[List] = None,
72+
default_df: Optional[pd.DataFrame] = None,
7473
) -> pd.DataFrame:
7574
"""
7675
Load a single CSV file into a DataFrame
7776
"""
7877

7978
try:
79+
handlers = []
80+
81+
# Set a proxy
82+
if proxy:
83+
handlers.append(ProxyHandler({"http": proxy, "https": proxy}))
8084

8185
# Read CSV file from Meteostat endpoint
82-
df = pd.read_csv(
83-
endpoint + path,
84-
compression="gzip",
85-
names=columns,
86-
dtype=types,
87-
parse_dates=parse_dates,
88-
)
89-
90-
# Force datetime conversion
91-
if coerce_dates:
92-
df.iloc[:, parse_dates] = df.iloc[:, parse_dates].apply(
93-
pd.to_datetime, errors="coerce"
94-
)
86+
with build_opener(*handlers).open(Request(endpoint + path)) as response:
87+
# Decompress the content
88+
with GzipFile(fileobj=BytesIO(response.read()), mode="rb") as file:
89+
df = pd.read_csv(
90+
file,
91+
names=names,
92+
dtype=dtype,
93+
parse_dates=parse_dates,
94+
)
9595

9696
except (FileNotFoundError, HTTPError):
97-
98-
# Create empty DataFrane
99-
df = pd.DataFrame(columns=[*types])
97+
df = default_df if default_df is not None else pd.DataFrame(columns=names)
10098

10199
# Display warning
102100
warn(f"Cannot load {path} from {endpoint}")

‎meteostat/core/warn.py‎

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -16,7 +16,7 @@ def _format(message, category, _filename, _lineno, _line=None) -> str:
1616
Print warning on a single line
1717
"""
1818

19-
return "%s: %s\n"% (category.__name__, message)
19+
return f"{category.__name__}: {message}\n"
2020

2121

2222
# Set warning format

‎meteostat/interface/base.py‎

Lines changed: 10 additions & 7 deletions
Original file line numberDiff line numberDiff line change
@@ -9,28 +9,31 @@
99
"""
1010

1111
import os
12+
from typing import Optional
1213

1314

1415
class Base:
15-
1616
"""
1717
Base class that provides features which are used across the package
1818
"""
1919

2020
# Base URL of the Meteostat bulk data interface
21-
endpoint: str = "https://bulk.meteostat.net/v2/"
21+
endpoint = "https://bulk.meteostat.net/v2/"
22+
23+
# Proxy URL for the Meteostat (bulk) data interface
24+
proxy: Optional[str] = None
2225

2326
# Location of the cache directory
24-
cache_dir: str = os.path.expanduser("~") + os.sep + ".meteostat" + os.sep + "cache"
27+
cache_dir = os.path.expanduser("~") + os.sep + ".meteostat" + os.sep + "cache"
2528

2629
# Auto clean cache directories?
27-
autoclean: bool = True
30+
autoclean = True
2831

2932
# Maximum age of a cached file in seconds
30-
max_age: int = 24 * 60 * 60
33+
max_age = 24 * 60 * 60
3134

3235
# Number of processes used for processing files
33-
processes: int = 1
36+
processes = 1
3437

3538
# Number of threads used for processing files
36-
threads: int = 1
39+
threads = 1

‎meteostat/interface/daily.py‎

Lines changed: 44 additions & 31 deletions
Original file line numberDiff line numberDiff line change
@@ -8,7 +8,7 @@
88
The code is licensed under the MIT license.
99
"""
1010

11-
from datetime import datetime
11+
from datetime import datetime, timedelta
1212
from typing import Union
1313
import pandas as pd
1414
from meteostat.enumerations.granularity import Granularity
@@ -18,61 +18,68 @@
1818

1919

2020
class Daily(TimeSeries):
21-
2221
"""
2322
Retrieve daily weather observations for one or multiple weather stations or
2423
a single geographical point
2524
"""
2625

2726
# The cache subdirectory
28-
cache_subdir: str = "daily"
27+
cache_subdir = "daily"
2928

3029
# Granularity
3130
granularity = Granularity.DAILY
3231

32+
# Download data as annual chunks
33+
# This cannot be changed and is only kept for backward compatibility
34+
chunked = True
35+
3336
# Default frequency
34-
_freq: str = "1D"
37+
_freq = "1D"
38+
39+
# Source mappings
40+
_source_mappings = {
41+
"dwd_daily": "A",
42+
"eccc_daily": "A",
43+
"ghcnd": "B",
44+
"dwd_hourly": "C",
45+
"eccc_hourly": "C",
46+
"isd_lite": "D",
47+
"synop": "E",
48+
"dwd_poi": "E",
49+
"metar": "F",
50+
"model": "G",
51+
"dwd_mosmix": "G",
52+
"metno_forecast": "G",
53+
}
3554

3655
# Flag which represents model data
3756
_model_flag = "G"
3857

3958
# Columns
40-
_columns: list = [
41-
"date",
42-
"tavg",
59+
_columns = [
60+
"year",
61+
"month",
62+
"day",
63+
{"tavg": "temp"},
4364
"tmin",
4465
"tmax",
4566
"prcp",
46-
"snow",
47-
"wdir",
67+
{"snow": "snwd"},
68+
{"wdir": None},
4869
"wspd",
4970
"wpgt",
5071
"pres",
5172
"tsun",
5273
]
5374

5475
# Index of first meteorological column
55-
_first_met_col = 1
56-
57-
# Data types
58-
_types: dict = {
59-
"tavg": "float64",
60-
"tmin": "float64",
61-
"tmax": "float64",
62-
"prcp": "float64",
63-
"snow": "float64",
64-
"wdir": "float64",
65-
"wspd": "float64",
66-
"wpgt": "float64",
67-
"pres": "float64",
68-
"tsun": "float64",
69-
}
76+
_first_met_col = 3
7077

7178
# Columns for date parsing
72-
_parse_dates: dict= {"time": [0]}
79+
_parse_dates= ["year", "month", "day"]
7380

7481
# Default aggregation functions
75-
aggregations: dict = {
82+
aggregations = {
7683
"tavg": "mean",
7784
"tmin": "min",
7885
"tmax": "max",
@@ -88,12 +95,18 @@ class Daily(TimeSeries):
8895
def __init__(
8996
self,
9097
loc: Union[pd.DataFrame, Point, list, str], # Station(s) or geo point
91-
start: datetime = None,
92-
end: datetime = None,
93-
model: bool = True, # Include model data?
94-
flags: bool = False, # Load source flags?
98+
start=datetime(1781, 1, 1, 0, 0, 0),
99+
end=datetime.combine(
100+
datetime.today().date() + timedelta(days=10), datetime.max.time()
101+
),
102+
model=True, # Include model data?
103+
flags=False, # Load source flags?
95104
) -> None:
96-
105+
# Extract relevant years
106+
if self.chunked:
107+
self._annual_steps = [
108+
start.year + i for i in range(end.year - start.year + 1)
109+
]
97110
# Initialize time series
98111
self._init_time_series(loc, start, end, model, flags)
99112

0 commit comments

Comments
(0)

AltStyle によって変換されたページ (->オリジナル) /