Iterate over chunks of date range in Python

Question

def forward_date_range(start_dt, end_dt, span_days):
 #
 # Generate tuples with intervals from given range of dates (forward)
 #
 # forward_date_range('2012-01-01', '2012-01-5', 2)
 #
 # 1st yield = ('2012-01-01', '2012-01-03')
 # 2nd yield = ('2012-01-04', '2012-01-05')
 start_dt = datetime.datetime.strptime(start_dt, '%Y-%m-%d')
 end_dt = datetime.datetime.strptime(end_dt, '%Y-%m-%d')
 span = datetime.timedelta(days=span_days)
 step = datetime.timedelta(days=1)
 while start_dt + span < end_dt:
 current = start_dt + span
 yield start_dt.strftime('%Y-%m-%d'), current.strftime('%Y-%m-%d')
 start_dt = current + step
 else:
 yield start_dt.strftime('%Y-%m-%d'), end_dt.strftime('%Y-%m-%d')
def backward_date_range(start_dt, end_dt, span_days):
 #
 # Generate tuples with intervals from given range of dates (backward)
 #
 # backward_date_range('2012-01-01', '2012-01-5', 2)
 #
 # 1st yield = ('2012-01-03', '2012-01-05')
 # 2nd yield = ('2012-01-01', '2012-01-02')
 start_dt = datetime.datetime.strptime(start_dt, '%Y-%m-%d')
 end_dt = datetime.datetime.strptime(end_dt, '%Y-%m-%d')
 span = datetime.timedelta(days=span_days)
 step = datetime.timedelta(days=1)
 while end_dt - span > start_dt:
 current = end_dt - span
 yield current.strftime('%Y-%m-%d'), end_dt.strftime('%Y-%m-%d')
 end_dt = current - step
 else:
 yield start_dt.strftime('%Y-%m-%d'), end_dt.strftime('%Y-%m-%d')

This is currently what I have, needless to say it looks ugly. Is there a more elegant solution, especially if it is possible to get rid of the while loop altogether? or maybe compress it into one single line?

I am currently using python3.5

• 301_Moved_Permanently: Hi, you might be able to get better reviews by providing a bit more context. Can you add a bit of explanation of why you need such feature and/or some code calling these functions?
• AlanSTACK: @MathiasEttinger its a helper function that is used to supply start and end "range subsets" to SQL queries that wouldve clogged up a network. Something like "1996-2016" is split into smaller chunks for convenience
• Gareth Rees: @Alan: Surely adding LIMIT to the SQL queries would have been a better solution to this problem?
• AlanSTACK: @GarethRees The data structure of this case prevents this
• J_H: FYI, relational databases usually perform better if you give them one big query rather than requesting the same rows via many component queries. That is to say, it's better to let the DB do the looping than to have your loop generate related queries. Why? A single big query lets the backend planner see the future, lets it optimize access paths across all result rows rather than just the rows of a component query. Depending on the DB vendor, daily partitions of a large table can be an attractive approach. postgresql.org/docs/current/ddl-partitioning.html

Answer 1

I'm going to assume that the corner cases are what you want, i.e. a date "range" of a single day is acceptable - otherwise you'd have to overhaul the calculation a bit.

I'm not going to attempt to remove the while loops - I don't know a better alternative; six lines self-contained logic each isn't too bad IMO.

First things first, I'd recommend a couple of things for better readability and interactivity:

• Use proper docstrings instead of comments, so that both tools and human readers know what the functions are about. At the moment help(forward_date_range) isn't too helpful, but with that change it would be much more so.
• Similarly, the _dt suffix isn't that readable either. I'd either drop it or expand it to _date so that it's more obvious.

Next, I'd try to remove some more duplication:

• The string '%Y-%m-%d' comes up a total of twelve times. Either put that into a constant (if you don't ever want users to supply a different format), or make it an optional function parameter.
• The structure of both functions is also quite similar, so while the loop might not be worth to factor out, the pre- and postprocessing definitely is. Also, timedelta(days=1) is another constant in the code.
• Importing the right names (e.g. datetime and timedelta) could also cut down the number of tokens to read.
• There's also an opportunity to cache the result of end - span into a separate variable so as to not repeat it in every loop.
• (From @mathias-ettinger) The loop doesn't have a break to exit it early, so the else branch will always be executed. In that case it makes sense to just put it on its own line without the else which doesn't change the meaning in any respect.

Lastly, I'd probably say that the conversion from and to datetime objects doesn't belong into these functions and should be done separately instead. If you want to keep it like it is there are still some opportunities for helper functions to cut down the noise.

The result I'm posting below can still be compressed further, but at that point it would be generally getting more functional and less like regular Python code.

So there we have it:

from datetime import datetime, timedelta
DATE_FORMAT = '%Y-%m-%d'
DATE_STEP = timedelta(days=1)
def _strptime(string):
 return datetime.strptime(string, DATE_FORMAT)
def _strftime(date):
 return date.strftime(DATE_FORMAT)
def _date_range_parameters(start, end, span_days):
 start = _strptime(start)
 end = _strptime(end)
 span = timedelta(days=span_days)
 return start, end, span
def forward_date_range(start, end, span_days):
 """
 Generate tuples with intervals from given range of dates (forward).
 forward_date_range('2012-01-01', '2012-01-5', 2)
 1st yield = ('2012-01-01', '2012-01-03')
 2nd yield = ('2012-01-04', '2012-01-05')
 """
 start, end, span = _date_range_parameters(start, end, span_days)
 stop = end - span
 while start < stop:
 current = start + span
 yield _strftime(start), _strftime(current)
 start = current + DATE_STEP
 yield _strftime(start), _strftime(end)
def backward_date_range(start, end, span_days):
 """
 Generate tuples with intervals from given range of dates (backward)
 backward_date_range('2012-01-01', '2012-01-5', 2)
 1st yield = ('2012-01-03', '2012-01-05')
 2nd yield = ('2012-01-01', '2012-01-02')
 """
 start, end, span = _date_range_parameters(start, end, span_days)
 stop = start + span
 while end > stop:
 current = end - span
 yield _strftime(current), _strftime(end)
 end = current - DATE_STEP
 yield _strftime(start), _strftime(end)