FW: Reading a remove csv file

Thu Nov 2 18:57:12 EDT 2017

On 11/2/2017 9:18 AM, ROGER GRAYDON CHRISTMAN wrote:
> I have a partial answer to my own question:
> This seems to work for me:
>> ---
> link = urllib.request.urlopen(urlpath)
> data = link.read().decode('utf-8').split('\n')
>> reader = csv.DictReader(data)
> for row in reader:
> ---
>> I think here my concern is that now 'data' is now a variable
> in my program's memory (all of the data),
> instead of streamlike. I suppose I did read
> somewhere about setting a stream option.

csv.reader and csv.DictReader are transformation iterators whose first 
argument must be an iterable of string lines. Given an iterator of 
bytes lines, you just need to interpose a bytes to string iterator -- 
something like (untested)
def strgen(bytesource, encoding):
 for line in bytesource:
 yield line.decode(encoding)
with urllib.request.urlopen(urlpath) as link:
 lines = strgen(link, 'utf-8')
 reader = csv.DictReader(lines) # plus any other args
 for row in reader:
 process(row)
Iterators are intended to be chained together like this.
-- 
Terry Jan Reedy