-
-
Notifications
You must be signed in to change notification settings - Fork 73
warcio cannot write wet files #150
Open
Description
I am trying to use warcio to write WET files that hold text-only conversion records, but I am not able to find a way to write a record using warcio without having to make a live web request.
This is a sample of what I am trying to do:
def create_wet_file(url):
with open(f"BigData/test1.wet.gz", 'wb+') as wet:
writer = WARCWriter(wet, gzip=True)
try:
resp = requests.get(url, headers={'Accept-Encoding': 'identity, gzip', 'Content-Type': 'text/html; charset=utf-8'}, stream=True)
headers_list = resp.raw.headers.items()
headers = StatusAndHeaders('200 OK', headers_list, protocol='HTTP/1.0')
record = writer.create_warc_record(uri=url, record_type='response', payload=resp.raw, http_headers=headers)
writer.write_record(record)
except requests.exceptions.ConnectionError as e:
print(e)
Is this a limitation in warcio, or is there a way around it?
Thank you for any pointers.
Activity
Metadata
Metadata
Assignees
Labels
No labels