Commit d45965d

committed

big_file_process

1 parent fe0df46 commit d45965dCopy full SHA for d45965d

1 file changed

Lines changed: 63 additions & 0 deletions

File tree

README.md

`‎README.md‎`

Lines changed: 63 additions & 0 deletions

Original file line number	Diff line number	Diff line change
`@@ -3168,7 +3168,70 @@ Out[69]: time.struct_time(tm_year=2020, tm_mon=2, tm_mday=22, tm_hour=11, tm_min`
`3168`	`3168`	`%b Locale's abbreviated month name.`
`3169`	`3169`	```
`3170`	`3170`
	`3171`	`+#### 26 4G 内存处理 10G 大小的文件`
	`3172`	`+`
	`3173`	`+4G 内存处理 10G 大小的文件,单机怎么做?`
	`3174`	`+`
	`3175`	`+下面的讨论基于的假定:可以单独处理一行数据,行间数据相关性为零。`
	`3176`	`+`
	`3177`	`+方法一:`
	`3178`	`+`
	`3179`	`+仅使用 Python 内置模板,逐行读取到内存。`
	`3180`	`+`
	`3181`	`+使用 yield,好处是解耦读取操作和处理操作:`
	`3182`	`+`
	`3183`	+```python
	`3184`	`+def python_read(filename):`
	`3185`	`+ with open(filename,'r',encoding='utf-8') as f:`
	`3186`	`+ while True:`
	`3187`	`+ line = f.readline()`
	`3188`	`+ if not line:`
	`3189`	`+ return`
	`3190`	`+ yield line`
	`3191`	+```
	`3192`	`+`
	`3193`	`+以上每次读取一行,逐行迭代,逐行处理数据`
	`3194`	`+`
	`3195`	+```python
	`3196`	`+if __name__ == '__main__':`
	`3197`	`+ g = python_read('./data/movies.dat')`
	`3198`	`+ for c in g:`
	`3199`	`+ print(c)`
	`3200`	`+ # process c`
	`3201`	+```
	`3202`	`+`
	`3203`	`+方法二:`
	`3204`	`+`
	`3205`	`+方法一有缺点,逐行读入,频繁的 IO 操作拖累处理效率。是否有一次 IO ,读取多行的方法?`
	`3206`	`+`
	`3207`	+`pandas` 包 `read_csv` 函数,参数有 38 个之多,功能非常强大。
	`3208`	`+`
	`3209`	+关于单机处理大文件,`read_csv` 的 `chunksize` 参数能做到,设置为 `5`, 意味着一次读取 5 行。
	`3210`	`+`
	`3211`	+```python
	`3212`	`+def pandas_read(filename,sep=',',chunksize=5):`
	`3213`	`+ reader = pd.read_csv(filename,sep,chunksize=chunksize)`
	`3214`	`+ while True:`
	`3215`	`+ try:`
	`3216`	`+ yield reader.get_chunk()`
	`3217`	`+ except StopIteration:`
	`3218`	`+ print('---Done---')`
	`3219`	`+ break`
	`3220`	+```
	`3221`	`+`
	`3222`	`+使用如同方法一:`
	`3223`	+```python
	`3224`	`+if __name__ == '__main__':`
	`3225`	`+ g = pandas_read('./data/movies.dat',sep="::")`
	`3226`	`+ for c in g:`
	`3227`	`+ print(c)`
	`3228`	`+ # process c`
	`3229`	+```
	`3230`	`+`
	`3231`	`+以上就是单机处理大文件的两个方法,推荐使用方法二,更加灵活。除了工作中会用到,面试中也有时被问到。`
	`3232`	`+`
`3171`	`3233`	`### 四、Python三大利器`
	`3234`	`+`
`3172`	`3235`	Python中的三大利器包括:`迭代器`,`生成器`,`装饰器`,利用好它们才能开发出最高性能的Python程序,涉及到的内置模块 `itertools`提供迭代器相关的操作。此部分收录有意思的例子共计`14`例。
`3173`	`3236`
`3174`	`3237`

0 commit comments

Comments

(0)

Navigation Menu

Search code, repositories, users, issues, pull requests...

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Commit d45965d

File tree

`‎README.md‎`

0 commit comments