Commit 778823a

committed

set up and test run extract for contest ranking pipeline

1 parent acd0117 commit 778823aCopy full SHA for 778823a

File tree

+10

-2

lines changed

+10

-2

lines changed

Lines changed: 10 additions & 2 deletions

Original file line number	Diff line number	Diff line change
`@@ -7,7 +7,7 @@`
`7`	`7`
`8`	`8`	`sys.path.append(os.path.dirname(os.path.dirname(os.path.abspath(__file__)))) # Fix ModuleNotFoundError`
`9`	`9`
`10`		`-from operators.contest_ranking_ops import extract_contest_ranking`
	`10`	`+from operators.contest_ranking_ops import extract_contest_ranking, transform_contest_ranking`
`11`	`11`
`12`	`12`	`default_args = {`
`13`	`13`	`"owner": "minhduc29",`
`@@ -25,8 +25,16 @@`
`25`	`25`
`26`	`26`	`# Extract raw data directly from API and store in local/cloud storage`
`27`	`27`	`extract = PythonOperator(`
`28`		`- task_id=f"extract_contest_ranking",`
	`28`	`+ task_id="extract_contest_ranking",`
`29`	`29`	`python_callable=extract_contest_ranking,`
`30`	`30`	`op_args=[4],`
`31`	`31`	`dag=dag`
`32`	`32`	`)`
	`33`	`+`
	`34`	`+transform = PythonOperator(`
	`35`	`+ task_id="transform_contest_ranking",`
	`36`	`+ python_callable=transform_contest_ranking,`
	`37`	`+ dag=dag`
	`38`	`+)`
	`39`	`+`
	`40`	`+extract >> transform`

Comments

(0)