Linked Questions

53 votes

6 answers

127k views

Below is my code that I'd like some help with. I am having to run it over 1,300,000 rows meaning it takes up to 40 minutes to insert ~300,000 rows. I figure bulk insert is the route to go to speed ...

15 votes

4 answers

31k views

Using pandas dataframe's to_sql method, I can write a small number of rows to a table in oracle database pretty easily: from sqlalchemy import create_engine import cx_Oracle dsn_tns = "(DESCRIPTION=(...

19 votes

2 answers

26k views

I have stock price data that is stored in a pandas DataFrame as shown below (actually it was in a panel, but I converted it to a DataFrame) date ticker close tsr 0 2013年03月28日 abc ...

9 votes

3 answers

20k views

I have a 1,000,000 x 50 Pandas DataFrame that I am currently writing to a SQL table using: df.to_sql('my_table', con, index=False) It takes an incredibly long time. I've seen various explanations ...

3 votes

2 answers

8k views

I come to you because i cannot fix an issues with pandas.DataFrame.to_sql() method. I've made the connection between my script and my database, i can send queries, but actually it's too slow for me. I ...

4 votes

2 answers

5k views

I am having this problem when I am trying to copy to AWS redshift. This is the code I am trying to run: with open('path/to/files, 'rb') as fo: cursor.copy_from(fo, 'schema.table', sep=',') ...

2 votes

2 answers

6k views

I have to read an online csv-file into a postgres database, and in that context I have some problems reading the online csv-file properly. If I just import the file it reads as bytes, so I have to ...

4 votes

1 answer

4k views

I am working on a project where I have to write a data frame with Millions of rows and about 25 columns mostly of numeric type. I am using Pandas DataFrame to SQL Function to dump the dataframe in ...

2 votes

4 answers

1k views

Let's say I have a script that reads data into a dataframe from a database, runs some logic on that dataframe, and then exports the resulting dataframe into another database table like below. The ...

1 vote

1 answer

4k views

import cx_Oracle import pandas as pd from sqlalchemy import create_engine # credentials username = "user" password = "password" connectStr = "ip:port/service_name" df = pd.read_csv("data.csv") # ...

0 votes

1 answer

3k views

I am migrating from using pyodbc directly in favor of sqlalchemy as this is recommended for Pandas. I could do a simple executemany(con, df.values.tolist()) to bulk insert all rows from my pandas ...

0 votes

1 answer

1k views

I have a dataframe with 300,000 rows and 20 columns with a lot of them containing text. In Excel format this is 30 to 40 MB. My target is to write this to the database in below 10min. How can I do ...

1 vote

1 answer

843 views

Having a worksheet with ~20.000 rows of data, what is the best approach to insert those into a postgres database? The table in which I will insert the data consists of many foreign keys, which means ...

0 votes

1 answer

338 views

Appending to a SQLite database iteratively using pandas to_sql() functionality appears to be decreasing in its performance with every iteration. The dataframe that I am appending within each ...

1 vote

0 answers

323 views

I'm trying to write 300,000 rows to a postgresql database with pandas.to_sql and SQLalchemy. The rows contain some JSON, but mainly String columns (~25 columns total). Current implementation takes ~17 ...