QVariant error when converting DataFrame into attribute table using PyQGIS

Question

I have a df which looks like that:

ID	Happy
0	Very
1	Little

I'm trying to convert it into an attribute table:

headers = [col for col in df.columns]
fieldlist = QgsFields()
fieldlist.append(QgsField(headers[0],QVariant.Int))
for name in headers[1:]:
 fieldlist.append(QgsField(name, QVariant.String))
for i in df.index.to_list():
 featur = QgsFeature()
 newrow = df[headers].iloc[i].tolist()
 featur.setAttributes(newrow)
 sink.addFeature(featur,QgsFeatureSink.FastInsert)

Although I get this error:

Feature could not be written to Output_layer_9a58cfa4_ee1e_4b97_a6e8_dfc0410a280a: Could not store attribute "ID": Could not convert value "" to target type

Why do I get this even though I state that the first column (QgsField) has numbers and the others have Strings?

Answer 1

A standalone sample correction. The issue is the fact you try to fill QGIS content using for integer a <class 'numpy.int64'> (due to pandas type) whereas it should be a native Python int.

The important part is newrow = [c.item() if hasattr(c, 'item') else c for c in newrow] that cast numpy objects to their Python equivalent and then avoid your code to fail.

import pandas as pd
 
# initialize list of lists
data = [[0, 'Very'], [1, 'Little']]
 
# Create the pandas DataFrame
df = pd.DataFrame(data, columns = ['ID', 'Happy'])
headers=[col for col in df.columns]
fieldlist=QgsFields()
fieldlist.append(QgsField(headers[0],QVariant.Int))
for name in headers[1:]:
 fieldlist.append(QgsField(name,QVariant.String))
sink = QgsVectorLayer("NoGeometry?crs=EPSG:4326&field=ID:integer(10,0)&field=Happy:string(10,0)", "temp", "memory")
for i in df.index.to_list():
 featur=QgsFeature()
 newrow=df[headers].iloc[i].tolist()
 print(type(newrow[0]))
 newrow = [c.item() if hasattr(c, 'item') else c for c in newrow]
 featur.setAttributes(newrow)
 sink.dataProvider().addFeature(featur)
QgsProject.instance().addMapLayer(sink)

Answer 2

I believe it is because you are passing Pandas datatypes to the attributes of the QgsFeatures.

import pandas as pd
## make a toy dataframe
data = [[0, 'Very'], [1, 'Little']]
df = pd.DataFrame(data, columns = ['ID', 'Happy'])
## check dtypes
df.dtypes
## returns
df.dtypes
ID int64
Happy object
dtype: object

Here is a function that will convert int and float Pandas dtypes to native Python types. It may have to be be extended if you have more fields with different types.

def convert_dtype(data):
 """ function to convert pandas data types to native python type """
 
 def conversion(element):
 ## try and except are used because strings (in this context) behave differently 
 ## to ints/floats and do not have a dtype attribute
 try:
 if element.dtype.name == 'int64':
 return int(element)
 elif element.dtype.name == 'float64':
 return float(element)
 else:
 return element
 except:
 return element
 
 return [conversion(x) for x in data]
 
headers=[col for col in df.columns]
fieldlist=QgsFields()
fieldlist.append(QgsField(headers[0],QVariant.Int))
for name in headers[1:]:
 fieldlist.append(QgsField(name,QVariant.String))
for i in df.index.to_list():
 featur=QgsFeature()
 newrow=df[headers].iloc[i].tolist()
 ######
 ## convert the pandas dtypes to python types
 converted = convert_dtype(newrow)
 ######
 ## use the converted types to pass to the feature
 featur.setAttributes(converted)
 sink.addFeature(featur)

enter image description here

• Annie S.: The only thing I had to change was 'int64' to 'int32'.