Python Script to convert a mass of KMLs to a single fGDB

Question

I have a mass of KML files, approx 350+ that I need in a single fGDB. I lifted a string of code from ESRI help to attempt to do this - I got it working to the point of creating all the individual gGDBs (one per KML) but its failing with an invalid character in a variable somewhere. Below is the code, and then the error that it throws out. I've added some print commands to show whats in the variables right before it fails. Help? Also, this is my first time trying to use python for actual work, so I apologize in advance for n00bish mistakes.

# Name: BatchKML_to_GDB.py
# Description: Converts a directory of KMLs and copies the output into a single fGDB.
# A 2 step process: first convert the KML files, and then copy the featureclases
# Import system models
import arcpy, os
# Set workspace (where all the KMLs are)
 arcpy.env.workspace="C:\\Users\\mperez\\Desktop\\IT-Paths"
# Set local variables and location for the consolidated file geodatabase
outLocation = "V:\2155円\\active\215510456円\\GDB"
MasterGDB = r'AllKMLLayers.gdb'
MasterGDBLocation = os.path.join(outLocation, MasterGDB)
# Create the master FileGeodatabase
 arcpy.CreateFileGDB_management(outLocation, MasterGDB)
# Convert all KMZ and KML files found in the current workspace
 for kmz in arcpy.ListFiles('*.KM*'):
 print "CONVERTING: " + os.path.join(arcpy.env.workspace,kmz)
 arcpy.KMLToLayer_conversion(kmz, outLocation)
# Change the workspace to fGDB location
arcpy.env.workspace = outLocation
# Loop through all the FileGeodatabases within the workspace
wks = arcpy.ListWorkspaces('*', 'FileGDB')
# Skip the Master GDB
wks.remove(MasterGDBLocation)
for fgdb in wks: 
 # Change the workspace to the current FileGeodatabase
 arcpy.env.workspace = fgdb 
 # For every Featureclass inside, copy it to the Master and use the name from the original fGDB
 featureClasses = arcpy.ListFeatureClasses('*', '', 'Placemarks')
 for fc in featureClasses:
 print r"COPYING: " + fc + r" FROM: " + fgdb 
 fcCopy = fgdb + os.sep + 'Placemarks' + os.sep + fc
 print r"fcCopy = " + fcCopy
 print r"MasterGDBLocation = " + MasterGDBLocation
 print r"fgdb = " + fgdb
 arcpy.FeatureClassToFeatureClass_conversion(fcCopy, MasterGDBLocation, fgdb[fgdb.rfind(os.sep)+1:-4])
# Clean up
del kmz, wks, fc, featureClasses, fgdb

Output from shell showing error:

COPYING: Points FROM: V:2155円\active215510456円\GDB\canal-move-2-golden-gate-blvd.gdb
fcCopy = V:2155円\active215510456円\GDB\canal-move-2-golden-gate-blvd.gdb\Placemarks\Points
MasterGDBLocation = V:2155円\active215510456円\GDB\AllKMLLayers.gdb
fgdb = V:2155円\active215510456円\GDB\canal-move-2-golden-gate-blvd.gdb
Traceback (most recent call last):
 File "C:\Users\mperez\Desktop\batch_kml_to_gdb.py", line 46, in <module>
 arcpy.FeatureClassToFeatureClass_conversion(fcCopy, MasterGDBLocation, fgdb[fgdb.rfind(os.sep)+1:-4])
 File "C:\Program Files (x86)\ArcGIS\Desktop10.1\arcpy\arcpy\conversion.py", line 1547, in FeatureClassToFeatureClass
 raise e
ExecuteError: ERROR 000354: The name contains invalid characters
Failed to execute (FeatureClassToFeatureClass).

• om_henners: One other this, could you add which version of ArcGIS you're using to the tags as well? (and +1 for actually posting the stacktrace - you'd be amazed at how often this doesn't happen)

Answer 1

In the traceback you can see where the error in the code was raised (the last line), and every function in the call leading to that error - just go backwards up the list.

From this you can see that the error was in line 46 of your code;

arcpy.FeatureClassToFeatureClass_conversion(fcCopy, MasterGDBLocation, fgdb[fgdb.rfind(os.sep)+1:-4])

You know the third argument in Feature Class to Feature Class is the feature class name. Your first step should be to take a look at os.path.basename and os.path.splitext for cleaner ways to get the name.

Your second step should be to look at ValidateTableName in arcpy, which when you pass it the proposed name and workspace, will return a valid name.

• blord-castillo: What om_henners gave you is the solution, but just to understand why this particular name, canal-move-2-golden-gate-blvd failed, - is an invalid character. The list is `~@#$%^&()-+=,{}.![]; /\:*?'<>|

Answer 2

This is not meant to be an answer, just a contribution (taken from my comment on om_henners answers)

import arcpy
invalidcharacters = """`~@#$%^&()-+=,{}.![]; /\:*?'<>|"""
valid = arcpy.ValidateTableName(invalidcharacters)
print valid

Output:

______________________________

(that means every character in `~@#$%^&()-+=,{}.![]; /\:*?'<>| is an invalid character)

• om_henners: Thanks - I wasn't sure of the complete list of invalid charactes, so I didn't want to mislead anyone!
• blord-castillo: I'm not sure that is complete either, but that's all I have been able to find :) I thought " was invalid too, but it was not for fgdb when I tested it.

Answer 3

I know this is a very old question by now, but here is a script I've put together that can deal with a bunch of corner cases of the KML/KMZ to GDB conversion process.

I hope someone else finds this useful!

Btw, not sure if this is obvious, but it might be worth mentioning that the code below was written for Python 3.

# This script was heavily modified from the script available in the ArcGIS Pro documentation:
# https://pro.arcgis.com/en/pro-app/latest/tool-reference/conversion/kml-to-layer.htm
# Also includes information from the following GIS Stack Overflow question: 
# https://gis.stackexchange.com/questions/55259/python-script-to-convert-a-mass-of-kmls-to-a-single-fgdb?noredirect=1&lq=1
# Converts a directory of KMZs/KMLs and copies the output into a single GDB
# Import system modules
import arcpy
import os
import pathlib
import string
def export_multiple_kmzs_to_single_gdb(kmz_folder, output_gdb, temp_folder):
 '''
 Function that reads an input folder with multiple KMZ/KML files and exports
 all the geometric data contained in those files into a single GDB file.
 
 Parameters
 ----------
 kmz_folder : str
 The input folder that contains all the KMZs/KMLs 
 that will be converted
 output_gdb : str
 Path for the consolidated GeoDataBase (.gdb) file
 temp_folder : str
 Temporary folder where multiple GDB and LRYX files will be generated
 and later deleted.
 
 Returns
 -------
 None.
 
 Outputs
 -------
 This function creates a ".gdb" file that contains all the geometric 
 features from the KMZ/KML files in the input directory. 
 The data from the KMZs/KMLs are all blended into one layer per geometry 
 type. For example, suppose the following file structure:
 -"A.kmz" has points, 
 -"B.kmz" has points and linestrings,
 -"C.kmz" has points and polygons,
 -"D.kmz" has points. 
 In that case, the output GDB file will have 3 layes:
 -One layer containing all the points (called "All_Points"), another 
 containing all the polylines (called "All_Polylines"), and yet another 
 containing all the polygons (called "All_Polygons").
 
 '''
 # Fishing out the output folder and the GDB filename
 output_gdb_path = pathlib.Path(output_gdb)
 output_folder = str(output_gdb_path.parent)
 master_gdb = output_gdb_path.stem + output_gdb_path.suffix
 
 # Creating the temp folder in case it doesn't exist
 temp_newly_created = False
 if not os.path.exists(temp_folder):
 # Flag to indicate that this folder should be deleted at the end of
 # the process
 temp_newly_created = True
 os.mkdir(temp_folder)
 
 # Creating master GDB. This will store many temporary layers and the 
 # finalized combined layers.
 master_gdb_fullpath = os.path.join(output_folder, master_gdb)
 if os.path.exists(master_gdb_fullpath):
 arcpy.Delete_management(master_gdb_fullpath)
 arcpy.CreateFileGDB_management(output_folder, master_gdb)
 
 # Get list of all KMZs/KMLs in the input folder
 list_of_kmzs = [f for f in os.listdir(kmz_folder) 
 if ((f[-4:].lower()=='.kmz') or (f[-4:].lower()=='.kml'))]
 
 # Set that contains the names of the consolidated layers created thus far
 # (i.e., "All_Lines", "All_Polygons", ...). This set starts empty and 
 # gets populated as the execution progresses. 
 consolidated_layers = set()
 
 # Looping over every KMZ/KML file
 for i,kmz in enumerate(list_of_kmzs):
 # Convert KMZ/KML file into a separate temporary GDB
 kmz_fullpath = os.path.join(kmz_folder, kmz)
 kmz_gdb_name = os.path.join(temp_folder, kmz)[:-3] + 'gdb'
 if os.path.exists(kmz_gdb_name):
 arcpy.Delete_management(kmz_gdb_name)
 print(f"CONVERTING: {kmz_fullpath}")
 arcpy.env.workspace = kmz_folder
 arcpy.KMLToLayer_conversion(kmz_fullpath, temp_folder)
 
 feature_classes = arcpy.ListFeatureClasses('*', '', 'Placemarks')
 
 arcpy.env.workspace = kmz_gdb_name
 
 # Looping over every Featureclass from the newly-created GDB
 feature_classes = arcpy.ListFeatureClasses('*', '', 'Placemarks')
 
 # If the input KMZ/KML is empty, the `feature_class` object from the 
 # function above will contain None. This needs to be fixed for the 
 # script to work.
 if feature_classes is None:
 feature_classes = []
 
 for this_fc in feature_classes:
 
 arcpy.env.workspace = kmz_gdb_name
 
 # Adding the "kmz_filename" column
 arcpy.management.AddField(in_table=this_fc,
 field_name='kmz_filename', 
 field_type='TEXT', 
 field_length=1024)
 
 # Updating the values of the "kmz_filename" column
 with arcpy.da.UpdateCursor(this_fc,'kmz_filename') as rows:
 for row in rows:
 row[0] = kmz
 rows.updateRow(row)
 del(row)
 del(rows)
 
 # Creating the name for the new layer
 new_layer_name = kmz_gdb_name[kmz_gdb_name.rfind(os.sep) + 1:-4]
 
 # Fixing some invalid characters in the layer name
 valid_characters = set(string.ascii_letters + string.digits + '_')
 invalid_characters = set([x for x in new_layer_name if x not in valid_characters])
 for this_char in invalid_characters:
 new_layer_name = new_layer_name.replace(this_char,'_')
 new_layer_name = arcpy.ValidateTableName(new_layer_name)
 
 # Ensuring that the first character isn't a number
 if new_layer_name[0] in string.digits:
 new_layer_name = '_' + new_layer_name
 
 # Copying the Featureclass from the newly-created GDB to the 
 # Master GDB
 print(f"COPYING: {this_fc} FROM: {kmz_gdb_name}")
 fcCopy = os.path.join(kmz_gdb_name, 'Placemarks', this_fc)
 
 arcpy.FeatureClassToFeatureClass_conversion(
 fcCopy, master_gdb_fullpath, new_layer_name)
 
 # Copying the features from each individual Featureclass into the 
 # main "All_Points" or "All_Polylines" Featureclasses (i.e., the
 # consolidated layers).
 master_layer_name = f'All_{this_fc}'
 arcpy.env.workspace = master_gdb_fullpath
 print(f"COPYING: {this_fc} TO: {master_layer_name}")
 
 # If the consolidated layer has already been created, append 
 # the KMZ/KML's data to it. If not, the layer needs to be created
 # and the set of consolidated layers needs to be updated.
 if master_layer_name in consolidated_layers:
 arcpy.management.Append(new_layer_name, master_layer_name)
 else:
 arcpy.CopyFeatures_management(new_layer_name, master_layer_name)
 consolidated_layers.add(master_layer_name)
 
 # Delete the temporary Featureclass layer inside the master GDB
 arcpy.Delete_management(new_layer_name)
 
 # Deleting the newly-generated GDB and lyrx files
 arcpy.env.workspace = master_gdb_fullpath
 if os.path.isdir(kmz_gdb_name):
 arcpy.Delete_management(kmz_gdb_name)
 lyrx_filename = kmz_gdb_name[:-3] + 'lyrx'
 if os.path.isfile(lyrx_filename):
 os.remove(lyrx_filename)
 
 # If the temp folder didn't exist at the beginning of this process, it
 # should be deleted at the end.
 if temp_newly_created:
 os.rmdir(temp_folder)