Optimizing combining multiple feature classes into one via arcpy

Question

I have about 1000 feature classes spread over about 50 geodatabases which I am trying to append into an empty layer in a separate geodatabase.

When I previously did this on shapefiles, the process completed quickly. However due to earlier processes (requiring Query Tables) I had to move the data into a GDB, and this attempt to append the layers has been running for about 50 minutes.

Are there any recommendations to decrease the time required to merge these feature classes? Should I copy the feature classes to shapefiles first?

The layers all have the same field set; I am not using any field mappings.

General process:

1. Loop through the GDBs using arcpy.da.walk; add feature class path/name into a list
2. Create a feature class in the output GDB using one of the fcs above as a template
3. Use arcpy.Append_management(fclist, out_layername)

Note: While the first step (obtaining the list of FCs via arcpy.da.walk) takes a considerable amount of time, that is acceptable - I'd like to improve the append or merge step.

The specific code snippet I am trying to improve is:

# Set the workspace environment setting
arcpy.env.workspace = os.path.join(rootworkingdir, gdbName)
out_path = gdbpath
out_name = fcname
geometry_type = "POLYLINE" 
template = fclist[0]
# Execute CreateFeatureclass
arcpy.CreateFeatureclass_management(
 out_path, out_name, geometry_type, template)
out_name = out_path + "/" + out_name
arcpy.Append_management(fclist, out_layername)

The total number of expected features is about 5000.

• Vince: There is never any reason to copy a file geodatabase table to a shapefile to populate a file geodatabase table (unless you want your data corrupted). The only thing you can do to reduce time is to drop the spatial index (and any other indexes) of the target table before you start inserts.
• PolyGeo: It's always hard to suggest alternative coding patterns to what you are currently doing when a code snippet that describes precisely what you are currently doing has not been presented. Also, if it is code that works, which you are simply trying to improve, then the Code Review Stack Exchange specializes in that.
• Michael Stimson: It sounds like you're appending each feature class one by one, you might find it quicker to append them all at the same time, python lists can be quite large. You could also try search/insert cursor to insert the features. Python can be a trifle slow at some operations, if speed is really important you could code this in ArcObjects using search/insert cursors with a feature buffer object for peak performance; there are lots of feature classes but how many features are we talking about? As is stated before, without seeing your code it's very hard to make any specific suggestions.
• SMiller: Edited to explicitly include code snippet and number of expected features in the output. I'm appending using a list of feature classes (append_management uses a list as input). @Vince, could you clarify what you mean by dropping spatial indexes? I am not aware of any on the input datasets.
• Vince: The input ought to have spatial indexes, unless something is very wrong. But it's the output table that loads significantly faster without an index. Your code sample doesn't show how ficlist or 'fclist` are populated (and I expect they should be the same).

Answer 1

As commented by @Vince:

There is never any reason to copy a file geodatabase table to a shapefile to populate a file geodatabase table (unless you want your data corrupted). The only thing you can do to reduce time is to drop the spatial index (and any other indexes) of the target table before you start inserts.

Don't forget that converting a geodatabase feature class to a shapefile is likely to truncate some of its field names for starters, but a number of other shapefile/dBase limitations are likely to come into play.

Answer 2

Using Merge_management in step-wise fashion (described below) was faster in this case than the Append_management for all of the layers at once. This did not require removing the indexes (via ListIndexes and RemoveIndex_Management or RemoveSpatialIndex_management, although removing the indexes per @Vince may speed processing; I didn't have a chance to test. Note: The Merge method was much faster within ArcMap (interactively) than via arcpy.

For each geodatabase:

1. Identify the desired feature classes
2. create a temporary list of fcs
3. Merge the fcs into a new layer in the final geodatabase

Once all GDBs are processed, merge the 50 layers from step 3 above.