Regex failure despite valid regex when creating GeoServer Image Mosaic

Question

I'm attempting to create an ImageMosaic on GeoServer with various custom regex files. I am letting GeoServer create a shapefile index for me. I'm using the example GeoTIFFs from this GeoServer ImageMosaic tutorial and just renaming them with the filenames I will use when I create my own GeoTIFFs later, so I know the GeoTIFFs are valid and work. I was able to do the tutorial itself, so this isn't a file permissions problem either. Currently, I am testing my regex files one by one to figure out which is causing this unhelpful error:

Could not list layers for this store, an error occurred retrieving them: 
Failed to create reader from file:data/hydroalp and hints Hints: 
EXECUTOR_SERVICE = java.util.concurrent.ThreadPoolExecutor@1c93b3d3[Running, pool size = 0, active threads = 0, queued tasks = 0, completed tasks = 0] 
REPOSITORY = org.geoserver.catalog.CatalogRepository@3963d8be System defaults: 
LENIENT_DATUM_SHIFT = true 
FORCE_AXIS_ORDER_HONORING = http 
FILTER_FACTORY = FilterFactoryImpl 
COMPARISON_TOLERANCE = 1.0E-8 
GRID_COVERAGE_FACTORY = GridCoverageFactory 
TILE_ENCODING = null 
STYLE_FACTORY = org.geotools.styling.StyleFactoryImpl@7cd38e49 
FORCE_LONGITUDE_FIRST_AXIS_ORDER = true 
FEATURE_FACTORY = org.geotools.feature.LenientFeatureFactoryImpl@fafec7d

I'm confident the thing causing the error is my regex so I'll cut to the chase, but if you really want to go through the debug stack, I'll post it at the end. Anyway, this is an example of the file name I'm parsing:

bcc_csm2_mr_ssp585_dbcca_daymet_vic4_wind_2020_2059_sep.tif

After testing all my regex files I have identified the problematic regex being the one where I am trying to pick up on the string 2020_2059 in the filename. There are other combinations of 4 digits either side of the underscore could be, so I'm trying to avoid hardcoding the numbers. I am using double backslashes because the docs say to do so and I tested a single backslash (\d) in a different regex file and it didn't work:

regex=.*_(\\d{4}_\\d{4})_.*

I've been testing my regex on regex101 to ensure it's valid. I've tried many variations, all that are valid on that website but GeoServer throws the error for each one I have tried. These are the others I have tried, including hardcoding the numbers just to see if it would work:

regex=(\\d{4}_\\d{4})
regex=.*_(20[0-9]{2}_20[0-9]{2})_.*
regex=.*_(20[0-9][0-9]_20[0-9][0-9])_.*
regex=.*_(2020_2059)_.*
regex=.*(2020_2059).*
regex=(2020_2059)
regex=2020_2059

My indexer.properties file looks like this, the problematic regex is the clm_period_regex. I added carriage returns here for readability but in the actual file there are no spaces or returns after commas:

Schema=*the_geom:Polygon,
location:String,
clm_model:String,
em_scenario:String,
downscale:String,
meteor_force:String,
model_grid:String,
clm_variable:String,
clm_period:String,
time_period:String
PropertyCollectors=StringFileNameExtractorSPI[clm_model_regex](clm_model),
StringFileNameExtractorSPI[em_scenario_regex](em_scenario),
StringFileNameExtractorSPI[downscale_regex](downscale),
StringFileNameExtractorSPI[meteor_force_regex](meteor_force),
StringFileNameExtractorSPI[model_grid_regex](model_grid),
StringFileNameExtractorSPI[clm_variable_regex](clm_variable),
StringFileNameExtractorSPI[clm_period_regex](clm_period),
StringFileNameExtractorSPI[time_period_regex](time_period)
AdditionalDomainAttributes=clm_model,
em_scenario,
downscale,
meteor_force,
model_grid,
clm_variable,
clm_period,
time_period

My other regex sidecar files look very similar to this problematic one, the only difference is that all the other ones either look to capture only letters or a mix of letters and numbers, not only numeric digits and non-letter characters. Given the _ character between the digits though, I would think I would still need to use the StringFileNameExtractorSPI and not IntegerFileNameExtractorSPI.

I'm sure I can come up with a workaround for this, but I really would like to understand why it's not working as I expect it to. I feel like it must have something to do with however GeoServer is interpreting the underscore character(s). I wonder if GeoServer is reading the regex and splitting the filename up like this _2020_, _2059_, trying to find the right group when I need it to evaluate 2020_2059 as a whole.

LOGS

It starts off by giving this message for each .properties file:

22 Oct 17:41:31 DEBUG [gce.imagemosaic] - Trying to load properties file from URL:file:/geoserver_data/data/hydroalp/clm_model_regex.properties
22 Oct 17:41:31 INFO [gce.imagemosaic] - Required key Levels not found.

It then continues to do this. I think this is also for each .properties file?:

22 Oct 16:37:04 DEBUG [gce.imagemosaic] - Checking file:/geoserver_data/data/hydroalp/
isHidden:false
exists:false
isFilefalse
canRead:false
canWritefalse
canExecute:false
isAbsolute:true
lastModified:0
length:0
22 Oct 16:37:04 DEBUG [gce.imagemosaic] - Checking file:/geoserver_data/data/hydroalp/
isHidden:false
exists:false
isFilefalse
canRead:false
canWritefalse
canExecute:false
isAbsolute:true
lastModified:0
length:0
22 Oct 16:37:04 DEBUG [gce.imagemosaic] - Checking file:/geoserver_data/data/hydroalp/
isHidden:false
exists:true
isFiletrue
canRead:true
canWritefalse
canExecute:false
isAbsolute:true
lastModified:1729613816842
length:791
22 Oct 16:37:04 DEBUG [gce.imagemosaic] - Checking file:/geoserver_data/data/hydroalp/
isHidden:false
exists:true
isFiletrue
canRead:true
canWritetrue
canExecute:false
isAbsolute:true
lastModified:1729398104969
length:91
22 Oct 16:37:04 DEBUG [gce.imagemosaic] - Checking file:/geoserver_data/data/hydroalp/
isHidden:false
exists:true
isFiletrue
canRead:true
canWritetrue
canExecute:false
isAbsolute:true
lastModified:1729399676720
length:24
22 Oct 16:37:04 DEBUG [gce.imagemosaic] - Checking file:/geoserver_data/data/hydroalp/
isHidden:false
exists:true
isFiletrue
canRead:true
canWritetrue
canExecute:false
isAbsolute:true
lastModified:1729399710754
length:25
22 Oct 16:37:04 DEBUG [gce.imagemosaic] - Checking file:/geoserver_data/data/hydroalp/
isHidden:false
exists:true
isFiletrue
canRead:true
canWritetrue
canExecute:false
isAbsolute:true
lastModified:1729612228723
length:27
22 Oct 16:37:04 DEBUG [gce.imagemosaic] - Checking file:/geoserver_data/data/hydroalp/
isHidden:false
exists:true
isFiletrue
canRead:true
canWritetrue
canExecute:false
isAbsolute:true
lastModified:1729612598530
length:23
22 Oct 16:37:04 DEBUG [gce.imagemosaic] - Checking file:/geoserver_data/data/hydroalp/
isHidden:false
exists:true
isFiletrue
canRead:true
canWritetrue
canExecute:false
isAbsolute:true
lastModified:1729612764763
length:107
22 Oct 16:37:04 DEBUG [gce.imagemosaic] - Checking file:/geoserver_data/data/hydroalp/
isHidden:false
exists:true
isFiletrue
canRead:true
canWritetrue
canExecute:false
isAbsolute:true
lastModified:1729613794788
length:24
22 Oct 16:37:04 DEBUG [gce.imagemosaic] - Checking file:/geoserver_data/data/hydroalp/
isHidden:false
exists:true
isFiletrue
canRead:true
canWritetrue
canExecute:false
isAbsolute:true
lastModified:1729613505155
length:92
22 Oct 16:37:04 DEBUG [gce.imagemosaic] - Checking file:/geoserver_data/data/hydroalp/
isHidden:false
exists:false
isFilefalse
canRead:false
canWritefalse
canExecute:false
isAbsolute:true
lastModified:0
length:0

Then it starts indexing and creating the shapefile. There's an error near the end that might mean something though I don't understand what:

22 Oct 16:37:04 INFO [gce.imagemosaic] - Now indexing file bcc_csm2_mr_ssp245_dbcca_daymet_vic4_tmax_2060_2099_winter.tif
22 Oct 16:37:04 DEBUG [gce.imagemosaic] - Thread Name http-nio-8080-exec-7
class org.geotools.gce.imagemosaic.ImageMosaicEventHandlers
Now indexing file bcc_csm2_mr_ssp245_dbcca_daymet_vic4_tmax_2060_2099_winter.tif
22 Oct 16:37:04 DEBUG [gce.arcgrid] - ArcGridFormatFactory is available.
22 Oct 16:37:04 DEBUG [gce.arcgrid] - Creating a new ArcGriFormat.
22 Oct 16:37:04 DEBUG [referencing.factory] - Failure in the primary factory: No code "EPSG:32632" from authority "European Petroleum Survey Group" found for object of type "interface org.geotools.api.referencing.IdentifiedObject". Now trying the fallback factory...
22 Oct 16:37:04 DEBUG [imageio.geotiff] - Logging tiePoints:
Tie point
 Raster Space point (0.0, 0.0, 0.0)
 Model Space point (624800.0, 5184500.0, 0.0)
Logging pixScales:
Pixel Scale
 scaleX=100.0 is set? true
 scaleY=100.0 is set? true
 scaleZ=0.0 is set? false
22 Oct 16:37:04 DEBUG [shapefile.files] - Write lock: file:/geoserver_data/data/hydroalp/hydroalp.shp by ShpFiles for deleting all files
22 Oct 16:37:04 DEBUG [shapefile.files] - Write lock: file:/geoserver_data/data/hydroalp/hydroalp.shp by ShpFiles for deleting all files
22 Oct 16:37:04 WARN [shapefile.dbf] - FieldName em_scenario is longer than 10 bytes in given charset, truncating to em_scenari
22 Oct 16:37:04 WARN [shapefile.dbf] - FieldName meteor_force is longer than 10 bytes in given charset, truncating to meteor_for
22 Oct 16:37:04 WARN [shapefile.dbf] - FieldName clm_variable is longer than 10 bytes in given charset, truncating to clm_variab
22 Oct 16:37:04 WARN [shapefile.dbf] - FieldName time_period is longer than 10 bytes in given charset, truncating to time_perio
22 Oct 16:37:04 DEBUG [shapefile.files] - Write lock: file:/geoserver_data/data/hydroalp/hydroalp.shp by StorageFile: hydroalp17194304480828940836.shp
22 Oct 16:37:04 DEBUG [shapefile.files] - Write lock: file:/geoserver_data/data/hydroalp/hydroalp.shp by StorageFile: hydroalp17194304480828940836.shp
22 Oct 16:37:04 DEBUG [shapefile.files] - Unable to rename temporary file to the file: /geoserver_data/data/hydroalp/hydroalp.shp when attempting to replace with temporary copy
22 Oct 16:37:04 DEBUG [shapefile.files] - Write lock: file:/geoserver_data/data/hydroalp/hydroalp.shp by StorageFile: hydroalp17194304480828940836.shp
22 Oct 16:37:04 DEBUG [shapefile.files] - Write lock: file:/geoserver_data/data/hydroalp/hydroalp.shx by StorageFile: hydroalp17940657596150269062.shx
22 Oct 16:37:04 DEBUG [shapefile.files] - Unable to rename temporary file to the file: /geoserver_data/data/hydroalp/hydroalp.shx when attempting to replace with temporary copy
22 Oct 16:37:04 DEBUG [shapefile.files] - Write lock: file:/geoserver_data/data/hydroalp/hydroalp.shx by StorageFile: hydroalp17940657596150269062.shx
22 Oct 16:37:04 DEBUG [shapefile.files] - Write lock: file:/geoserver_data/data/hydroalp/hydroalp.dbf by StorageFile: hydroalp3683525594061069055.dbf
22 Oct 16:37:04 DEBUG [shapefile.files] - Unable to rename temporary file to the file: /geoserver_data/data/hydroalp/hydroalp.dbf when attempting to replace with temporary copy
22 Oct 16:37:04 DEBUG [shapefile.files] - Write lock: file:/geoserver_data/data/hydroalp/hydroalp.dbf by StorageFile: hydroalp3683525594061069055.dbf
22 Oct 16:37:04 DEBUG [shapefile.files] - Write lock: file:/geoserver_data/data/hydroalp/hydroalp.prj by StorageFile: hydroalp6261481280904968373.prj
22 Oct 16:37:04 DEBUG [shapefile.files] - Unable to rename temporary file to the file: /geoserver_data/data/hydroalp/hydroalp.prj when attempting to replace with temporary copy
22 Oct 16:37:04 DEBUG [shapefile.files] - Write lock: file:/geoserver_data/data/hydroalp/hydroalp.prj by StorageFile: hydroalp6261481280904968373.prj
22 Oct 16:37:04 DEBUG [shapefile.files] - Write lock: file:/geoserver_data/data/hydroalp/hydroalp.shp by StorageFile: hydroalp17194304480828940836.shp
22 Oct 16:37:04 DEBUG [shapefile.files] - Read lock: file:/geoserver_data/data/hydroalp/hydroalp.shp by org.geotools.data.shapefile.shp.ShapefileReader
22 Oct 16:37:04 DEBUG [shapefile.files] - Read lock: file:/geoserver_data/data/hydroalp/hydroalp.shx by org.geotools.data.shapefile.shp.IndexFile
22 Oct 16:37:04 DEBUG [shapefile.files] - Read lock: file:/geoserver_data/data/hydroalp/hydroalp.dbf by org.geotools.data.shapefile.dbf.DbaseFileReader
22 Oct 16:37:04 DEBUG [shapefile.files] - Read lock: file:/geoserver_data/data/hydroalp/hydroalp.prj by org.geotools.data.shapefile.ShapefileSetManager
22 Oct 16:37:04 DEBUG [shapefile.files] - Read lock: file:/geoserver_data/data/hydroalp/hydroalp.prj by org.geotools.data.shapefile.ShapefileSetManager
22 Oct 16:37:04 DEBUG [shapefile.files] - Read lock: file:/geoserver_data/data/hydroalp/hydroalp.dbf by org.geotools.data.shapefile.dbf.DbaseFileReader
22 Oct 16:37:04 DEBUG [shapefile.files] - Read lock: file:/geoserver_data/data/hydroalp/hydroalp.shp by org.geotools.data.shapefile.shp.ShapefileReader
22 Oct 16:37:04 DEBUG [shapefile.files] - Read lock: file:/geoserver_data/data/hydroalp/hydroalp.shx by org.geotools.data.shapefile.shp.IndexFile
22 Oct 16:37:04 DEBUG [imagemosaic.catalog] - BBOXFilterExtractor::extractBasicProperties(): geometryPropertyName is set to 'the_geom'.
22 Oct 16:37:04 DEBUG [properties.string] - No matches found for this property extractor:
22 Oct 16:37:04 ERROR [gce.imagemosaic] - Index 0 out of bounds for length 0
java.lang.IndexOutOfBoundsException: Index 0 out of bounds for length 0
 at java.base/jdk.internal.util.Preconditions.outOfBounds(Preconditions.java:64)
 at java.base/jdk.internal.util.Preconditions.outOfBoundsCheckIndex(Preconditions.java:70)
 at java.base/jdk.internal.util.Preconditions.checkIndex(Preconditions.java:248)
 at java.base/java.util.Objects.checkIndex(Objects.java:374)
 at java.base/java.util.ArrayList.get(ArrayList.java:459)
 at java.base/java.util.Collections$UnmodifiableList.get(Collections.java:1310)
 at org.geotools.gce.imagemosaic.properties.string.StringFileNameExtractor.setProperties(StringFileNameExtractor.java:55)
 at org.geotools.gce.imagemosaic.ImageMosaicConfigHandler.updateAttributesFromCollectors(ImageMosaicConfigHandler.java:813)
 at 
....

Eventually it seems to cancel some operation(s):

22 Oct 16:37:04 INFO [gce.imagemosaic] - Stopping requested at file 1 of 4 files
22 Oct 16:37:04 DEBUG [gce.imagemosaic] - Thread Name http-nio-8080-exec-7
class org.geotools.gce.imagemosaic.ImageMosaicEventHandlers
Stopping requested at file 1 of 4 files
22 Oct 16:37:04 WARN [gce.imagemosaic] - Failure occurred while collecting the granules
org.apache.commons.io.DirectoryWalker$CancelException: Operation Cancelled
...
22 Oct 16:37:04 DEBUG [gce.imagemosaic] - Canceled!!!
22 Oct 16:37:04 DEBUG [gce.imagemosaic] - Thread Name http-nio-8080-exec-7
class org.geotools.gce.imagemosaic.ImageMosaicEventHandlers
Canceled!!!
22 Oct 16:37:04 DEBUG [gce.imagemosaic] - Indexing complete

After indexing is "complete", it goes through the Checking file:/geoserver_data/data/hydroalp/ process again, then tells me this:

22 Oct 16:37:04 DEBUG [gce.imagemosaic] - This plugin accepts File, URL or String. The string may describe a File or an URL

Eventually it culminates in these two errors:

22 Oct 16:37:04 INFO [geoserver.web] - Getting list of coverages for saved store file:data/hydroalp
java.lang.RuntimeException: Could not list layers for this store, an error occurred retrieving them: Failed to create reader from file:data/hydroalp and hints Hints:
 EXECUTOR_SERVICE = java.util.concurrent.ThreadPoolExecutor@1c93b3d3[Running, pool size = 0, active threads = 0, queued tasks = 0, completed tasks = 0]
 REPOSITORY = org.geoserver.catalog.CatalogRepository@3963d8be 
System defaults:
 LENIENT_DATUM_SHIFT = true 
 FORCE_AXIS_ORDER_HONORING = http 
 FILTER_FACTORY = FilterFactoryImpl 
 COMPARISON_TOLERANCE = 1.0E-8 
 GRID_COVERAGE_FACTORY = GridCoverageFactory 
 TILE_ENCODING = null 
 STYLE_FACTORY = org.geotools.styling.StyleFactoryImpl@7cd38e49 
 FORCE_LONGITUDE_FIRST_AXIS_ORDER = true 
 FEATURE_FACTORY = org.geotools.feature.LenientFeatureFactoryImpl@fafec7d
...
Caused by: java.io.IOException: Failed to create reader from file:data/hydroalp and hints Hints:
 EXECUTOR_SERVICE = java.util.concurrent.ThreadPoolExecutor@1c93b3d3[Running, pool size = 0, active threads = 0, queued tasks = 0, completed tasks = 0]
 REPOSITORY = org.geoserver.catalog.CatalogRepository@3963d8be 
System defaults:
 LENIENT_DATUM_SHIFT = true 
 FORCE_AXIS_ORDER_HONORING = http 
 FILTER_FACTORY = FilterFactoryImpl 
 COMPARISON_TOLERANCE = 1.0E-8 
 GRID_COVERAGE_FACTORY = GridCoverageFactory 
 TILE_ENCODING = null 
 STYLE_FACTORY = org.geotools.styling.StyleFactoryImpl@7cd38e49 
 FORCE_LONGITUDE_FIRST_AXIS_ORDER = true 
 FEATURE_FACTORY = org.geotools.feature.LenientFeatureFactoryImpl@fafec7d

• bixb0012: I wonder if the inclusion of a capturing group is causing an issue. Does .*(?:_[0-9]{4}_[0-9]{4}_).* work?
• Ian Turton: It may not be your regex - what about Required key Levels not found. ?
• MKF: @bixb0012 Unfortunately that doesn’t make a difference, but thanks for the suggestion.
• MKF: @IanTurton - that’s an INFO message, so I’m not sure it’s really saying it’s a problem, despite the suggestion it’s "required." Plus, if you look at the docs, Levels is actually not required in any .properties file except the <mosaic_dir>.properties file that GeoServer is supposed to create for you. I don’t know why it propagates that message for regex .properties files.

Answer 1

TLDR; The moral of story is you need to check all these things if you get the same cryptic error above because any one of them could be the culprit as they all produce an identical error message:

1. Make sure your regex is valid. I suggest testing it on regex101.com. If it is valid but uses characters like \ or # or =, you must escape them as GeoServer requires. E.g. valid regex might be \d{2} but GeoServer needs you to escape the backslash, so for use with GeoServer, your regex would need to look like this \\d{2}.
2. You cannot use quotes, at least with Wildcard, in your indexer.properties file. If the reason you're using quotes is to help translate spaces in text, you could just escape the spaces with \ and not use quotes. You could also try escaping the quotes with \ but I do not know if that would work.
3. Similar to the above, you must escape spaces in your datastore.properties file if you're creating your index in a database. They must be escaped in keys, as well as values, in the file. This is, more or less, noted in the documentation itself.
4. Ensure that the user running GeoServer has read and write permissions on your mosaic directory or is at least part of a group that has read/write permissions on the mosaic directory. In my setup, we run GeoServer on Tomcat, so I made the tomcat user own the directory that contains the GeoTIFFs. Essentially, I ran these two commands in the mosaic directory:

# Updates the directory to be owned by user tomcat 
chown tomcat <mosaic dir name>
# Updates permissions so the user has read and write permissions on the directory
chmod u+rw <mosaic dir name>
# If you don't want your GeoServer user to be the owner of the mosaic directory but instead it is part of a group that should be able to read/write the directory, you can update group permissions like this
chmod g+rw <mosaic dir name>

5. READ ABOUT WHAT THESE DO BEFORE YOU DO THEM: On your GeoServer web administration interface, go to Server Status and click the "Clear" button next to Resource Cache. Also click the "Reload" button next to Configuration and Catalog.
6. While I don't think this is a requirement at all, if you're trying to create an Image Mosaic from GeoTIFF files, I would suggest making your GeoTIFFs with GDAL as that is what the tutorials in GeoServer docs use, thus your GeoTIFFs are less likely be the culprit of the cryptic error which can potentially help you narrow down what the real problem is.

Long Explanation

I removed my problem regex files and just focused on getting a database index to work next, to which I was successful using only the regex files I knew were fine and had gotten to work before, along with the same TIF files. No other changes were made. I created the Image Mosaic, published the layer, and checked that the database table was created as I expected; it was. All was fine. I then wanted to see if I could create the Image Mosaic twice in a row with the exact same data and set of files just to make sure GeoServer was being consistent, so I deleted the layer I published, deleted the Image Mosaic store I created, deleted the table in the database that was created to hold the index, and deleted both sample_image.dat and <mosaic dir name>.properties file that GeoServer created in my mosaic directory. Upon trying to recreate the Image Mosaic, lo and behold, I received the same error message (and logs) reported in my OP even though I had changed nothing between my attempts. That suggested something was being cached and the error was somehow propagating from the cache every time I was trying to create a new Image Mosaic. I decided to go to Server Status in the GeoServer web admin interface and click the "Clear" button next to Resource Cache. I also decided to click the "Reload" button next to Configuration and Catalog on the same page. I then tried to recreate the Image Mosaic with the same set of files and it worked again.

Now, with this new information that GeoServer is just not letting go of the cache somehow, I went on to testing my "problematic" regex (I tested with regex=.*_(\\d{4}_\\d{4})_.*). I deleted all the Image Mosaic things as mentioned above, then added my problematic regex back to my mosaic directory and updated my indexer.properties to incorporate it. For sh*ts and giggles, I decided to try creating the Image Mosaic first without clearing the resource cache or reloading the configuration and catalog. That resulted in the same error message. I then tried it by clearing the resource cache and reloading the config and catalog, and suddenly, everything published. Database index was created as expected, my Image Mosaic had no issues. I repeated the process again, making sure the clear the cache and reload config/catalog between attempts, and it worked.

To be thorough, I then decided to add things to my indexer.properties file, like Wildcard="*.tif". I think I interpreted the example in the docs too literally which is why I included the quotes around *.tif. I deleted my previous Image Mosaic/db table, cleared/reloaded cache/config/catalog, and tried again. Same error, even though I cleared/reloaded cache/config/catalog. Tried it one more time, same error. Turns out Wildcard can't include quotes; it needs to look like this Wildcard=*.tif. Once I changed that and tried again, it worked. The fact that I received the exact same error message though as when I had problems in my regex files, or when I had problems with stale cache, is what makes these problems so incredibly difficult to debug; you can't tell if it's one problem generating the error or five. Hopefully this will help someone not spend hours trying everything under the sun to get past such an error later.