Optimizing river network features queries in ArcGIS JavaScript SDK

Question

I am working on a map to visualize river network data using ArcGIS SDK for JavaScript. The data I’m using is from this ArcGIS feature service published by the CLMS. The data includes a dozen feature layers, each corresponding to a Strahler number of the rivers systems, with fields such as OBJECT_ID, upstream and downstream nodes IDs (and more).

Objective: When a river is clicked, I want to highlight all upstream rivers on the map to visualize the patterns formed by all streams of a common drainage basin.

Approach:

• I use JavaScript code with the ArcGIS SDK to display the data on a map and highlight features when clicked.
• My code involves querying the upstream features based on the clicked feature and its upstream node, and iterating the query to reach all upstream features of the river system.

(EDIT: I have added below my question a complete minimal code example as asked in comments, instead of just a short snippet showing the heart of the approach).

Performance issue: The current approach is functioning, but for large rivers, it is very slow. When clicking near the mouth of a ~50 km river, it takes about 20 seconds to find all features to highlight, but when clicking a ~300 km river it takes about 15 minutes. The queries are performed on LayerViews (not the full Layers) to optimize the process, which limits the scope to visible features, but performance remains an issue.

Question: Is there a way to speed up the process of querying upstream features?

I believe the performance bottleneck might be the repeated query inside a while loop, but I can’t find a way to avoid it. Since flow direction is important, I can’t only use a geospatial query (for example querying all adjacent features), as I need to take features attributes (up- and downstream nodes) into account.

I am thinking of maybe writing a small backend service in python, which I am more familiar with, to handle these queries, and possibly of creating a graph structure for the data. But that would involve downloading and pre-processing the whole dataset.

Code

Here is the full functioning code as requested in comments (stripped of a couple of unnecessary things, to obtain a complete minimal reproducible example). For the moment this requires an ArcGIS API key to work.

import esriConfig from '@arcgis/core/config.js';
import Map from '@arcgis/core/Map.js';
import MapView from '@arcgis/core/views/MapView.js';
import FeatureLayer from '@arcgis/core/layers/FeatureLayer.js';
import Handles from '@arcgis/core/core/Handles.js';
esriConfig.apiKey = "XXX";
const LAYERS_IDS = ['5', '6', '7', '8', '9', '10', '11', '12', '13', '15', '16', '17', '18'];
const SERVICE_URL = 'https://image.discomap.eea.europa.eu/arcgis/rest/services/EUHydro/EUHydro_RiverNetworkDatabase/MapServer'; // eslint-disable-line max-len -- URL is too long
// Create Map and MapView
const map = new Map({basemap: 'arcgis/topographic'});
const view = new MapView({
 map: map,
 center: [-1.5, 43.5],
 zoom: 8,
 container: 'mapDiv',
});
// Add layers to Map
const riverLayers = []; // Initiate array of river layers for access in click event
const riverLayerViews = [];
LAYERS_IDS.forEach((layerId) => {
 const url = `${SERVICE_URL}/${layerId}`;
 const riverLayer = new FeatureLayer({
 url: url,
 outFields: ['CatchID', 'FNODE', 'nameText', 'OBJECT_ID', 'TNODE'],
 });
 map.add(riverLayer);
 riverLayers.push({id: layerId, layer: riverLayer});
 view.whenLayerView(riverLayer).then((layerView) => {
 riverLayerViews.push({id: layerId, layerView: layerView});
 });
});
// Declare variables to handle features highlighting
let clickedFeatureId = '';
const highlightHandles = new Handles();
// Define click event to interact with the riverLayers
view.on('immediate-click', (event) => {
 // Restrict click events to riverLayers
 const opts = {include: riverLayers.map((riverLayer) => riverLayer.layer)};
 view.hitTest(event, opts).then(
 async (response) => {
 if (response.results.length) { // Check if a feature was clicked
 const feature = response.results[0].graphic;
 const objectId = feature.attributes.OBJECT_ID;
 if (clickedFeatureId != objectId) { // Check if new feature was clicked
 // Read feature information
 const layerId = feature.layer.layerId.toString();
 const upstreamNode = feature.attributes.FNODE;
 clickedFeatureId = objectId;
 // Remove previous highlight
 if (highlightHandles.has()) {
 highlightHandles.removeAll();
 }
 // Find all upstream features
 const upstreamFeatures = await queryUpstreamFeatures(
 layerId, objectId, upstreamNode, riverLayerViews,
 );
 // Highlight upstream features
 highlightFeatures(upstreamFeatures, riverLayerViews, highlightHandles);
 }
 } else { // If no feature was clicked, remove highlight
 highlightHandles.removeAll();
 clickedFeatureId = '';
 }
 },
 );
});
/** Query upstream features. The list of features returned includes the feature given as input. */
async function queryUpstreamFeatures(layerId, objectId, baseNode, riverLayerViews) {
 const upstreamFeatures = [{layerId: layerId, objectId: objectId}];
 let nodes = [baseNode];
 while (nodes.length > 0) {
 const newNodes = [];
 const promises = [];
 nodes.forEach((node) => {
 riverLayerViews.forEach((riverLayerView) => {
 const query = riverLayerView.layerView.createQuery();
 query.where = `TNODE = '${node}'`;
 const promise = riverLayerView.layerView.queryFeatures(query).then((results)=>{
 results.features.forEach((feature)=>{
 const attributes = feature.attributes;
 upstreamFeatures.push({layerId: riverLayerView.id, objectId: attributes.OBJECT_ID});
 newNodes.push(attributes.FNODE);
 });
 });
 promises.push(promise);
 });
 });
 await Promise.all(promises);
 nodes = newNodes;
 }
 return upstreamFeatures;
}
/** Highlight a list of features. */
async function highlightFeatures(upstreamFeatures, riverLayerViews, highlightHandles) {
 upstreamFeatures.map(async (feature) =>{
 const riverLayerView = riverLayerViews.find(
 (layerView) => layerView.id === feature.layerId,
 ).layerView;
 const query = riverLayerView.createQuery();
 query.where = `OBJECT_ID = '${feature.objectId}'`;
 const ids = await riverLayerView.queryObjectIds(query);
 highlightHandles.add(riverLayerView.highlight(ids));
 });
}

• TomazicM: I would like to have a look at this. Please add code for riverLayerViews and highlightFeatures to the question.
• tvoirand: @TomazicM I just added the rest of the code.
• TomazicM: OK, I'll have a look.
• TomazicM: Just to let you know I'm still working on it. Just when I thought I got it, I discovered features/layers behave strangely when zooming out lower than zoom 8 and then zooming back in. I'm looking for a way to solve this now.
• tvoirand: Thanks for your interest and help. Regarding zoom, just so you know, my code also includes a refresh of the highlighted features when the view changes (queryUpstreamFeatures and highlightFeatures are called when view.stationary === true). I did not include in my question in order to give a minimal example of the performance issue. As a side note, I currently intend to apply the solutions suggested by @Hornbydd, of investigating trace network, or building a graph structure. I just haven't taken the time to do so yet. But if you find another solution it would also be welcome.

Answer 1

Solution below is more a proof of concept than a bullet proof solution. It's just to illustrate one possible approach.

It's based on the following logic:

• Since there were some problems querying features/nodes from all feature layers at the same time when zooming in/out, layers are divided in two layer groups: first from '5' to '13' and second from '15' to '18. Second group is used for zooms up to 7 including, first group is used for zoom 8 and higher zooms.
• When river node is clicked, all features/nodes from active group are saved in features[selectedGroupInd] object, where keys are OBJECT_ID and value is array consisting of feature and its view. For tree walk object tnodes[selectedGroupInd] is created, where keys are TNODE and value is an array of feature OBJECT_ID.
• Then keys of tnodes are recursively traveled, starting with selected node, building and array of features/node in the selected tree. During the building of the tree all map zoom/pan actions are disabled.
• Finally selected features are highlighted.
• When river tree is highlighted and map is zoomed/panned, object features[selectedGroupInd] and tnodes[selectedGroupInd] are updated with possible new features from views and then node tree walked again.

Here is complete code:

<!DOCTYPE html>
<html lang="en">
<head>
<title>ArcGIS API for JavaScript: EUHydro/EUHydro_RiverNetworkDatabase</title>
<meta charset="utf-8">
<meta name="viewport" content="initial-scale=1, maximum-scale=1,user-scalable=no">
<link href="https://js.arcgis.com/4.14/esri/css/main.css" rel="stylesheet" type="text/css" />
<link rel="stylesheet" href="https://cdnjs.cloudflare.com/ajax/libs/font-awesome/4.7.0/css/font-awesome.min.css">
<script type="text/javascript" src="https://js.arcgis.com/4.29/"></script>
<style>
 html,
 body {
 padding: 0;
 margin: 0;
 height: 100%;
 width: 100%;
 }
 #viewDiv {
 float: left;
 padding: 0;
 margin: 0;
 height: 100%;
 width: 100%;
 }
 #workingDiv {
 position: absolute;
 top: 50%;
 left: 50%;
 margin-right: -50%;
 transform: translate(-50%, -50%);
 color: rgba(0, 169, 230, 0.9);
 text-align: center;
 background-color: transparent;
 }
</style>
<script type="text/javascript">
 require([
 "esri/Map",
 "esri/views/MapView",
 "esri/layers/GroupLayer",
 "esri/layers/FeatureLayer",
 "esri/Graphic",
 "esri/core/Handles",
 "esri/core/reactiveUtils"
 ], (
 Map, MapView, GroupLayer, FeatureLayer, Graphic, Handles, reactiveUtils
 ) => {
 const LAYERS_IDS = ['5', '6', '7', '8', '9', '10', '11', '12', '13', '15', '16', '17', '18'];
 const SERVICE_URL = 'https://image.discomap.eea.europa.eu/arcgis/rest/services/EUHydro/EUHydro_RiverNetworkDatabase/MapServer'; // eslint-disable-line max-len -- URL is too long
 const GROUP1_MIN_ZOOM = 8;
 
 var selectedGroupInd = 0;
 var groupLayer = [];
 groupLayer[0] = new GroupLayer();
 groupLayer[1] = new GroupLayer();
 const map = new Map({
 basemap: "topo-vector"
 });
 const view = new MapView({
 container: "viewDiv",
 map: map,
 center: [-1.5, 43.5],
 zoom: 8
 });
 LAYERS_IDS.forEach((layerId, i) => {
 const url = `${SERVICE_URL}/${layerId}`;
 const riverLayer = new FeatureLayer({
 url: url,
 outFields: ['CatchID', 'FNODE', 'nameText', 'OBJECT_ID', 'TNODE'],
 });
 if (i <= 8) {
 groupLayer[0].add(riverLayer);
 }
 else {
 groupLayer[1].add(riverLayer);
 }
 });
 map.add(groupLayer[0]);
 map.add(groupLayer[1]);
 let clickedFeatureId = null;
 const highlightHandles = new Handles();
 var features = [];
 var tnodes = [];
 var visited = [];
 var forHighlight = [];
 
 function highlightRecursive(featureId) { 
 updateWorking();
 var selectedFeature = features[selectedGroupInd][featureId];
 if (!selectedFeature || visited[selectedGroupInd][featureId]) return;
 
 visited[selectedGroupInd][featureId] = true;
 forHighlight.push([selectedFeature[0], selectedFeature[1]]);
 var nodes = tnodes[selectedGroupInd][selectedFeature[0].attributes.FNODE];
 if (!nodes) return;
 nodes.forEach(function(nodeId) {
 if (features[selectedGroupInd][nodeId]) {
 setTimeout(function() {
 highlightRecursive(nodeId);
 }, 0);
 }
 });
 }
 view.on('immediate-click', (event) => {
 const opts = {include: groupLayer[selectedGroupInd].layers.toArray()};
 view.hitTest(event, opts).then(async (response) => {
 if (response.results.length && response.results[0].graphic.attributes.OBJECT_ID) {
 const feature = response.results[0].graphic;
 const objectId = feature.attributes.OBJECT_ID;
 if (clickedFeatureId != objectId) {
 const layerId = feature.layer.layerId.toString();
 const upstreamNode = feature.attributes.FNODE;
 clickedFeatureId = objectId;
 
 if (highlightHandles.has()) {
 highlightHandles.removeAll();
 }
 for (var i = 0; i <= 1; i++) {
 features[i] = {};
 tnodes[i] = {};
 visited[i] = {};
 }
 
 var nDone = 0;
 showWorking();
 view.layerViews.items[selectedGroupInd].layerViews.forEach((riverLayerView) => {
 riverLayerView.queryFeatures().then((result)=>{
 result.features.forEach(feature => {
 features[selectedGroupInd][feature.attributes.OBJECT_ID] = [feature, riverLayerView];
 if (!tnodes[selectedGroupInd][feature.attributes.TNODE]) tnodes[selectedGroupInd][feature.attributes.TNODE] = [];
 tnodes[selectedGroupInd][feature.attributes.TNODE].push(feature.attributes.OBJECT_ID);
 });
 nDone++;
 if (view.layerViews.items[selectedGroupInd].layerViews.length == nDone) {
 initWorking(true);
 setTimeout(function() {
 highlightRecursive(clickedFeatureId);
 }, 100);
 }
 });
 });
 }
 } else {
 highlightHandles.removeAll();
 clickedFeatureId = null;
 }
 });
 });
 var currZoom = null;
 var currWidth = null;
 var currHeight = null;
 var currLatitude = null;
 var currLongitude = null;
 
 reactiveUtils.watch(
 () => [view.stationary, view.ready, view.updating, view.zoom, view.width, view.height, view.center.latitude, view.center.longitude],
 ([stationary, ready, updating, zoom, width, height, latitude, longitude]) => {
 if (!ready || !stationary || updating) return;
 
 var processChange = false;
 
 if (zoom != currZoom) {
 if (currZoom != null) processChange = true;
 currZoom = zoom;
 selectedGroupInd = (currZoom >= GROUP1_MIN_ZOOM) ? 0 : 1;
 }
 if (width != currWidth) {
 if ((currWidth != null) && (width > currWidth)) processChange = true;
 currWidth = width;
 }
 if (height != currHeight) {
 if ((currHeight != null) && (height > currHeight)) processChange = true;
 currHeight = height;
 }
 if (latitude != currLatitude) {
 if (currLatitude != null) processChange = true;
 currLatitude = latitude;
 }
 if (longitude != currLongitude) {
 if (currLongitude != null) processChange = true;
 currLongitude = longitude;
 }
 if (processChange && clickedFeatureId) {
 visited[selectedGroupInd] = {};
 var nDone = 0;
 showWorking();
 view.layerViews.items[selectedGroupInd].layerViews.forEach((riverLayerView) => {
 riverLayerView.queryFeatures().then((result) => {
 result.features.forEach(feature => {
 if (!features[selectedGroupInd][feature.attributes.OBJECT_ID]) {
 features[selectedGroupInd][feature.attributes.OBJECT_ID] = [feature, riverLayerView];
 if (!tnodes[selectedGroupInd][feature.attributes.TNODE]) tnodes[selectedGroupInd][feature.attributes.TNODE] = [];
 tnodes[selectedGroupInd][feature.attributes.TNODE].push(feature.attributes.OBJECT_ID);
 }
 });
 nDone++;
 if (view.layerViews.items[selectedGroupInd].layerViews.length == nDone) {
 initWorking(true);
 setTimeout(function() {
 highlightRecursive(clickedFeatureId);
 }, 0);
 }
 });
 });
 }
 }
 );
 
 var viewEventHandlers = [];
 
 function stopEvtPropagation(event) {
 event.stopPropagation();
 }
 
 function disablePanAndZoom() {
 viewEventHandlers.push(view.on("mouse-wheel", stopEvtPropagation));
 viewEventHandlers.push(view.on("double-click", stopEvtPropagation));
 viewEventHandlers.push(view.on("double-click", ["Control"], stopEvtPropagation));
 viewEventHandlers.push(view.on("drag", stopEvtPropagation));
 viewEventHandlers.push(view.on("drag", ["Shift"], stopEvtPropagation));
 viewEventHandlers.push(view.on("drag", ["Shift", "Control"], stopEvtPropagation));
 viewEventHandlers.push(view.on("key-down", (event) => {
 const prohibitedKeys = ["+", "-", "Shift", "_", "=", "ArrowUp", "ArrowDown", "ArrowRight", "ArrowLeft"];
 const keyPressed = event.key;
 if (prohibitedKeys.indexOf(keyPressed) !== -1) {
 stopEvtPropagation(event);
 }
 }));
 }
 function enablePanAndZoom() {
 viewEventHandlers.forEach(handler => {
 handler.remove();
 });
 viewEventHandlers = [];
 }
 
 var workingActive = false;
 
 function updateWorking() {
 workingActive = true;
 }
 
 function showWorking() {
 document.getElementById('workingDiv').style.display = 'block';
 disablePanAndZoom();
 }
 
 function initWorking(init) {
 if (init) {
 showWorking();
 updateWorking();
 }
 if (workingActive || view.updating) {
 workingActive = view.updating;
 setTimeout(function() {
 initWorking();
 }, 200);
 }
 else {
 document.getElementById('workingDiv').style.display = 'none';
 enablePanAndZoom();
 forHighlight.forEach(feature => {
 highlightHandles.add(feature[1].highlight(feature[0]));
 });
 forHighlight = [];
 }
 }
 });
</script>
</head>
<body>
 <div id="viewDiv"></div>
 <div id="workingDiv" style="display: none;">
 <i class="fa fa-spinner fa-spin" style="font-size: 42px"></i>
 </div>
</body>
</html>

And here is JSFiddle: https://jsfiddle.net/TomazicM/7sL1o9ng/

• tvoirand: Sorry for the long delay in my answer. Your solution is really helpful, thank you! If I understand correctly, you first query all features visible in the view, and then build the tree based on the clicked feature, instead of recursively querying features based on their downstream node as in my first approach. This makes the process a lot more efficient, without requiring to preprocess the dataset.
• TomazicM: Yes, only features that visible in the view are queried.

Answer 2

I'm the author behind RivEX so know a little about accessing river networks. I've never used the ArcGIS SDK for JavaScript. If you are using a cursor to read features from a layer "one at a time" as you traverse upstream then yes that does not scale well as the catchment becomes larger.

You have several options:

• move the featurelayer into a network data structure. The Trace network is optimised for rivers and the sort of querying you are interested in, but I have to admit I don't know if that is accessible through some web service?
• build the graph as you have suggested and use that, again I would not know how to access it via ArcGIS SDK as I'm not a web developer.

• tvoirand: Thank you for your insights. The first option looks like the optimal solution, but I’m not sure if it’s feasible since I’m just accessing the layers through a feature service published by the CLMS and over which I don’t have control. I might go with the second option, applying it only to a subset of the data if necessary (after all I am just playing with this for now, although it’s tempting to dive deeper after having taken a look at RivEX, it looks great, congratulations for building this!)