access arrow data

Question

I am fetching arrow tabular data from an API. The content-type of the response is application/vnd.apache.arrow.stream. In my react code, I am trying to read the response as such.

I am not sure if this is the optimized way of reading arrow data.

function extractDataFromTable(table) {
 let extractedData = [];
 
 // Iterate over each row of the table
 for (const row of table) {
 const rowJson = row.toJSON();
 if (rowJson.origin_lat instanceof Float64Array) {
 rowJson.origin_lat = Array.from(rowJson.origin_lat)[0]; // Assuming only one element in the array
 }
 if (rowJson.origin_lon instanceof Float64Array) {
 rowJson.origin_lon = Array.from(rowJson.origin_lon)[0]; // Assuming only one element in the array
 }
 
 extractedData.push(rowJson);
 }
 
 return extractedData;
 }
const response = await axios.get(`${baseURL}/api/new_drives`, {
 params: {
 name: cname,
 },
 responseType: "arraybuffer",
 });
 
 // Convert the response data to an Apache Arrow table
 const buffer = new Uint8Array(response.data);
 const table = tableFromIPC(buffer);
 
 // Process the table to extract your data
 const extractedData = extractDataFromTable(table); 
 data.setApiData(extractedData);

Is there a better and faster way of reading arrow data?

Answer 1

no JSON

 const rowJson = row.toJSON();

faster way of reading arrow data?

That .toJSON() call looks like it's more expensive than necessary.

Didn't arrow hand you a vector of floats? Store a bunch of small integers and then go xamine memory in gdb to verify.

Can't you get a pointer into that array? Or memcpy() the values into your own array?

 extractedData.push(rowJson);

That similarly looks more expensive than storing a float and bumping an index.