Rootable's export is modeled after our old system (the robot) which grouped
every task log onto one line per task. For example, these three task logs:
- 50 lbs of Produce: apples and pears
- 100 lbs of Dairy: eggs
- 10 lbs of Bread: community didn't want it, took it to the food bank
would export like:
| item types |
item weights |
item descriptions |
| Produce:Dairy:Bread |
50:100:10 |
apples and pears:eggs:community didn't want it, took it to the food bank |
That information is challenging to disentangle for people wanting to use
spreadsheet formulas, sorting and filtering to understand their data.
So we're splitting it up here to be one row per task log:
| item type |
item weight |
item description |
| Produce |
50 |
apple and pears |
| Dairy |
100 |
eggs |
| Bread |
10 |
community didn't want it, took it to the food bank |
This changes several columns. Plurals are now singular e.g. volunteers is now
volunteer. The total weight column (now total location weight) is on every
row, summing the weight for every log at that location. The the donor and
recipients are merged into a location column because each log is for a
single location.
I tried for a day to make the csv exporter performant, I could not do it with
ActiveRecord alone. So I switched to a database view called task_log_summaries.
It sped up the generation of the csv exports by many, many times; something like
57s for a month of logs down to 350ms.
You never want every task log, so filtering is done by passing in a list ids
from some other model, like event_id, task_id, etc. Time range filtering is
also recomended for building a report. Associations can then be easily
pre-loaded.
The ExportService still needs to act on events, not task logs, because you can
have an event that appears in the export with no task logs. But again, all the
task log associations can be preloaded and it's still fast.
Rootable's export is modeled after our old system (the robot) which grouped
every task log onto one line per task. For example, these three task logs:
- 50 lbs of Produce: apples and pears
- 100 lbs of Dairy: eggs
- 10 lbs of Bread: community didn't want it, took it to the food bank
would export like:
| item types | item weights | item descriptions |
| ------------------- | ------------ | ------------------------------------------------------------------------ |
| Produce:Dairy:Bread | `50:100:10` | apples and pears:eggs:community didn't want it, took it to the food bank |
That information is challenging to disentangle for people wanting to use
spreadsheet formulas, sorting and filtering to understand their data.
So we're splitting it up here to be one row per task log:
| item type | item weight | item description |
| --------- | ----------- | -------------------------------------------------- |
| Produce | 50 | apple and pears |
| Dairy | 100 | eggs |
| Bread | 10 | community didn't want it, took it to the food bank |
This changes several columns. Plurals are now singular e.g. `volunteers` is now
`volunteer`. The `total weight` column (now `total location weight`) is on every
row, summing the weight for every log at that location. The the `donor` and
`recipients` are merged into a `location` column because each log is for a
single location.
I tried for a day to make the csv exporter performant, I could not do it with
ActiveRecord alone. So I switched to a database view called task_log_summaries.
It sped up the generation of the csv exports by many, many times; something like
57s for a month of logs down to 350ms.
You never want _every_ task log, so filtering is done by passing in a list ids
from some other model, like `event_id`, `task_id`, etc. Time range filtering is
also recomended for building a report. Associations can then be easily
pre-loaded.
The `ExportService` still needs to act on events, not task logs, because you can
have an event that appears in the export with no task logs. But again, all the
task log associations can be preloaded and it's still fast.