OFB Cleaner

This is a Python script to take the OFB neighbour survey CSV file, clean the data, and write out a new CSV.

Please see the individual functions in the OFB_Cleaner class to see what is being cleaned.

In addition to those functions, it removes rows with and ID of "NA".

Usage

usage: OFB_cleaner.py [-h] [--input INPUT] [--output OUTPUT]
Clean the OFB data.
options:
 -h, --help show this help message and exit
 --input INPUT Input file path (default: data/neighbour_survey_clean-2024年06月14日.csv)
 --output OUTPUT Output file path (default: neighbour_clean.csv)

Example

Here's an example I used to test a small subset of the data:

python3 OFB_cleaner.py --input data/small.csv --output ./small_clean.csv

Called without arguments, it will look for data/neighbour_survey_clean-2024年06月14日.csv and output to ./neighbour_clean.csv.

Adding Cleaning Of Other Fields

To add cleaning for a new field, declare a new function for it in the OFB_Cleaner class.

For example, to add a cleaner for a column called "Foo", I would create a function in OFB_Cleaner called process_Foo that looks like this:

def process_Foo(self, fields: List[str]) -> List[str]:
 """
 - capitalize occcurances of "bar"
 """
 for index, field in enumerate(fields):
 if field == "bar":
 fields[index] = field.capitalize()
 return fields

This takes a list of fields - the data in the column - and returns a modified list. The list should be the same length (number of rows in the data). The data can be modified in place and returned like I do in the example.

Adding A General Normalization

There is a function in OFB_Cleaner called normalize(). This function is called on every field in the data. (Be careful with this one!)

License

This project is licensed under the MIT license - see the LICENSE file for details.

Individual source files may contain the following tag instead of the full license text:

SPDX-License-Identifier: MIT

Using SPDX enables machine processing of license information based on the SPDX License Identifiers and makes it easier for developers to see at a glance which license they are dealing with.