Toolhub/Data model/Research and design

Minimum viable set of attributes:

• The taxonomy should be "just enough" to cover the tools that exist. It should not cover concepts in great detail if the corpus of tools doesn't merit it.
• The taxonomy should not cover attributes that don't require human curation of the attribute values. Information that can be automatically extracted, or which cannot be controlled/curated, is not appropriate for inclusion in the taxonomy, though in many cases it should still be available for cataloging tools or filtering them in the UI.
• The taxonomy should not cover concepts already included in the toolinfo.json schema, and for which data has already been populated. "Tool type" is the prime example here.

Useful attributes only: The taxonomy should not include attributes simply because we *could* model them. The attributes should align with the personas and goals documented in Toolhub use cases. For attributes that we intend to include in the Toolhub browsing UI, there's a limited amount of screen real estate to use. Too many attributes reduces the utility of the browsing UI, and may not display well on small screens. Similarly, attribute values should not exist for concepts that only apply to a small set of tools. For example, attribute values like "photo contests" or "new page patrolling" are too specific to be useful – there are not more than 5 tools that would fit each of those classifications. (Those examples would be valuable as free-text annotations or tags, not as controlled terms).

Modular, atomic concepts: Attributes and their values must not combine multiple facets of a tool into one concept. This is the main area in which v2 of the taxonomy differs from v1. The initially-proposed "use case" attribute combines concepts like "audience" and "task" into one attribute. The principle of "one attribute per concept" ensures that attributes are mutually exclusive, and that attribute values are of the same type (this is sometimes called "level homogeneity"). Meanwhile, this approach still supports complex concepts through combination of multiple attributes (compositionality).

Concrete concepts and definitions: The taxonomy should not cover concepts that will or may evolve over time, like the stability of a tool or its number of active maintainers. While these design principles attempt to avoid ambiguous concepts, there is always some ambiguity in any such system. Consequently, attributes should be documented with clear definitions to prevent semantic drift over time. For example, concepts like "use case" or "tool purpose" can be interpreted many ways. If included, these attributes must be defined with clear semantics that constrain their potential attribute values. This also enables both future data model maintainers and data contributors to easily identify values that don't belong.

This and the following sections describe how User:TBurmeister_(WMF) drafted the taxonomy and controlled vocabulary.

• Revisited the categories proposed from earlier design phases.
• Reviewed Research Phase 1 resources, including multiple pre-Toolhub solutions that had any sort of audience/category/topic segmentation or controlled vocabulary:
  • • Magnus' Tools
    • Bittaker List (2014)
    • SFauconnier List (2017)
    • Commons:Upload Tools
    • w:Wikipedia:User_scripts/List
• Reviewed Toolhub use cases.
• Reviewed Toolhub feature requests / Phabricator tasks around discoverability and metadata.
• Analyzed and added vocab control to unstructured keywords that existed in Toolhub as of 2022 February, many (most) of which came from tool data in Hay's Directory. I limited this analysis to keywords with 3+ occurrences b/c the tail was long.

From all of the above sources, I extracted:

• Concepts people use to talk about tool discovery or navigating a catalog of tools
• Concepts people use to talk about where, when, why, and how they use tools
• Categories and attributes people used when they create lists or catalogs of tools

The result of this concept extraction is an uncontrolled list of concepts (attributes) and attribute values.

Using the "megalist" of uncontrolled concepts, I then analyzed the semantics of the various attributes and their values. As part of this process, I:

• Mapped most low-level concepts to a small set of high-level conceptual themes.
• Reviewed the uncontrolled values and attribute types that these themes clustered together.
• Analyzed the semantics of the various attributes and their values, looking specifically for recurring concepts, overlapping meanings, differing levels of granularity, and different tactics for approaching the same underlying concept.
• Documented common areas of ambiguity that the final taxonomy must address.
• Started building the controlled vocabulary as I encountered clear cases of synonymy.
• Developed a standardized set of attributes. The "Task" attribute required the most standardization, because it combined many values that appeared as different attributes in the various legacy categorization schemes. To make this attribute useful and manageable, I mapped all the uncontrolled task terms to higher-level categories that seek to capture the most common and important themes in the data.

For a shorter list of just the proposed values for the "Task" attribute, or to provide feedback on this item, see the main Data Model page.

(追記) (追記ここまで)

To improve the existing data model, it's important to understand areas where previous attempts to model this space have generated ambiguity or shown inconsistency in how they handled similar concepts. These areas of ambiguity are the most important areas for the new taxonomy to standardize and clarify.

Audience vs. target area / domain vs. wiki project: Previous categorization schemes and data models have often mixed together attribute values related to the people, fields, wikis, wiki projects, and locales for which (or for whom) a tool may be especially useful. The final taxonomy must have a clear definition of what any "Audience", "Domain", or "Project" attribute captures, and those attributes should be clearly mutually exclusive. Examples:

• GLAM, Education → can be modeled as an audience (the humans who are in the GLAM field), but also as a use case or application domain.
• Chinese Wikipedia, English Wikipedia → these examples follow the anti-pattern of combining two concepts, language/locale and specific wiki project (Wikipedia) – into one attribute value.

Subjects, verbs, objects: Previous data models frequently differ in how they model the type of task the tool helps with vs. who it helps vs. with what it helps them do stuff. Intersections of audience, content format, and task were sometimes grouped into a "use cases" attribute to avoid this complexity. Examples:

• "categories" as a content format that a tool acts upon, vs. "Categorizing" as a contributor task. In the data I reviewed, there were 3 instances of noun usage and only one of gerund, so the final taxonomy prefers modeling "categories" as a noun object in an attribute like "Content type".
• "contributing content" vs. "Contributions" vs. "Contributors"(verb vs. noun object vs. noun subject). This set of ambiguous concepts probably suffers from the overall concept being too vague. Better to more specifically model the type of content being contributed, and a more specific set of contributor types or audiences, like "editors", "developers".
• "vandalism" as a content format vs "Patrolling" as a task. In most existing data models, patrolling has high prominence as a task or activity, and "vandalism" can come in many different content formats, so this ambiguity is best resolved by keeping "Patrolling"-related activities as verbs in an attribute focused on user tasks. However, we should also ensure that all tools that involve patrolling have vandalism as a keyword in their description or tags.
  • A similar, but less clear example: is "statistics/metrics" a content format or a task (i.e. "analyzing")? For cases like this, we must be guided by user research, feedback, and data like search queries to determine the most likely way that users would expect to navigate to the relevant tools in this topic area. The current taxonomy uses the verb form and puts "Analysis" under the Task attribute, but we should test and iterate on this.

Audience vs. characteristics of audience vs. characteristics of the UI: Some previous data models highlighted "non-English speakers" as an audience, and many recommended a "language" attribute. The current Toolhub data model includes "available_ui_languages". The taxonomy must clearly differentiate and define the scope of any language- or locale-related attributes, so that it's clear whether they're describing attributes of the tool itself (UI language) or attributes of the audience it's meant to help.

Attributes of the tool itself vs. attributes of its use or context: In previous data models, "tool type" attributes have often conflated specific tool attributes with more generic attributes describing the type of general usage or application the tool has. These should be separate attributes. Examples:

• APIs, bots, coding environments, displays
• Productivity tools, editing tools

Levels of granularity: Past data models and existing, project-specific schemes often vary in how specific they are in capturing the following types of concepts:

• Projects and sub-projects: Wikidata, Structured Data, Commons, Structured Commons → There's limited utility in subdividing projects into sub-projects. Tools that serve these sub-domains of work on structured data can be easily found by combining a Project attribute like "Wikidata'' with a Content Type attribute like "images"' or "media".
• How specific to get in the wiki page structure: because there are scripts that operate on and help with specific subsections or pages, some data models have specified these elements. Is the granularity of "page" vs. "discussion" too much or too little to provide effective access to these tools? The taxonomy must definitely subdivide the tool space by the content types of "articles" vs. "media" vs. "data", so we need to be careful about whether we add too much granularity by modeling the subsections of pages as a content type.
• How specific to get in modeling file types and file formats? Data models or categorization schemes in specific domains have historically been more specific about the content formats most relevant to them. For example, Commons-focused tool descriptions and data models are likely to not just specify image vs. video, they care about whether the file type is svg, jpg, png, tiff, ogg, midi. The question for the Toolhub taxonomy is whether it can support discovery in those specific domains without modeling that level of granularity, and whether there are enough tools in the catalog to make detailed file formats a useful level of granularity.

From a UX perspective, the more shallow the taxonomy is, the better it is likely to work in the current Toolhub UI – there's not a ton of horizontal space for expanding subclasses and showing long lists of possible attributes for filtering.

Nominalizations vs. verb forms: A common inconsistency in past data models appears in how they formalize concepts that can be described with either a noun or a verb form. For example: "content recommenders" is a nominalization of the action "recommending content". The nominalization makes it easy to think of that attribute as a "tool type", and such nominalizations are common when describing bots (like in this article). Maybe we do this because we like to anthropomorphize bots, but fundamentally these types of attributes are describing actions that a user is trying to do with the help of a tool. Our taxonomy should choose one formulation and use it consistently. The proposed taxonomy uses verb forms to model the tasks that a tool helps its users do, rather than using ambiguous nominalizations of such actions.