Fields in Core

edited to add this was posted by David Strauss

My primary interest in having fields in code is the ability to retarget our node storage back-end. (And when I say "nodes," I really mean any object using the fields API.) Relational databases -- at least the open-source ones -- make a poor storage engine for nodes.

This document is the (evolving) plan for the Fields in Core Code Sprint at Acquia from December 15-19, 2008. This is a DRAFT. Many sections are incomplete. Comments encouraged!

This document has been moved to Google Docs for easier online collaboration. The link is:

To view the document: http://docs.google.com/Doc?id=dcz939d6_2gnh78thd
To edit the document: https://docs.google.com/a/acquia.com/Doc?id=dcz939d6_2gnh78thd

Everyone can view the document. People who have been invited as collaborators can edit it. If you want to edit it or if there is a problem, contact Barry Jaspan (firstname dot lastname at acquia dot com).

Below are the notes I already emailed a few people in the group. Those are my review of the kick-off work Barry did in his SVN, along with thoughts on tasks and open questions. Not all of those have to be adressed / answered before we can move forward, this is more of a reminder list.
I'm posting this here so that it's online somewhere instead of in a few mailboxes.

This page is a wiki, so it can be freely edited. Formatting gave me a hard time, there might still be a few glitches here and there.

Good news! During the week of December 15, we're organizing a 5-day Fields in Drupal core code sprint at Acquia! The goal is to get CCK functionality into Drupal 7.

So far, Karen, Yves and Barry have signed up -- Karen and Yves are the main CCK maintainers, and Barry has done a lot of work on CCK as well.

To help us fund the sprint, please consider making a donation using the ChipIn widget on this page. We need money for airline tickets, hotel rooms, food and transportation. It would also be great to fly in a few additional people with extensive core and CCK experience.

I've tentatively worked out a budget of 7,000ドル USD, which covers flight, food and hotel costs for at least four people (Karen, Yves, and two additional people). Since Acquia is covering my travel expenses and allowing Barry to participate all week long, that gives us six people working on CCK-fields-in-core for an entire week. Any excess money will be used to add more people, or donated to the Drupal Association.

To guarantee that Yves and Karen can attend, Acquia is funding Yves' and Karen's hotel and airplane tickets if enough money can't be raised through donations. Acquia is also providing working space in our Andover office.

We'll try to allow people to participate in the sprint remotely, and provide a daily update on our progress. If you're interested and available to participate, join the Fields in Core group, enable e-mail notifications, and block time in your calendar between December 15 and December 19. We'll use the Fields in Core group to plan and to let you know how you can contribute and participate.

With Drupal 6, menu tokens are able to automatically load database records given an object id, this is a great feature, however this requires developers to implement 'load functions' for loading their records. Those 'load functions' are often just a wrapper around 'SELECT * from {mytable} WHERE mykey = %d'.

We could do better, and allow automatically load operations, and even better, enable automatic basic CRUD.

Please see http://drupal.org/node/265604.

Much has been said (http://groups.drupal.org/node/9297) about how fields should be structured in D7 core, what aspects of fields can be changed, and how those changes are implemented. It is (past) time to move forward on implementing fields in core and in this post I am proposing an answer.

Disclaimer: I make a lot of declarative statements in this post. Obviously I do not have unilateral authority; this is just a proposal.

Field storage

This discussion will revolve around issues of Remote Field Revisions. This regards getting field population from single, multiple, and unknown sources. Sending data to other sites and how they can use, modify, and potentially change that data from source. Also regarding race state and origination of data sources and trusted sources for field revision.

Here are my notes from the Future of Fields presentation from this morning.

[Begins showing a picture of the Data Architecture design Sprint including: chx, yched, bjaspan, crell, karens and nedjo]

Drupalcon used to be a developer conference where they would meet and figure out what was going to happen for the next year, but in Barcelona they realized that it was harder to do that.

So they asked about the who are the right people to be involved, and they all flew into Chicago and did a sprint.

The way that fields are structured in CCK now is that any field that is multiple or shared has its own separate 'per field' table, and all other fields are grouped together in a 'per content type' table. Querying this data to create a node can be expensive, so the serialized node is cached and the cached data is used during node_load().

OK, so, it took a lot longer than I expected or planned, but here is the final summary report from the design sprint. I've provided it in both OpenDoc and PDF format. Hopefully there aren't too many spelling errors. :-)

This wiki page is for the development of the Drupalcon session we'll be presenting in Boston. We have 90 minutes on Tuesday morning; see our session page.

Please edit this page only if you attended the Design Sprint and will be participating in the session. Otherwise, please give us your feedback via comments. Thanks!

Session outline

At the recent Data Architecture Design Sprint, participants outlined a development path in which fields can be local or external. A given entity instance (node, user, etc.) can be fully local, fully external, or a combination of the two. For example, a node can have several fields stored and handled in a local SQL database and other fields both loaded from and written to an external SOAP service.

During the Data Architecture Design Sprint (or DADS, as I will henceforth call it), we discussed two general models for reaching our vision of "data anywhere, value-add anything". Those models became dubbed "Model 1" and "Model 2", because that's the order in which we happened to write them down.

Barry has already done a good job of explaining Model 1 in an earlier post. Here, I am going to lay out the structure of Model 2. I suspect that the final system, whever it looks like, will draw heavily from both models.

It's useful to explore how existing core functionality can or cannot be implemented as a field. Doing so may cause us to change how we decide to implement them. In this post I'm talking in terms of nodes but trying to avoid assuming that we are limited to nodes instead of Content objects and that all fields are stored locally in SQL. I'm also mostly just thinking as I go/type; I don't have a specific outcome in mind yet.

Six Drupal developers met for three days in Feburary 2008 to work on and re-design Drupal's core data architecture. We began with an overly broad list of topics and goals and spoke in very general terms about "Drupal's overall mission" and similarly vague concepts. By the end of the design sprint, we devised a tangible vision for Drupal's future and an understandable method for getting there.

Following the design sprint, drawing on what we had discussed, Yves, Karen, and Nedjo tried to map out some of the structure of a core patch and the accompanying short and longer term changes to CCK fields.

Specifically, we discussed the following approach:

• In the immediate term, make small changes as feasible to facilitate longer term changes, e.g., move CCK modules into their own directories in CVS.

We talked a lot about minimum viable functionality (MVF) for D7 core. We also talked briefly about what criteria to use to decide what should be in core vs. contrib. I'd like to revisit that.

Question: Why bother putting fields in core at all?

Being in core has many drawbacks. Much slower development/release cycles, necessity to gain broad consensus for everything, requirement to operate in every possible environment that Drupal supports (e.g. low-memory hosting), etc. We already have CCK and fields in contrib. They work great. Why are we going to incur all this extra pain?

Notes: Comments are now enabled on this page.

CCK simplifications for core

The day started with ways we can simplify current CCK functionality because there is a feeling that what we have is way to fragile and complex for core at this time. Functionality slated for destruction includes:

• Per-content type field storage, e.g. no more content_type_story tables with columns for multiple single-value fields. All fields are stored in today's equivalent of content_field_fieldname tables.
• All field tables always have a delta column even if it is a single-value field.