Blog

Often when things seem really complicated, I think it's because I must be doing it wrong.

Working on Drupal core since dependencies were removed from git has seemed really fiddly. For a long time I thought I must have missed something, that there was some undocumented technique I wasn't aware of.

But I've asked various people who work on core a lot more than I do, and they've confirmed that what I've been doing is pretty much the way that they do it:

1. Get a git clone of Drupal core.
2. Run composer install on it.
3. Write code!
4. Make a patch (well, a merge request now!)

That all sounds simple, right? But wait! If you're working on core, you're going to want Devel module for its useful debugging and inspection tools, right? And Drush, for quick cache clearing. And probably Admin Toolbar so going around the UI is quicker.

But you have to install all those with Composer. And doing that dirties the composer.lock file that's part of Drupal core's git clone.

It's fairly simple to keep changes to that file out of your merge request or patch, but pretty soon, you're going to do a git pull on core that's going to include changes to the composer.lock file, because core will have updates to dependencies.

And that's where it all starts to go wrong, because the git pull will fail because of conflicts in the composer.lock file and in other Composer files, and conflicts in that file are really painful to resolve.

So maintaining an ongoing Drupal install that uses a Drupal core git clone quickly becomes a mess. As far as I know, most core developers frequently reset the whole thing and reinstall from scratch.

The problem is caused by using the git clone of Drupal core as the Composer project, so that Drupal core's composer.json is being used as the project composer.json. But there's a better way...

Using Composer with git clones

Composer has a way of using a git clone for a package in a project:

1. Create your own git clone of the package
2. Declare that git clone as a Composer package repository in the project's composer.json
3. Install the package

The result is that Composer creates a symlink from your git clone into the project, and doesn't touch the git clone. You need to be fairly lax in the version requirement you give for the package, so that Composer doesn't object to the git clone being on a feature branch later on.

This, as far as I know, is the standard way for working on a Composer package that you need to operate in the context of a project. It works for library packages and Composer plugins.

For Drupal core...? Well, it works, but as you might have guessed it's a little more complicated.

Drupal has opinions about where it expects to be located in a project, and furthermore, has a scaffolding system which writes files into the webroot when you install it. All of that gets a bit confused if you put Drupal core out of the way and symlink it in.

But with a few symlinks, and one sneaky patch to a scaffold file, it works. It's all quite fiddly and so I've made...

The Drupal Core development project

This is a Composer project template, available at https://github.com/joachim-n/drupal-core-development-project. It handles all the necessary tweaks to get Drupal to work when symlinked into a project: install it as a Composer project.

There are still a few limitations: those are detailed in the README too.

Of course, I've now fallen down the rabbithole of doing more work towards making Drupal completely agnostic of its location, rather than the core issues I wanted to work on in the first place.

Please try it, report any problems, and happy coding on Drupal core!

A lot of the time, a custom content entity type only needs a single bundle: all the entities of this type have the same structure. But there are times where they need to vary, typically to have different fields, while still being the same type. This is where entity bundles come in.

Bundles are to a custom entity type what node types are to the node entity type. They could be called sub-types instead (and there is a long-running core issue to rename them to this; I'd say go help but I'm not sure what can be done to move it forward), but the name 'bundle' has stuck because initially the concept was just about different 'bundles of fields', and then the name ended up being applied across the whole entity system.

History lesson over; how do you define bundles on your entity type? Well there are several ways, and of course they each have their use cases.

And of course, Module Builder can help generate your code, whichever of these methods you use.

Quick and simple: hardcode

The simplest way to do anything is to hardcode it. If you have a fixed number of bundles that aren't going to change over time, or at least so rarely that requiring a code deployment to change them is acceptable, then you can simply define the bundles in code. The way to do this is with a hook (though there's a core issue -- which I filed -- to allow these to be defined in a method on the entity class).

Here's how you'd define your hardcoded bundles:

/**
 * Implements hook_entity_bundle_info().
 */
function mymodule_entity_bundle_info() {
  $bundles['my_entity_type'] = [
    'bundle_alpha_' => [
      'label' => t('Alpha'),
      'description' => t('Represents an alpha entity.')
    ],
    'bundle_beta_' => [
      'label' => t('Beta'),
      'description' => t('Represents a beta entity.')
    ],
  ];

  return $bundles;
}

The machine names of the bundles (which are used as the bundle field values, and in field names, admin paths, and so on) are the keys of the array. The labels are used in UI messages and page titles. The descriptions are used on the 'Add entity' page's list of bundles.

Classic: bundle entity type

The way Drupal core does bundles is with a bundle entity type. In addition to the content entity type you want to have bundles, there is also a config entity type, called the 'bundle entity type'. Each single entity of the bundle entity type defines a bundle of the content entity type. So for example, a single node type entity called 'page' defines the 'page' node type; a single taxonomy vocabulary entity called 'tags' defines the 'tags' taxonomy term type.

This is great if you want extensibility, and you want the bundles to be configurable by site admins in the UI rather than developers. The downside is that it's a lot of extra code, as there's a whole second entity type to define.

Very little glue is required between the two entity types, though. They basically each need to reference the other in their entity type annotations:

The content entity type needs:

 *   bundle_entity_type = "my_entity_type_bundle",

and the bundle entity needs:

 *   bundle_of = "my_entity_type",

and the bundle entity class should inherit from \Drupal\Core\Config\Entity\ConfigEntityBundleBase.

Per-bundle functionality: plugins as bundles

This third method needs more than just Drupal core: it's a technique provided by Entity module.

Here, you define a plugin type (an annotation plugin type, rather than YAML), and each plugin of that type corresponds to a bundle. This means you need a whole class for each bundle, which seems like a lot of code compared to the hook technique, but there are cases where that's what you want.

First, because Entity module's framework for this allows each plugin class to define different fields for each bundle. These so-called bundle fields are installed in the same way as entity base fields, but are only on one bundle. This gives you the diversification of per-bundle fields that you get with config fields, but with the definition of the fields in your code where it's easier to maintain.

Second, because in your plugin class you can code different behaviours for different bundles of your entity type. Suppose you want the entity label to be slightly different. No problem, in your entity class simply hand over to the bundle:

class PluginAlpha {

  public function label() {
    $bundle_plugin = \Drupal::service('plugin.manager.my_plugin_type')
    return $bundle_plugin->label($this);
  }
  
}

Add a label() method to the plugin classes, and you can specialise the behaviour for each bundle. If you want to have behaviour that's grouped across more than one plugin, one way to do it is to add properties to your plugin type's annotation, and then implement the functionality in the plugin base class with a conditional on the value in the plugin's definition.

/**
 * @MyPluginType(
 *   id = "plugin_alpha",
 *   label = @Translation("Alpha"),
 *   label_handling = "combolulate"

class MyPluginBase {

  public function label() {
    switch ($this->getPluginDefinition()['floopiness']) {
      case 'combolulate':
        // Return the combolulated label. 
    }
  }

}

For a plugin type to be used as entity bundles, the plugins need to implement \Drupal\entity\BundlePlugin\BundlePluginInterface, and your entity type needs to declare the plugin:

 *   bundle_plugin_type = "my_plugin_type",

Here, the string 'my_plugin_type' is the part of the plugin manager's service name that comes after the 'plugin.manager' prefix. (The plugin system unfortunately doesn't have a standard concept of a plugin type's name, but this is informally what is used in various places such as Entity module and Commerce.)

The bundle plugin technique is well-suited to entities that are used as 'machinery' in some way, rather than content. For example, in Commerce License, each bundle of the license entity is a plugin which provides the behaviour that's specific to that type of license.

One thing to note on this technique: if you're writing kernel tests, you'll need to manually install the bundle plugins for them to exist in the test. See the tests in Entity module for an example.

These three techniques for bundles each have their advantages, so it's not unusual to find all three in use in a single project codebase. It's a question of which is best for a particular entity type.

Lazy builders were introduced in Drupal 8's render system to solve the problem of pieces of your page that have a low cacheability affecting the cacheability of the whole page. Typical uses of this technique are with entire render arrays which are handed over to a lazy builder.

However, in retrofitting an existing site to make use of caching more effectively, I found that we have a large amount of content that would be cacheable were it not for a single string. Specifically, we have a DIV of the details for a product, but also in that DIV there is the serial number for the user's specific instance of that product.

The whole of this product DIV was output by a Twig template. I could have ripped this up and remade it as a big render array, putting the lazy builder in for the serial number, but this seemed like a lot of work, and would also make our front-end developers unhappy.

It turned out there was another way. The render system deals with placeholders for lazy builders automatically, so you typically will just do:

$build['uncacheable_bit'] = [
  '#lazy_builder' => [
    'my_service:myLazyBuilder',
    [],
  ],
];

but as I learned from a blog post by Borisson, you can make the placeholders yourself put them anywhere in the content and attach the lazy builders to the build array. This is what the render system does when it handles a '#lazy_builder' builder render array item.

One application of this is that you can pass the placeholder as a parameter to a twig template:

// Your placeholder can be anything, but using a hash like the render system
// does prevents accidental replacements.
$placeholder = Crypt::hashBase64('some_data');

$build['my_content'] = [
  '#theme' => 'my_template',
  '#var_1' => $var_1,
  // This is just a regular parameter as far as the Twig template cares, 
  // output with {{lazy_builder_var}} as normal.
  '#lazy_builder_var' => $placeholder,
];

$build['#attached']['placeholders'][$placeholder] = [
  '#lazy_builder' => [
    // The lazy builder callback and parameters.
    'my_service:myLazyBuilder',
    [],
  ],
];

With this approach, the Twig template just needed some minor changes. It doesn't care about the lazy builder; all it sees is the placeholder string which it treats like any other text. The render system finds the placeholder in the '#attached' attribute, and handles the replacement when the rendered template is retrieved from the render cache.

The result is that we're able to cache much more in the render cache, and improve site performance.

Now I just have to figure out the cache tags...

All fields are equal in Drupal 8! Back in Drupal 7 we had two different systems for data on entities, which worked fairly differently: entity properties that were defined as database fields and controlled by hardcoded form elements, and user-created fields on entities that had widgets and formatters. But now in Drupal 8, every value on an entity is a field, with the same configuration in the UI, access to the widgets and formatters, and whose data is accessed in the same way.

Is this the end of the story? No, not quite. For site builders, everything is unified, and that's great. For code that consumes data from entities, everything is unified, and that's great too. But for the developer actually working with those fields, whether a field is a base field or a config field still makes a big difference.

Config fields are easier...

Config fields are manipulated via a UI, and that makes them simple to work with.

Setup

Config fields win easily here: a few clicks in the UI to create the field, a few options to choose in dropdowns for the widget and formatter options and you're set. Export to config, commit: done!

Base fields are more fiddly here. You need to be familiar with the Entity system, and understand (or copy-pasta!) the base field definitions. Then, knowing the right widget and formatter to use and how to set their options requires close reading of the relevant plugin classes.

Update

Config fields win again: change in the UI, export to config, deploy!

For base fields, you once again need to know what you're doing with the code, and you now need a hook_update_N() to make the change to the database (now that entity updates are longer done automatically).

Removal

This is maybe a minor point, but while removing a config field is again just a UI click and a config export, removing a base field will again require a hook_update_N() to tell the Entity API to update the database tables.

...but base fields are more robust

Given the above, why bother with base fields? Well, if your fields just sit there holding data, you can stop reading. But if your fields are involved in your custom code, as a developer, this should give you an unpleasant feeling: your code in modules/custom/mymodule is dependent on configuration that lives in the site's config export.

Robustness

The fact that config fields are so easy to change in the UI is now a mark against them. Another developer could change a field without a full understanding of the code that works with it. The code might expect a value to always be there, but the field is now non-required. The code might expect certain options, but the field now has an extra one.

Instead of this brittle dependency, it feels much safer to have base fields defined closer to the code that makes use of them: in the same module.

Tests

The solution to the dependency problem is obviously tests, which would pick up any change that breaks the code's expectations of how the fields behave. But now we hit a big problem with config fields: your test site doesn't have those fields!

My first attempt at solving this problem was to use the Configuration development module. This allows you to export config from the site to a module's /config/install folder. A test that installs that module then gets those config items imported.

It's a quick and simple approach: when a test crashes because of a missing field, find it in the config folder, add it to the right module's .info.yml file, do drush cde MODULE and commit the changes and the newly created files.

This approach also works for all other sorts of config your test might need: taxonomy vocabularies, node types, roles, and more!

But now you have an additional maintenance burden because your site config is now in three places: the site itself, the config export, and now also in module config. And if developers who change config forget to also export to module config, your test is now no longer testing how the site actually works, and so will either fail, or worse, won't be actually covering what you expect any more.

Best of both worlds: import from config

To summarize: we want the convenience of config fields, but we want them to be close to our code and testable. If they're testable, we can maybe stand to forego the closeness.

My best solution to this so far is simple: allow tests to import configuration direct from the site config sync folder.

Here's the helper method I've been using:

/**
 * Imports config from a real local site's config folder.
 *
 * TODO: this currently only supports config entities.
 *
 * @param string $site_key
 *   The site key, that is, the subfolder in /config in which to find the
 *   config files.
 * @param string $config_name
 *   The config name. This is the same as the YML filename without the
 *   extension. Note that this is not checked for dependencies.
 *
 * @throws \Exception
 *   Throws an exception if the config can't be imported.
 */
protected function importSiteConfig(string $site_key, string $config_name) {
  $storage = $this->container->get('config.storage.sync');

  // Code cribbed from Config Devel module's ConfigImporterExporter.
  $config_filename = "../config/{$site_key}/{$config_name}.yml";

  if (!file_exists($config_filename)) {
    throw new \Exception("Unable to find config file $config_filename to import from.");
  }

  $contents = @file_get_contents($config_filename);

  $data = $storage->decode($contents);
  if (!$data) {
    throw new \Exception("Failed to import config $config_name from site $site_key.");
  }

  // This assumes we only ever import entities from site config for tests,
  // which so far is the case.
  $entity_type_id = $this->container->get('config.manager')->getEntityTypeIdByName($config_name);

  if (empty($entity_type_id)) {
    throw new \Exception("Non-entity config import not yet supported!");
  }

  $entity = $this->container->get('entity_type.manager')->getStorage($entity_type_id)->create($data);
  $entity->save();
}

Use it in your test's setUp() or test methods like this:

  // Import configuration from the default site config files.
  $this->importSiteConfig('default', 'field.storage.node.field_my_field');
  $this->importSiteConfig('default', 'field.field.node.article.field_my_field');

As you can see from the documentation, this so far only handles config that is entities. I've not yet had a use for importing config that's not an entity (and just about all config items are entities except for the ones that are a collection of a module's settings). And it doesn't check for dependencies: you'll need to import them in the right order (field storage before field) and ensure the modules that provide the various things are enabled in the test.

I should mention for the sake of completeness that there's another sort of field, sort of: bundle fields. These are in code like base fields, but limited to a particular bundle. They also have a variety of problems as the system that support them is somewhat incomplete.

Finally, it occurs to me that another way to bridge the gap would be to allow editing base fields in the UI, and then export that back to code consisting of the BaseFieldDefinition calls. But hang on... haven't I just reinvented Drupal 7-era Features?

UPDATE

It turned out this was crashing when importing fields with value options. This code (and that in Config Devel module, which is where I cribbed it from) wasn't properly using the Config API. Here's updated code, which incidentally, also now can handle non-entity config:

  /**
   * Imports config from a real local site's config folder.
   *
   * Needs config module to be enabled.
   *
   * @param string $site_key
   *   The site key, that is, the subfolder in /config in which to find the
   *   config files.
   * @param string $config_name
   *   The config name. This is the same as the YML filename without the
   *   extension.
   *
   * @throws \Exception
   *   Throws an exception if the config can't be imported.
   */
  protected function importSiteConfig(string $site_key, string $config_name) {
    $storage = $this->container->get('config.storage.sync');

    // Code cribbed from Config Devel module's ConfigImporterExporter.
    $config_filename = "../config/{$site_key}/{$config_name}.yml";

    if (!file_exists($config_filename)) {
      throw new \Exception("Unable to find config file $config_filename to import from.");
    }

    $contents = @file_get_contents($config_filename);

    $data = $storage->decode($contents);
    if (!$data) {
      throw new \Exception("Failed to import config $config_name from site $site_key.");
    }

    unset($data['uuid']);

    // We have to partially mock the source storage, because otherwise
    // SystemConfigSubscriber::onConfigImporterValidateSiteUUID() will complain
    // because the source config doesn't contain a system.site config. (For
    // general information, the single-import UI in core doesn't hit this
    // problem because it TOTALLY cheats and pretends its source is the full
    // site config! Though who knows how it manages not to trip up when the site
    // config folder is empty!)
    $source_storage = $this->getMockBuilder(StorageReplaceDataWrapper::class)
      ->setConstructorArgs([$this->container->get('config.storage')])
      ->setMethods(['exists'])
      ->getMock();
    // Satisfy SystemConfigSubscriber that we have a system.site config.
    $source_storage->expects($this->any())
      ->method('exists')
      ->willReturn(TRUE);

    $source_storage->replaceData($config_name, $data);

    // Similarly mock the storage comparer.
    $storage_comparer = $this->getMockBuilder(StorageComparer::class)
      ->setConstructorArgs([$source_storage, $this->container->get('config.storage')])
      ->setMethods(['validateSiteUuid'])
      ->getMock();
    // Satisfy SystemConfigSubscriber that system.site config is valid.
    $storage_comparer->expects($this->any())
      ->method('validateSiteUuid')
      ->willReturn(TRUE);

    $storage_comparer->createChangelist();

    $config_importer = new ConfigImporter(
      $storage_comparer->createChangelist(),
      $this->container->get('event_dispatcher'),
      $this->container->get('config.manager'),
      $this->container->get('lock'),
      $this->container->get('config.typed'),
      $this->container->get('module_handler'),
      $this->container->get('module_installer'),
      $this->container->get('theme_handler'),
      $this->container->get('string_translation'),
      $this->container->get('extension.list.module')
    );

    $config_importer->import();
  }

}

UPDATE 2

I've found an occasional problem with site config items that have third-party settings from modules that aren't relevant to the test. Examples include Menu UI settings in node types, and contrib translation settings in fields. Rather than enable extra modules in the test, these settings can be hacked out of the data after we decode it from the YAML. Here's an updated version of the code.

use Drupal\Core\Config\ConfigImporter;
use Drupal\Core\Config\StorageComparer;
use Drupal\config\StorageReplaceDataWrapper;

  /**
   * Imports config from a real local site's config folder.
   *
   * @param string $site_key
   *   The site key, that is, the subfolder in /config in which to find the
   *   config files.
   * @param string $config_name
   *   The config name. This is the same as the YML filename without the
   *   extension.
   * @param array $third_party_filter
   *   (optional) Keys in the config item's third party settings that are to be
   *   removed prior to import. This prevents unwanted dependencies on modules
   *   that are not relevant to a test. Defaults to an empty array; no keys
   *   removed.
   *
   * @throws \Exception
   *   Throws an exception if the config can't be imported.
   */
  protected function importSiteConfig(string $site_key, string $config_name, array $third_party_filter = []) {
    $storage = $this->container->get('config.storage.sync');

    // Code cribbed from Config Devel module's ConfigImporterExporter.
    $config_filename = "../config/{$site_key}/{$config_name}.yml";

    if (!file_exists($config_filename)) {
      throw new \Exception("Unable to find config file $config_filename to import from.");
    }

    $contents = @file_get_contents($config_filename);

    $data = $storage->decode($contents);
    if (!$data) {
      throw new \Exception("Failed to import config $config_name from site $site_key.");
    }

    unset($data['uuid']);

    // Remove third-party settings.
    foreach ($third_party_filter as $settings_key) {
      unset($data['third_party_settings'][$settings_key]);
    }

    // We have to partially mock the source storage, because otherwise
    // SystemConfigSubscriber::onConfigImporterValidateSiteUUID() will complain
    // because the source config doesn't contain a system.site config. (For
    // general information, the single-import UI in core doesn't hit this
    // problem because it TOTALLY cheats and pretends its source is the full
    // site config! Though who knows how it manages not to trip up when the site
    // config folder is empty!)
    $source_storage = $this->getMockBuilder(StorageReplaceDataWrapper::class)
      ->setConstructorArgs([$this->container->get('config.storage')])
      ->setMethods(['exists'])
      ->getMock();
    // Satisfy SystemConfigSubscriber that we have a system.site config.
    $source_storage->expects($this->any())
      ->method('exists')
      ->willReturn(TRUE);

    $source_storage->replaceData($config_name, $data);

    // Similarly mock the storage comparer.
    $storage_comparer = $this->getMockBuilder(StorageComparer::class)
      ->setConstructorArgs([$source_storage, $this->container->get('config.storage')])
      ->setMethods(['validateSiteUuid'])
      ->getMock();
    // Satisfy SystemConfigSubscriber that system.site config is valid.
    $storage_comparer->expects($this->any())
      ->method('validateSiteUuid')
      ->willReturn(TRUE);

    $storage_comparer->createChangelist();

    $config_importer = new ConfigImporter(
      $storage_comparer->createChangelist(),
      $this->container->get('event_dispatcher'),
      $this->container->get('config.manager'),
      $this->container->get('lock'),
      $this->container->get('config.typed'),
      $this->container->get('module_handler'),
      $this->container->get('module_installer'),
      $this->container->get('theme_handler'),
      $this->container->get('string_translation'),
      $this->container->get('extension.list.module')
    );

    $config_importer->import();
  }

The hardest thing I find with tests is understanding errors. Every time I think I've got debugging output sorted, I find a new layer where it doesn't work, I've got nothing, and I'm in the dark with something that's crashing and I don't know why.

The first layer is simple: errors in your test code itself. For example, make a typo in your tests/src/Functional/MyTest.php and PHPUnit crashes and you see the error in the terminal.

But when it's site code that's crashing, you're dealing with a system that is being driven by code, and therefore, you can't see it. And that's a major obstacle to figuring out a problem.

The HTML output that Drupal's Functional and Functional Javascript tests produce is a huge help: every time your test code makes a request to the test site, an HTML file is written to the test files directory. If your site crashes when your test makes a request, you'll see the error and the backtrace there.

However, there's no such output when in a Functional Javascript test you cause an AJAX request. And while you can create a screenshot of what the page looks like after the request, our another HTML file of the page (see https://www.drupal.org/project/drupal/issues/3090498 for instructions how; the issue is about how to make that automatic but I have no idea how that might be possible), you can't see the actual error, because AJAX requests that fail just sit there doing nothing. There's nothing useful to see in the browser.

So we need to see logs. When a real site has an AJAX crash, with a human being-controlled web browser making the request, you can go and look in the logs. With a test site, the log table is zapped when the test is completed.

Fortunately, Drupal 8's pluggable logging means there are other ways of getting hold of them, more permanent ways.

I first tried log_stdout module. This outputs log errors to STDOUT. If you're running on Docksal, as I am, you have an extra layer to get though to see that. You can monitor the cli container with fin logs -f cli, and with that module add a | ag WATCHDOG to filter.

However, I wasn't seeing backtrace in this output, and I gave up figuring out why.

So I tried filelog module instead, which as the name implies, writes log to a simple text file. This needs a little bit more work, as by default it writes to 'public://logs'. This means that each run of the test gets its own log file, which is perhaps what you want, but for my own uses I wanted a single log file I could tail -f in a terminal window and have continual monitoring.

A quick bit of config setting in the test's setUp() does the trick:

$this->config('filelog.settings')
  ->set('location', '/var/www/docroot/sites/simpletest/logs')
  ->save();

And I think that's me at last sorted.