SJ cartoon avatar

Development Your First Pants Plugin

I recently wrote my first Pants plugin (currently in a draft PR, waiting for me to add some tests and cleanup and it's released in 2.10) and the process was both easier and more difficult than I expected.

I don't mean the complexity of writing the plugin per se, as when you’re creating a plugin for an existing large-scale system or process, there is a fair amount of pre-work you’ll need to do. But, I guess I was mostly surprised by where the time was taken.

To quote myself from the Pants Slack channel:

I spent a fair amount of time reading through the docs, the example-plugin repo, and then pants source to just figure out where to start. Once I was "started" (had something that could compile/run and do-nothing), everything kinda made more sense and I have a clear path forward where I should put code, how it all relates, and what to do next.

My hope in this post is to ease the “figure out where to start” burden for anyone creating their first plugin, and to highlight a few of the gotcha’s that I ran into along the way.

Disclaimer

In writing this, I’m going to assume you know what Pants is and how to use it. I’m going to dive into how to extend Pants to suit your needs, with a simple-ish plugin based on the one I recently wrote. Please note, that was my first Pants plug-in - so I might not have nailed everything here. Even before reading this, a good place to start might be the comprehensive documentation.

Resources

You’ll quickly notice that the documentation takes you “gently into the deep end” to paraphrase Josh Cannon from the Slack channel. The documentation is very detailed and technical, thus may be slightly intimidating for new users when creating their first plugin.

The existing plugins are great resources because they showcase what you’ll actually need to create and replicate at the end of the day. As I was writing my PyOxidizer plugin, the existing plugins I referred to the most were: Black, DocFormatter, and isort - because I know how those tools work and their functionality/usage is quite simple.

However, since I was adding a new target, I needed more resources. This led me to looking at the pex_binary target code, which is pretty complicated - and also the Docker plugin code. This code is also pretty complicated in its current incarnation, because the Docker plugin has so much functionality.

My hope here is that using my trivial made-up plugin as an example, I can showcase an even simpler Pants plugin and explain in detail why certain things are where they are (and if they’re even needed in the first place).

The common plugin tasks documentation is helpful if you have a linter or formatter tool to run over your code. My PyOxidizer plugin was slightly trickier, as I wanted to use the output of a package goal as the input to my target. Most of the existing plugin examples assume you’re running a tool over all your source files or performing code generation. After having written the plugin, I now realize that it’s not THAT different in the grand scheme, but when I was getting started, I was very lost.

Additional Context

While writing this post, I am intentionally glossing over some Pants jargon and terminology to simplify writing - otherwise things will get whacky and you might need a glossary at hand.

I am also attempting to follow folder and file-naming conventions as they're laid out in the documentation for consistency. I slip away from it from time to time, though.

Finally, I've created a GitHub repo (https://github.com/sureshjoshi/pants-example-plugin) where the main branch represents the final result and I've kept branches open which roughly correspond with each major section of work. You can follow along by reviewing the various commits to the main branch - or by comparing branches at any point.

Anywho, let’s get to it.

Getting Started

https://github.com/sureshjoshi/pants-example-plugin/tree/01-getting-started

The best place to start building would be via an in-repo plugin, so let’s get started there.

# pants.toml:

[GLOBAL]
pants_version = "2.9.0"
pythonpath = ["pants-plugins"] # Let Pants know that we want to reference modules from within here
print_stacktrace = true # You'll want this enabled - trust me

backend_packages = [
    "pants.backend.plugin_development", # Adds the \`pants_requirements\` target type
    "pants.backend.python",
]

[anonymous-telemetry]
enabled = false

# pants-plugins/BUILD:

pants_requirements(name="pants")

Compared to a “regular” pants.toml, there's not much to see here. We’ve added a pythonpath to point at where we’ll develop our in-repo plugins and added another backend_package to allow us to actually perform plugin development.

Running ./pants —-version will grab Pants and nothing much else will happen.

Registering Our Plugin

https://github.com/sureshjoshi/pants-example-plugin/tree/02-registering-a-plugin

I know we don’t technically have a plugin yet, so it’s weird that we’re going to register it already - but this is actually a good time to do it, since the registration code won’t change too much and this is the absolute fastest way to know you’re on the right track.

But first, it’s hard to show an example of how to write plugins without ... well... an example. A lot of the documentation shows examples where plugins operate on Python sources, so, to do something different - let’s create a plugin that takes in the output of the python_distribution target. More specifically, let’s expect to receive a Python wheel and then copy that wheel file into a new one appended with a .foo extension.

I call it... Fooify

In the repo, I am nesting this plugin under the experimental folder because if it ever gets pulled upstream, that’s probably where it will end up. There is no programmatic need to do this, but I also think fooify looks more aesthetically pleasing in pants.toml under a namespace.

# pants.toml:

backend_packages = [
    "pants.backend.plugin_development", # Adds the \`pants_requirements\` target type
    "pants.backend.python",
    "experimental.fooify",
]

# pants-plugins/experimental/fooify/register.py:

# This registers all of the plugin hooks that action our sources/dependencies
def rules():
    return []

# This registers our new plugin's target(s) (used in BUILD files)
def target_types():
    return []

Run ./pants —version again to make sure there are no compilation/build errors and we’re off to the races!

Our First Plugin

https://github.com/sureshjoshi/pants-example-plugin/tree/03-adding-the-fooify-target

Okay, time to get serious. Let’s add our awesome fooify target (which will end up being used in the BUILD files) and see what happens.

# pants-plugins/experimental/fooify/register.py:

class FooifyTarget(Target):
    alias = "fooify"
    core_fields = (
        *COMMON_TARGET_FIELDS,
    )
    help = "The \`fooify\` target will take in a wheel dependency and add a .foo extension to the end."


def target_types():
    return [FooifyTarget]

Now that we’ve registered our FooifyTarget in the target_types array, we can run the following:

./pants help fooify

\`fooify\` target
---------------

The \`fooify\` target will take in a wheel dependency and add a .foo extension to the end.


Valid fields:

description
    type: str | None
    default: None
    A human-readable description of the target.

    Use \`./pants list --documented ::\` to see all targets with descriptions.

tags
    type: Iterable[str] | None
    default: None
    Arbitrary strings to describe a target.

    For example, you may tag some test targets with 'integration_test' so that you could run \`./pants --tag='integration_test' test ::\` to only run on targets with that tag.

Sweet! We’ve now created our first Pants plugin and target! It doesn’t do anything, but it’s fun to look at!

Refactoring

https://github.com/sureshjoshi/pants-example-plugin/tree/04-refactoring

It's a bit strange that the module we’re using to register our plugin also is defining targets, and will soon be defining rules, and activating the plugin... I have a feeling that if we try to stuff everything in this file, things will get nutty and unmaintainable... Let’s be good developers and split that out ahead of time so that we don’t get hit with technical debt down the road.

Out of convention, you’ll see the following four files in most plugins: register.py, rules.py, subsystem.py, target_types.py:

  • register.py: This file is responsible for activating your plugin and telling Pants where to look for rules and targets
  • subsystem.py: Global configurations and options related to your plugin are here (not always required - but very nice to have)
  • rules.py: This is where the magic sauce happens - here you define what you want the plugin to actually do
  • target_types.py: Here is where you define your brand new target and the fields it expects
    • It is also convention to create a new class per target field - this is about customizability, but also about discoverability (discussed later)

We don’t need all of those now, so we’ll just separate out the target from the registration.

# pants-plugins/experimental/fooify/target_types.py:

class FooifyTarget(Target):
    alias = "fooify"
    core_fields = (
        *COMMON_TARGET_FIELDS,
    )
    help = "The \`fooify\` target will take in a wheel dependency and add a .foo extension to the end."

Adding a Sample Project

https://github.com/sureshjoshi/pants-example-plugin/tree/05-adding-a-sample-project

Before we go too far down the plugin rabbit hole, we should come back a step and notice that even if we made the most amazing plugin in the world - we’d never know, since there is no code to run it on. So, now seems like a good time to create a sample project that I’ll uniquely call “hello world”.

# helloworld/helloworld/main.py:

def main():
    print("Hello, world!")

if __name__ == "__main__":
    print("Launching HelloWorld from __main__")
    main()

When using in-repo plugins, you can continue using your mono-repo structuring at the root level (or however you do it, frankly). I’ve structured this example the way I tend to do it which doesn’t necessarily line up with how Pants does in their documentation.

# pants.toml:

[source]
root_patterns = [
    "helloworld",
]

# helloworld/BUILD:

python_sources(name="libhelloworld", sources=["**/*.py"])

python_distribution(
    name="helloworld-dist",
    dependencies=[":libhelloworld"],
    wheel=True,
    sdist=False,
    provides=python_artifact(
        name="helloworld-dist",
        version="0.0.1",
        description="A distribution for the hello world library.",
    ),
)

I also tend not to use the silent defaults for the BUILD files (even though they’re perfectly usable) because I find that sometimes I lose the thread when I’m going through all of my files looking for mistakes or errors.

You can test if the new project works by packaging a wheel and ensuring that there is a dist/helloworld_dist-0.0.1-py3-none-any.whl file in your folder. Run the following:

./pants package helloworld:helloworld-dist

Adding Dependencies to a Target

https://github.com/sureshjoshi/pants-example-plugin/tree/06-adding-a-dependencies-field

Going back to what we get out of ./pants help fooify - thanks to COMMON_TARGET_FIELDS, we can see that the description and tags fields were added to our target, so we could use those right now in a BUILD file if we wanted to. But that’s not what we want. We want to take in the output of another target (a wheel) - so we’ll need to make sure our target can take in a dependency.

# pants-plugins/experimental/fooify/target_types.py:

class FooifyTarget(Target):
    alias = "fooify"
    core_fields = (
        *COMMON_TARGET_FIELDS,
        Dependencies,
    )
    help = "The \`fooify\` target will take in a wheel dependency and add a .foo extension to the end."

There we go - easy peasy.

Let’s add the fooify target to our helloworld/BUILD file and then take a dependency on our python_distribution which is set to create a wheel of our source code.

# helloworld/BUILD:

fooify(
    name="helloworld-foo",
    dependencies=[":helloworld-dist"],
)

And now let’s run the target!

Running the Target

https://github.com/sureshjoshi/pants-example-plugin/tree/07-running-the-target

... wait, how do we actually “run” the target?

Whelp, you need to tell Pants what your end goal is with that target, so you need to decide what your goal will be. You can create a new goal, but practically, you'll want to select one of the pre-existing goals.

We’re not formatting sources, we're not linting sources, we're not REPLing... I think the most reasonable option is “package”.

./pants package helloworld:helloworld-foo

Hmm, that didn’t work... ... ... Let’s check what our packaging target options are.

./pants list package

  * archive
  * pex_binary
  * python_distribution

Oh! We should probably tell Pants that we want to link our new fooify target with the package command. To do that, we're going to need some rules - so let's start adding some in the new rules.py file (and make sure to register them).

# pants-plugins/experimental/fooify/register.py:

def rules():
    return [*fooify_rules()]

# pants-plugins/experimental/fooify/rules.py:

@rule(level=LogLevel.DEBUG)
async def run_fooify() -> BuiltPackage:
    # Make a rule that can compile safely
    empty_digest = await Get(Digest, CreateDigest())
    return BuiltPackage(
        digest=empty_digest,
        artifacts=(),
    )

def rules():
    return [*collect_rules()]

The structure of a rule is a pure async function. For the package goal, our output seems to reasonably be a BuiltPackage. How did I find this out? I looked at the rules for pex_binary and python_distribution for ideas.

Okay, after trying to package fooify against, it's still not even running. Hmmm, we’re just trying to make a do-nothing rule, what could go wrong...

Well, Pants needs a bit more info on how running the package goal lines up with my brand new target. This is where things get a little magical, and frankly, I don’t entirely understand it myself.

Create a new FieldSet class with our list of required dependencies, and make a UnionRule with PackageFieldSet so that our custom object can register with the package goal.

# pants-plugins/experimental/fooify/target_types.py:

@dataclass(frozen=True)
class FooifyFieldSet(PackageFieldSet):
    required_fields = (Dependencies,)

    dependencies: Dependencies

def rules():
    return [*collect_rules(), UnionRule(PackageFieldSet, FooifyFieldSet)]

Re-running the package list - whoa, what happened? There are so many items.

./pants list package

  * archive
  * file
  * files
  * fooify
  * pex_binary
  * python_distribution
  * python_requirement
  * python_source
  * python_sources
  * python_test
  * python_test_utils
  * python_tests
  * resource
  * resources
  * target

Dependencies Gotcha!

https://github.com/sureshjoshi/pants-example-plugin/tree/08-fixing-dependencies-gotcha

You have to be careful when using required_fields in a class that is passed into a UnionRule. From Andreas on Slack:

you want to have a field unique to your target to put in required fields for your fields set, to limit down what targets you get

So, in target_types.py, let’s create a class that subclasses Dependencies, and use that in the FieldSet. This change should limit us to just our target.

# pants-plugins/experimental/fooify/target_types.py:

class FooifyDependenciesField(Dependencies):
    pass

class FooifyTarget(Target):
    alias = "fooify"
    core_fields = (
        *COMMON_TARGET_FIELDS,
        FooifyDependenciesField,
    )
    help = "The \`fooify\` target will take in a wheel dependency and add a .foo extension to the end."

# pants-plugins/experimental/fooify/rules.py:

@dataclass(frozen=True)
class FooifyFieldSet(PackageFieldSet):
    required_fields = (FooifyDependenciesField,)

    dependencies: FooifyDependenciesField

./pants list package

  * archive
  * fooify
  * pex_binary
  * python_distribution

Ahh, that looks better.

Running ./pants package helloworld:helloworld-foo succeeds, but doesn’t actually do anything yet - because there are no rules setup.

Subsystem Interlude

https://github.com/sureshjoshi/pants-example-plugin/tree/09-subsystems

A little (long) while ago I mentioned the idea of a subsystem and that it is used for configurations and options related to your plugin. Before we jump into rules, this feels like a good time to do a pre-emptive refactor and setup a subsystem.

There’s really not much to do here though - but it’s nice to have for later. If you were writing a plugin that called out to a Python tool (like Black, isort, PyOxidizer, etc) - you could place default versions of the tools, lockfiles, Python interpreter constraints, etc here.

# pants-plugins/experimental/fooify/subsystem.py:

class Fooify(PythonToolBase):
    options_scope = "fooify"
    help = """The Fooify utility for adding .foo to a wheel."""

def rules():
    return [*collect_rules()]

# # pants-plugins/experimental/fooify/register.py:

def rules():
    return [*fooify_rules(), *subsystem.rules()]

For the sake of seeing what happens with the help, let’s comment out the target_types array in register.py and re-run help.

./pants help fooify

Unknown entity: fooify
pants.base.exceptions.BackendConfigurationError: Failed to load the experimental.fooify.register backend:
    ImportError("cannot import name 'FooifyTarget' from 'experimental.fooify' (/pants-example-plugin/pants-plugins/experimental/fooify/**init**.py)")

Gotcha!

The fooify keyword doesn’t work anymore (on 2.9.0 anyways). The reason is that Pants doesn’t register subsystems unless they’re used in rules. It's not a big deal, but can be hard to debug if you don't know what you're looking for. I created an issue about it, since I ran into this when running a code snippet in the documentation and burned a decent chunk of time trying to figure out what I did wrong.

If you did hook it up to a rule, you would see:

./pants help fooify

\`fooify\` subsystem options
--------------------------

The Fooify utility for adding .foo to a wheel.

Config section: [fooify]

None available.

Either way, this doesn’t actually affect us, so uncomment the target_types in register.py and let’s move on to the rules.

Rules

https://github.com/sureshjoshi/pants-example-plugin/tree/10-rules

Here we go, the meat and potatoes of your plugin - everything else is just window dressing (or, maybe salad dressing?).

Because this plugin is so trivial, there will only be one rule - but that’s not a limitation, just how things shook out.

First off, let’s add the subsystem to your rule - because you might want to extract some of your configurations or options out of it (not in this case - since our subsystem does nothing, but who knows). Oh, one thing I forgot to mention in the previous section - you can’t name the target and subsystem with the same external name, so I’ll be re-naming our target alias fooify_distribution for clarity.

Next, let’s add your field_set in, because we’re going to want the fooify_distribution target dependency (i.e. a wheel).

# pants-plugins/experimental/fooify/rules.py:

@rule(level=LogLevel.DEBUG)
async def run_fooify(fooify: Fooify, field_set: FooifyFieldSet) -> BuiltPackage:
    # Make a rule that can compile safely
    empty_digest = await Get(Digest, CreateDigest())
    return BuiltPackage(
        digest=empty_digest,
        artifacts=(),
    )

If we were doing something more sophisticated, this is probably where we’d pip install our dependencies and run them on our source files or wheels or whatever. But, this is the Fooify plugin - it has one purpose only. To take in the wheel dependency and copy it with a .foo appended...

# pants-plugins/experimental/fooify/rules.py:

@rule(level=LogLevel.DEBUG)
async def run_fooify(fooify: Fooify, field_set: FooifyFieldSet) -> BuiltPackage:
    logger.info(f"Incoming field set: {field_set}")

    # What is Get?
    # https://www.pantsbuild.org/docs/rules-api-concepts#await-get---awaiting-results-in-a-rule-body
    targets = await Get(Targets, DependenciesRequest(field_set.dependencies))

    # NOTE: This is hardcoded for this example
    wheel_target = targets[0]

    # All of the following is to eventually unwrap to get the wheel file itself
    packages = await Get(
        FieldSetsPerTarget,
        FieldSetsPerTargetRequest(PackageFieldSet, [wheel_target]),
    )
    wheel_field_set = packages.field_sets[0]
    wheel_package = await Get(BuiltPackage, PackageFieldSet, wheel_field_set)

    # What is a Digest?
    # https://www.pantsbuild.org/docs/rules-api-file-system#core-abstractions-digest-and-snapshot

    # Grab the Wheel file entry, and re-created it with the .foo extension
    digest_entries = await Get(DigestEntries, Digest, wheel_package.digest)
    wheel_file_entry = digest_entries[0]
    assert isinstance(wheel_file_entry, FileEntry)
    foo_file_entry = FileEntry(path=wheel_file_entry.path + ".foo", file_digest=wheel_file_entry.file_digest)
    fooified = await Get(Digest, CreateDigest([foo_file_entry]))

    # Ensure the new file is correctly logged during the build process
    artifact = BuiltPackageArtifact(relpath=foo_file_entry.path)

    # Ensure the fooified file is output to dist/ and
    return BuiltPackage(
        digest=fooified,
        artifacts=(artifact,),
    )

This all looks pretty complicated - but it’s mostly wrapping and unwrapping data representations. I won’t delve into it, as I’ve added links to the associated documentation which explains it all much, MUCH better than I ever could.

That’s It!

And there we go. We’ve created a trivial Pants plugin, ran into and resolved a few Gotchas along the way, and have a bit of a scaffold of what future plugins would look like.

In a subsequent post, I’ll tackle the challenge of testing plugins (which I personally find trickier than creating them).