Development Your First Pants Plugin
I recently wrote my first Pants plugin (currently in a draft PR, waiting for me to add some tests and cleanup and it's released in 2.10) and the process was both easier and more difficult than I expected.
I don't mean the complexity of writing the plugin per se, as when you’re creating a plugin for an existing large-scale system or process, there is a fair amount of pre-work you’ll need to do. But, I guess I was mostly surprised by where the time was taken.
To quote myself from the Pants Slack channel:
I spent a fair amount of time reading through the docs, the example-plugin repo, and then pants source to just figure out where to start. Once I was "started" (had something that could compile/run and do-nothing), everything kinda made more sense and I have a clear path forward where I should put code, how it all relates, and what to do next.
My hope in this post is to ease the “figure out where to start” burden for anyone creating their first plugin, and to highlight a few of the gotcha’s that I ran into along the way.
Disclaimer
In writing this, I’m going to assume you know what Pants is and how to use it. I’m going to dive into how to extend Pants to suit your needs, with a simple-ish plugin based on the one I recently wrote. Please note, that was my first Pants plug-in - so I might not have nailed everything here. Even before reading this, a good place to start might be the comprehensive documentation.
Resources
You’ll quickly notice that the documentation takes you “gently into the deep end” to paraphrase Josh Cannon from the Slack channel. The documentation is very detailed and technical, thus may be slightly intimidating for new users when creating their first plugin.
The existing plugins are great resources because they showcase what you’ll actually need to create and replicate at the end of the day. As I was writing my PyOxidizer plugin, the existing plugins I referred to the most were: Black, DocFormatter, and isort - because I know how those tools work and their functionality/usage is quite simple.
However, since I was adding a new target, I needed more resources. This led me to looking at the pex_binary target code, which is pretty complicated - and also the Docker plugin code. This code is also pretty complicated in its current incarnation, because the Docker plugin has so much functionality.
My hope here is that using my trivial made-up plugin as an example, I can showcase an even simpler Pants plugin and explain in detail why certain things are where they are (and if they’re even needed in the first place).
The common plugin tasks documentation is helpful if you have a linter or formatter tool to run over your code. My PyOxidizer plugin was slightly trickier, as I wanted to use the output of a package goal as the input to my target. Most of the existing plugin examples assume you’re running a tool over all your source files or performing code generation. After having written the plugin, I now realize that it’s not THAT different in the grand scheme, but when I was getting started, I was very lost.
Additional Context
While writing this post, I am intentionally glossing over some Pants jargon and terminology to simplify writing - otherwise things will get whacky and you might need a glossary at hand.
I am also attempting to follow folder and file-naming conventions as they're laid out in the documentation for consistency. I slip away from it from time to time, though.
Finally, I've created a GitHub repo (https://github.com/sureshjoshi/pants-example-plugin) where the main
branch represents the final result and I've kept branches open which roughly correspond with each major section of work. You can follow along by reviewing the various commits to the main branch - or by comparing branches at any point.
Anywho, let’s get to it.
Getting Started
https://github.com/sureshjoshi/pants-example-plugin/tree/01-getting-started
The best place to start building would be via an in-repo plugin, so let’s get started there.
# pants.toml:
[GLOBAL]
pants_version = "2.9.0"
pythonpath = ["pants-plugins"] # Let Pants know that we want to reference modules from within here
print_stacktrace = true # You'll want this enabled - trust me
backend_packages = [
"pants.backend.plugin_development", # Adds the \`pants_requirements\` target type
"pants.backend.python",
]
[anonymous-telemetry]
enabled = false
# pants-plugins/BUILD:
pants_requirements(name="pants")
Compared to a “regular” pants.toml, there's not much to see here. We’ve added a pythonpath
to point at where we’ll develop our in-repo plugins and added another backend_package
to allow us to actually perform plugin development.
Running ./pants —-version
will grab Pants and nothing much else will happen.
Registering Our Plugin
https://github.com/sureshjoshi/pants-example-plugin/tree/02-registering-a-plugin
I know we don’t technically have a plugin yet, so it’s weird that we’re going to register it already - but this is actually a good time to do it, since the registration code won’t change too much and this is the absolute fastest way to know you’re on the right track.
But first, it’s hard to show an example of how to write plugins without ... well... an example. A lot of the documentation shows examples where plugins operate on Python sources, so, to do something different - let’s create a plugin that takes in the output of the python_distribution
target. More specifically, let’s expect to receive a Python wheel and then copy that wheel file into a new one appended with a .foo
extension.
I call it... Fooify
In the repo, I am nesting this plugin under the experimental
folder because if it ever gets pulled upstream, that’s probably where it will end up. There is no programmatic need to do this, but I also think fooify
looks more aesthetically pleasing in pants.toml
under a namespace.
# pants.toml:
backend_packages = [
"pants.backend.plugin_development", # Adds the \`pants_requirements\` target type
"pants.backend.python",
"experimental.fooify",
]
# pants-plugins/experimental/fooify/register.py:
# This registers all of the plugin hooks that action our sources/dependencies
def rules():
return []
# This registers our new plugin's target(s) (used in BUILD files)
def target_types():
return []
Run ./pants —version
again to make sure there are no compilation/build errors and we’re off to the races!
Our First Plugin
https://github.com/sureshjoshi/pants-example-plugin/tree/03-adding-the-fooify-target
Okay, time to get serious. Let’s add our awesome fooify
target (which will end up being used in the BUILD files) and see what happens.
# pants-plugins/experimental/fooify/register.py:
class FooifyTarget(Target):
alias = "fooify"
core_fields = (
*COMMON_TARGET_FIELDS,
)
help = "The \`fooify\` target will take in a wheel dependency and add a .foo extension to the end."
def target_types():
return [FooifyTarget]
Now that we’ve registered our FooifyTarget
in the target_types
array, we can run the following:
./pants help fooify
\`fooify\` target
---------------
The \`fooify\` target will take in a wheel dependency and add a .foo extension to the end.
Valid fields:
description
type: str | None
default: None
A human-readable description of the target.
Use \`./pants list --documented ::\` to see all targets with descriptions.
tags
type: Iterable[str] | None
default: None
Arbitrary strings to describe a target.
For example, you may tag some test targets with 'integration_test' so that you could run \`./pants --tag='integration_test' test ::\` to only run on targets with that tag.
Sweet! We’ve now created our first Pants plugin and target! It doesn’t do anything, but it’s fun to look at!
Refactoring
https://github.com/sureshjoshi/pants-example-plugin/tree/04-refactoring
It's a bit strange that the module we’re using to register our plugin also is defining targets, and will soon be defining rules, and activating the plugin... I have a feeling that if we try to stuff everything in this file, things will get nutty and unmaintainable... Let’s be good developers and split that out ahead of time so that we don’t get hit with technical debt down the road.
Out of convention, you’ll see the following four files in most plugins: register.py
, rules.py
, subsystem.py
, target_types.py
:
register.py
: This file is responsible for activating your plugin and telling Pants where to look for rules and targetssubsystem.py
: Global configurations and options related to your plugin are here (not always required - but very nice to have)rules.py
: This is where the magic sauce happens - here you define what you want the plugin to actually dotarget_types.py
: Here is where you define your brand new target and the fields it expects- It is also convention to create a new class per target field - this is about customizability, but also about discoverability (discussed later)
We don’t need all of those now, so we’ll just separate out the target from the registration.
# pants-plugins/experimental/fooify/target_types.py:
class FooifyTarget(Target):
alias = "fooify"
core_fields = (
*COMMON_TARGET_FIELDS,
)
help = "The \`fooify\` target will take in a wheel dependency and add a .foo extension to the end."
Adding a Sample Project
https://github.com/sureshjoshi/pants-example-plugin/tree/05-adding-a-sample-project
Before we go too far down the plugin rabbit hole, we should come back a step and notice that even if we made the most amazing plugin in the world - we’d never know, since there is no code to run it on. So, now seems like a good time to create a sample project that I’ll uniquely call “hello world”.
# helloworld/helloworld/main.py:
def main():
print("Hello, world!")
if __name__ == "__main__":
print("Launching HelloWorld from __main__")
main()
When using in-repo plugins, you can continue using your mono-repo structuring at the root level (or however you do it, frankly). I’ve structured this example the way I tend to do it which doesn’t necessarily line up with how Pants does in their documentation.
# pants.toml:
[source]
root_patterns = [
"helloworld",
]
# helloworld/BUILD:
python_sources(name="libhelloworld", sources=["**/*.py"])
python_distribution(
name="helloworld-dist",
dependencies=[":libhelloworld"],
wheel=True,
sdist=False,
provides=python_artifact(
name="helloworld-dist",
version="0.0.1",
description="A distribution for the hello world library.",
),
)
I also tend not to use the silent defaults for the BUILD files (even though they’re perfectly usable) because I find that sometimes I lose the thread when I’m going through all of my files looking for mistakes or errors.
You can test if the new project works by packaging a wheel and ensuring that there is a dist/helloworld_dist-0.0.1-py3-none-any.whl
file in your folder. Run the following:
./pants package helloworld:helloworld-dist
Adding Dependencies to a Target
https://github.com/sureshjoshi/pants-example-plugin/tree/06-adding-a-dependencies-field
Going back to what we get out of ./pants help fooify
- thanks to COMMON_TARGET_FIELDS
, we can see that the description
and tags
fields were added to our target, so we could use those right now in a BUILD file if we wanted to. But that’s not what we want. We want to take in the output of another target (a wheel) - so we’ll need to make sure our target can take in a dependency.
# pants-plugins/experimental/fooify/target_types.py:
class FooifyTarget(Target):
alias = "fooify"
core_fields = (
*COMMON_TARGET_FIELDS,
Dependencies,
)
help = "The \`fooify\` target will take in a wheel dependency and add a .foo extension to the end."
There we go - easy peasy.
Let’s add the fooify
target to our helloworld/BUILD
file and then take a dependency on our python_distribution
which is set to create a wheel of our source code.
# helloworld/BUILD:
fooify(
name="helloworld-foo",
dependencies=[":helloworld-dist"],
)
And now let’s run the target!
Running the Target
https://github.com/sureshjoshi/pants-example-plugin/tree/07-running-the-target
... wait, how do we actually “run” the target?
Whelp, you need to tell Pants what your end goal is with that target, so you need to decide what your goal will be. You can create a new goal, but practically, you'll want to select one of the pre-existing goals.
We’re not formatting sources, we're not linting sources, we're not REPLing... I think the most reasonable option is “package”.
./pants package helloworld:helloworld-foo
Hmm, that didn’t work... ... ... Let’s check what our packaging target options are.
./pants list package
* archive
* pex_binary
* python_distribution
Oh! We should probably tell Pants that we want to link our new fooify
target with the package
command. To do that, we're going to need some rules - so let's start adding some in the new rules.py
file (and make sure to register them).
# pants-plugins/experimental/fooify/register.py:
def rules():
return [*fooify_rules()]
# pants-plugins/experimental/fooify/rules.py:
@rule(level=LogLevel.DEBUG)
async def run_fooify() -> BuiltPackage:
# Make a rule that can compile safely
empty_digest = await Get(Digest, CreateDigest())
return BuiltPackage(
digest=empty_digest,
artifacts=(),
)
def rules():
return [*collect_rules()]
The structure of a rule is a pure async function. For the package goal, our output seems to reasonably be a BuiltPackage. How did I find this out? I looked at the rules for pex_binary
and python_distribution
for ideas.
Okay, after trying to package fooify against, it's still not even running. Hmmm, we’re just trying to make a do-nothing rule, what could go wrong...
Well, Pants needs a bit more info on how running the package
goal lines up with my brand new target. This is where things get a little magical, and frankly, I don’t entirely understand it myself.
Create a new FieldSet
class with our list of required dependencies, and make a UnionRule
with PackageFieldSet
so that our custom object can register with the package
goal.
# pants-plugins/experimental/fooify/target_types.py:
@dataclass(frozen=True)
class FooifyFieldSet(PackageFieldSet):
required_fields = (Dependencies,)
dependencies: Dependencies
def rules():
return [*collect_rules(), UnionRule(PackageFieldSet, FooifyFieldSet)]
Re-running the package list - whoa, what happened? There are so many items.
./pants list package
* archive
* file
* files
* fooify
* pex_binary
* python_distribution
* python_requirement
* python_source
* python_sources
* python_test
* python_test_utils
* python_tests
* resource
* resources
* target
Dependencies Gotcha!
https://github.com/sureshjoshi/pants-example-plugin/tree/08-fixing-dependencies-gotcha
You have to be careful when using required_fields
in a class that is passed into a UnionRule
.
From Andreas on Slack:
you want to have a field unique to your target to put in required fields for your fields set, to limit down what targets you get
So, in target_types.py
, let’s create a class that subclasses Dependencies
, and use that in the FieldSet
. This change should limit us to just our target.
# pants-plugins/experimental/fooify/target_types.py:
class FooifyDependenciesField(Dependencies):
pass
class FooifyTarget(Target):
alias = "fooify"
core_fields = (
*COMMON_TARGET_FIELDS,
FooifyDependenciesField,
)
help = "The \`fooify\` target will take in a wheel dependency and add a .foo extension to the end."
# pants-plugins/experimental/fooify/rules.py:
@dataclass(frozen=True)
class FooifyFieldSet(PackageFieldSet):
required_fields = (FooifyDependenciesField,)
dependencies: FooifyDependenciesField
./pants list package
* archive
* fooify
* pex_binary
* python_distribution
Ahh, that looks better.
Running ./pants package helloworld:helloworld-foo
succeeds, but doesn’t actually do anything yet - because there are no rules setup.
Subsystem Interlude
https://github.com/sureshjoshi/pants-example-plugin/tree/09-subsystems
A little (long) while ago I mentioned the idea of a subsystem and that it is used for configurations and options related to your plugin. Before we jump into rules, this feels like a good time to do a pre-emptive refactor and setup a subsystem.
There’s really not much to do here though - but it’s nice to have for later. If you were writing a plugin that called out to a Python tool (like Black, isort, PyOxidizer, etc) - you could place default versions of the tools, lockfiles, Python interpreter constraints, etc here.
# pants-plugins/experimental/fooify/subsystem.py:
class Fooify(PythonToolBase):
options_scope = "fooify"
help = """The Fooify utility for adding .foo to a wheel."""
def rules():
return [*collect_rules()]
# # pants-plugins/experimental/fooify/register.py:
def rules():
return [*fooify_rules(), *subsystem.rules()]
For the sake of seeing what happens with the help, let’s comment out the target_types
array in register.py
and re-run help.
./pants help fooify
Unknown entity: fooify
pants.base.exceptions.BackendConfigurationError: Failed to load the experimental.fooify.register backend:
ImportError("cannot import name 'FooifyTarget' from 'experimental.fooify' (/pants-example-plugin/pants-plugins/experimental/fooify/**init**.py)")
Gotcha!
The fooify
keyword doesn’t work anymore (on 2.9.0 anyways). The reason is that Pants doesn’t register subsystems unless they’re used in rules. It's not a big deal, but can be hard to debug if you don't know what you're looking for. I created an issue about it, since I ran into this when running a code snippet in the documentation and burned a decent chunk of time trying to figure out what I did wrong.
If you did hook it up to a rule, you would see:
./pants help fooify
\`fooify\` subsystem options
--------------------------
The Fooify utility for adding .foo to a wheel.
Config section: [fooify]
None available.
Either way, this doesn’t actually affect us, so uncomment the target_types
in register.py
and let’s move on to the rules.
Rules
https://github.com/sureshjoshi/pants-example-plugin/tree/10-rules
Here we go, the meat and potatoes of your plugin - everything else is just window dressing (or, maybe salad dressing?).
Because this plugin is so trivial, there will only be one rule - but that’s not a limitation, just how things shook out.
First off, let’s add the subsystem to your rule - because you might want to extract some of your configurations or options out of it (not in this case - since our subsystem does nothing, but who knows). Oh, one thing I forgot to mention in the previous section - you can’t name the target and subsystem with the same external name, so I’ll be re-naming our target alias fooify_distribution
for clarity.
Next, let’s add your field_set
in, because we’re going to want the fooify_distribution
target dependency (i.e. a wheel).
# pants-plugins/experimental/fooify/rules.py:
@rule(level=LogLevel.DEBUG)
async def run_fooify(fooify: Fooify, field_set: FooifyFieldSet) -> BuiltPackage:
# Make a rule that can compile safely
empty_digest = await Get(Digest, CreateDigest())
return BuiltPackage(
digest=empty_digest,
artifacts=(),
)
If we were doing something more sophisticated, this is probably where we’d pip install our dependencies and run them on our source files or wheels or whatever. But, this is the Fooify plugin - it has one purpose only. To take in the wheel dependency and copy it with a .foo
appended...
# pants-plugins/experimental/fooify/rules.py:
@rule(level=LogLevel.DEBUG)
async def run_fooify(fooify: Fooify, field_set: FooifyFieldSet) -> BuiltPackage:
logger.info(f"Incoming field set: {field_set}")
# What is Get?
# https://www.pantsbuild.org/docs/rules-api-concepts#await-get---awaiting-results-in-a-rule-body
targets = await Get(Targets, DependenciesRequest(field_set.dependencies))
# NOTE: This is hardcoded for this example
wheel_target = targets[0]
# All of the following is to eventually unwrap to get the wheel file itself
packages = await Get(
FieldSetsPerTarget,
FieldSetsPerTargetRequest(PackageFieldSet, [wheel_target]),
)
wheel_field_set = packages.field_sets[0]
wheel_package = await Get(BuiltPackage, PackageFieldSet, wheel_field_set)
# What is a Digest?
# https://www.pantsbuild.org/docs/rules-api-file-system#core-abstractions-digest-and-snapshot
# Grab the Wheel file entry, and re-created it with the .foo extension
digest_entries = await Get(DigestEntries, Digest, wheel_package.digest)
wheel_file_entry = digest_entries[0]
assert isinstance(wheel_file_entry, FileEntry)
foo_file_entry = FileEntry(path=wheel_file_entry.path + ".foo", file_digest=wheel_file_entry.file_digest)
fooified = await Get(Digest, CreateDigest([foo_file_entry]))
# Ensure the new file is correctly logged during the build process
artifact = BuiltPackageArtifact(relpath=foo_file_entry.path)
# Ensure the fooified file is output to dist/ and
return BuiltPackage(
digest=fooified,
artifacts=(artifact,),
)
This all looks pretty complicated - but it’s mostly wrapping and unwrapping data representations. I won’t delve into it, as I’ve added links to the associated documentation which explains it all much, MUCH better than I ever could.
That’s It!
And there we go. We’ve created a trivial Pants plugin, ran into and resolved a few Gotchas along the way, and have a bit of a scaffold of what future plugins would look like.
In a subsequent post, I’ll tackle the challenge of testing plugins (which I personally find trickier than creating them).