Project Builds

Background

A somewhat universal need when working with computer software is to be able to in some way “build” your project. For the case of distinction this can be reduced to supporting assorted verification and generation of artifacts which enable the project to be used. This is a very obvious step when the source needs to be compiled as there is a clear building division between source and the target, but for many projects the source itself is what ends up being released. Even in such projects there is likely to be significant benefit in preserving a line between the assorted machinery available to suport development and the hopefully more focused runtime packages.

A good way to frame what belongs to the build tooling is what a development flow looks like. Given the source code is obtained, what is in place to facilitate modifying the project safely and conveniently. It may be worth pointing out that some functions are outside of this process and therefore may be considered outside of what is considered part of the build. Version control and concerns such as releasing and publishing enable moving of source code and its products around rather than being primarily concerned with modification or generation. However these tasks may be performed by the same tools and may be highly interconnected within workflows or pipelines, so for simplicity they can be considered part of the build process within this context and later differentiated if there’s perceived value in doing so.

Perspective and Goals

Almost decent software project is likely to have the necessary support for tasks such as building any artifacts, performing assorted tests, and enabling local execution. Often however only the minimally requisite functionality is automated and concerns such as configurng the build environment, knowing how to invoke the relevant tasks, and managing how different tasks and anything external to the build need to fit together are left to the developer. Hopefully a fair amount of this information is documented, but even in those cases that documentation is likely to amount to sequences of steps which amount to providing a program to the reader which could presumably instead be sent to the computer. More importantly such information is likely to be some combination of inaccurate and incomplete as the assorted pieces shift oer time. If the documentation is up-to-date it as at best onerous but most likely is not going to provide targeted information for each developer’s system and the assorted states that it may currently be in: most likely reflecting a narrow pristine state that is an egg that is cracked as soon as work is started on the project. This is an open invitation for churn.

If instead all of these aspects of project builds are provided by software tools then the overhead of working on the project can be reduced and refined. By codifying all expectations within the build tooling the project itself can deliver the appropriate feedback or remediation for any given scenario. This provides an organized framework to collect knowledge and improve processes: where the myriad concerns and permutations that may arise over the course of working with a software project can be addressed as needed at the source truth. Delivering such information through other channels is likely to produce a clunky and overwhelming haystack of information housing the needles of immediate answers.

My goal when working with any project is therefore that when working with source code the build tool provides all of the knowledge that is needed to work with that project and that any desired operation is either successful or provides specific information as to what should be done to enable success.

Additional Considerations

Often builds also suffer from some level of fragmentation caused by supporting tools. For local development this often takes the form of things like IDE support. IDE configurations can be beneficial if they are coordinated across teams or organizations but they are often not, and the prospect raises additional concerns around one of seemingly dictating which tools should be used, only supporting a selection of developers, or expending effort to support a range of tools.

Another cause of fragmentation is CI tools. Very often the workflow for such tools is captured in a designated CI configuration. This almost certainly incompatible with the above notion of shared IDE configuration, and regardless of whether that approach is used it is likely to represent duplicated and likely divergent configuration from what can be readily executed which means that any behavior in CI may not be cumbersome to attempt to reproduce locally. This ties into a thread I plan on covering at some point around the notion that one of the proposed values of CI is to prevent “golden boxes” but too often the CI environment just becomes the new golden box and the potential to use CI as a contributor to build reproducibility is squandered.

While neither of these issues preclude the goals outlined for a build system they pose additional considerations without promising solutions. Starting from an independent maintainable system with desired behavior and then integrating as needed additional users of that system seems more tractable than reconciling and attempting to converge different specialized uses.

Approach and Maturity Levels

For personal projects I typically start off configuring the project build to provide all of the behavior that I want, but when joining new projects there may be a sigificant amount of issues that need attention, and on larger projects it may not be reasonable to address them all at once. After working through this process several times it seems valuable to define a more tactical path involving tiers of maturity for project build. Those levels will be defined here.

Help

One of the most basic benefits a build system can provide is conveying how to work with the project. This should include information such as how to invoke assorted tasks the build tool provides as part of development. This is a distinct role within all of the documentation for a project as it reflects information which supports the build concerns of the project itself. Other documentation should provide information about the project itself and answer questions such as the whats and the whys of the work being done, but when when viewing the project and wanting to know how to apply some function to your source code the build tool is a perfect location. This can include basic and more targeted information. Some examples could include:

The build system is a natural fit because it would already be in use and can therefore provide a uniform means to interact with the project. Additionally the build tool can enable navigation of possibly specialized knowledge and can integrate some information to be displayed when it seems to be relevant.

For example in addition to specific help there could exist subcommand such as help-foo-service which provides information about interacting with the foo service which may not typically be necessary so it can help organize the information. Then if a task is executed which interacts with foo service but fails the build system can point the developer to that subcommand or deliver the additional information directly. This coalesces into a unified pursuit of providing automation and complementary information.

Predictable Success or Failure

A major stumbling block with many build configurations is that they make assumptions about the state in which tasks will be executed. If the state is not as expected the invocation may or may not succeed with potentially confusing output. Some typical examples include preconditions which are documented but not validated (such as certain things being installed or running (or not running)) or an implicit ordering of expected steps (i.e. run setup and then run do-it). If the enclosing state is not either isolated or validated then any resulting feedback is likely to be mysterious which leads to time wasted chasing down issues resulting from something or other drifting and needing to be knocked back into place. I’d imagine pretty much every developer can relate to the experience of being blocked from being productive by having to spend time poking around local environmental issues or burning cycles because something that wasn’t cleaned up or started up properly leads to a wild goose chase.

These issues can be largely avoided (or at least reduced) but not making assumptions about the execution environment. Isolation using technologies such as containerization offers a particularly appealing solution to this, and in cases where the environment cannot be controlled it should be validated and specific feedback provided. Any relationships within the project should be explicitly coordinated. This is certainly not a novel idea as most decent build tools enable modeling of the dependencies between tasks such that a given invocation would lead to the evaluation of an appropriate directed acyclic graph, but this functionality is too often underutilized.

Taken all together the goal at this stage is that any given build task should reliably succeed, or if that is not possible should fail with useful feedback. Such behavior should not be dependent on other commands having been executed, the end result should be that if a developer wants to do something they should be able to run the equivalent of do-it and the build system takes care of everything necessary to enable do-it to run and any potential cleanup…or if there’s something it can’t do for any reason it provides clear feedback and ideally suggestions. Beyond things the system can’t do it can also provide feedback for things that is just doesn’t do yet, for example given the dependency described above the do-it task could communicate that the system has not yet been setup.

There are of course limitations to this. If the build environment itself has been corrupted in some way then it may need to be corrected and made more robust. Any process is also subject to external factors so something getting killed after it has been started or validated but prior to it being needed is unlikely to be worthwhile addressing outside of runtime.

Delivering this is almost certainly going to be an ongoing effort. New problematic scenarios within the project being built will arise both from the project growing and from the discovery of new and interesting manifestations of dirty state. New problems are also likely to arise within the build system itself as the mechanisms that it uses to discern and reliably perform needed work are refined. Both of these angles should still only require occasional bite-sized adjustments as issues are encountered, and as the catalog of practices grows the costs should decrease as the benefits increase.

Only What is Needed

The outcome of the above sounds fairly ideal but it omits a further aspect of optimization that is likely to be very valuable in practice. Basic pursuit of independently invocable tasks may lead to a significant amount of duplication where the lack of making assumptions about state manifests as needlessly modifying such state. This can lead to significant time wasted from redundant work being done which in scenarios such as repeatedly running tests can be disruptive.

Similar to dependency modeling this is a solved problem in most decent build tools. A given task can be configured to determine whether it needs to be run and when combined with the task graph this can enable just the right amount of work to be done to consistently produce the desired end state.

While this may provide immediately beneficial it is split out for multiple reasons. The most significant challenge is that it can complicate the goal of predictable success. If this is not configured completely then it can lead to tasks not being run that should have which is another form of assumed state that can lead to unhelpful feedback. Defining and enforcing everything a task requires is likely to involve a fair amount of work, but the additional undertaking of detecting whether to trust that such dependencies have been resolved introduces additional overhead such as further knowledge of the tasks themselves (when exactly does something need to be done?) and further coordination of how that information is tracked (how is it communicated that a task is unneeded and how trustworthy is that signal in the face of faults).

Another lurking concern in this section is that many build tools support this type of behavior but some do not. If the build provides the other desired characteristics then this may represent a significant undertaking with the tool on-hand and switching tools may not be warranted. In such cases it may be worth providing such optimizations to specific tasks but it is unlikely to be worth implementing project-wide.

Often the commands that are invoked by the tasks may be able to intelligently avoid doing unnecessary work. This can be very helpful and may prove enough to satisfy needs, but a risk is that this may be more fragile than realized as often whether work is needed is determined by changes along a path in the graph of tasks rather than something that is immediately visible to the executed command. In such cases relying on a command which lies at a particular vertex in that graph is unlikely to detect the changes which can lead to work not being done that should have been and very typically manifests as changes that were made not being included in the more recent build. This is a general concern in that countermeasures should be taken to make sure that any such tasks receive whatever stimulus or configuration is necessary to reliably perform work when it is neeeded.

For all of these reasons the goal of providing this optimiziation is split out such that it is treated as an enhancement to an already well-behaved build which should not compromise the behavior of that build. This introduces the likelihood of additional ongoing needed work. This can be done systematically through timing how long particular tasks take and identifying those that seem to be wasting time and implementing this as a means to reclaim time where there is clear value. Like similar types of optimization this should amount to a removable layer on top of the slower but more potentially more reliable behavior.