Thoughts on Paths

or what you need to know about using paths in a compiler

Where It All Begins

What does the compiler know about its environment when it starts working?

Of what it guaranteed knows, one can only name the path to the binary file that served as the «launcher». How do we know this?

We know this from a convention that all operating systems I am aware of try to adhere to — at application startup, the path to the executable is passed as an argv parameter:

сКопировать

int main(int argc, char** argv) {
   ...
}

By convention, the zeroth element of the array passes a pointer to a string containing the path by which the program’s executable was launched — in our case, the compiler. Situations where this is not the case are possible, but have not been encountered in practice. Therefore, we proceed from the assumption that the compiler knows this.

Furthermore, other elements of this array may pass command-line parameters into the compiler. This information is also guaranteed to be available to the compiler.

All other information the compiler has to obtain itself.

Environment Information

What the compiler learns next, it finds out by querying the operating system.

The information to be discovered can be slightly grouped.

Here and further below, we speak of the compiler as a program; the issue of code generation and target platforms deserves separate consideration.

1. Operating System
   For the compiler as a tool, knowledge of which OS we started on is not very interesting per se. It needs to be able to work with files and texts, and this, in turn, may depend on the operating system. Obvious among these dependencies is the structure of paths. In Windows, a path begins with a drive letter and a colon, like «C:». Different OSes use different path separators, «\» or «/», and in quite exotic cases, other types of separators might be used, and the way a filename is constructed differs significantly — for instance, the version might be part of the «filename.name; version».
2. Current Directory
   It is not always needed, but we can only find out by asking the operating system. This information is already strictly platform-dependent; different platforms use different methods (different functions) to get the current directory.
3. Environment Variables
   A lot of information in an OS is passed via a list of environment variables. This is a set of «name=value» pairs. Retrieving this list (or its individual elements) is also platform-dependent.

What the Compiler Needs to Work

Let us now enumerate what information the compiler needs to work. I will try to list as much as might be required; depending on the language and runtime properties, this list may change.

Moreover, I will mention degenerate cases I have encountered, as they provide a complete picture.

1. Location of the File Being Compiled
   The file we intend to process is usually specified explicitly. That is, the path to this file is typically passed as a command-line parameter, but this is not necessarily so. For example, a compiler might be designed such that search paths matter to it, and only the filename is fed as input, while its location is determined by some predefined rules. Or, alternatively, the compiler might be configured to process a file whose location is hard-coded. In any case, the file (or another entity) with which the compiler’s work begins is the first and necessary piece of information for operation.
2. Location of Library Code
   The compiler may require a set of entities used for handling the semantics of the language. This is «library» code and cannot be circumvented. It can be either part of the compiler or located externally.For example, Delphi has a module System.pas, which, if I remember correctly, wasn’t compiled but rather supplied so that «pseudo-definitions» would be available — i.e., entities (e.g., types) were implemented inside the compiler, yet they looked as if declared in a file.Haskell has a module Prelude, used during the compilation of any module.And so on; depending on the language, the workflow scheme may differ, but it is important to understand that we are talking about mandatory elements of language semantics. The presence of such modules is obligatory, and the compiler must be aware of them.
3. Imported Entities
   Programs are rarely written without using external modules. Again, this is implemented differently across languages, but the general scheme is roughly the same. When an import or use operation appears in the source code (the specific term is not crucial right now), the compiler must go to some code repository and retrieve information about the imported module there.
4. Cached Imported Entities
   Unlike the previous point, here we refer to entities that might have already been compiled and transformed into a form prepared for the final stages of compilation. We are talking about pre-compiled modules. Such entities can take up quite a bit of space and, in some cases, might even be located on a network and downloaded on demand. I did projects like this twice, and both times had to abandon them due to synchronization difficulties, timestamp management, and other technological complexities. However, the refusal was compensated by a general acceleration of compilation. Nevertheless, this group of artifacts must have its own access mechanism (and/or search).
5. Tooling
   During compilation, not only the compiler’s own tools but also external auxiliary instruments may be used. Many compilers make calls to C/C++ compilers or their components. Calling assemblers, linters, checkers, and other tools is also possible here. And although in interactive mode it is almost always sufficient that these tools can be found via the system PATH, within CI/CD server builds this becomes highly non-obvious and often requires additional configuration or direct path specification.
6. Annotation Processors
   Java and Lisp have mechanisms allowing intervention in the compilation process using code written in those languages (other languages feature this too, but less explicitly). And although formally one could say this falls under one of the previous definitions, such a tool might have its own setup and path-finding mechanism.
7. Target Storage
   When compilation ends, the created artifact must be saved somewhere. This place where the file or files are saved can also be configured. For some compilers, saving happens next to the source text; for others, it can be set explicitly, and for still others, the storage of the compiled result is determined by calling a separate tool (jar in the same Java).
8. Secondary Artifacts
   Compilation proper usually ends with creating an executable file. But a product rarely consists of just one file. There might also be media files, documentation, data files, UI files, etc. Furthermore, the compiler might include build management mechanisms and be capable of creating what are called «distributions». This, again, can be considered a tool invocation falling under item 5, or it can be part of the compiler, and where the distribution will be saved can and should also be configurable.

Where To Look?

Now let us talk about what the compiler can rely on while doing its job.

I can distinguish three mechanics that a compiler can use for its work: «portable build», «hard path», and «combined path».

Hard Path

Let’s start with the simplest, albeit very limited option — the hard path.

Such a compiler workflow scheme assumes that the entire compiler and all files for operation are located at a known, predefined path.

This scheme is good because all instrumentation can be reduced to a certain set of constants used everywhere. It is very convenient, allows one not to think about anything, and not to worry about placement.

Everything works fine until the question of cross-platform development or a significant change in environment arises.

Thus, back in the day, Windows systems noticeably complicated writing to disk root folders, which led to breakages in a large number of systems written based on the assumption that they would run from «C:\my_system\».

Moving to %USERPROFILE% didn’t change the situation much, as issues with long names and names with spaces began.

Furthermore, moving such a program to Linux or FreeBSD required non-trivial effort. Moreover, considering the differences between Unix-family operating systems, it often happens that a path exists and is accessible for execution on one OS, while on another it might be absent or present other difficulties, for example, requiring administrative access for normal operation.

Such a compiler workflow scheme generally has a right to exist, but requires a very clear understanding of the necessity to unify and control the environment in which this compiler operates.

Portable Build

This term emerged with the advent of USB drives, where the entire program could fit on that drive or in a separate folder on a hard disk. Such a program requires nothing to work; everything is located in one place.

With compilers, this scheme basically works well, with one small «but». During operation, the program might require a large amount of information, loadable modules, and other active development attributes, which will lead to a significant growth in the assembly contents, making it technologically very inconvenient to maintain.

Here perhaps a joke about the depth of a cave full of npm modules is appropriate, but it’s not really funny at all.

Therefore, although the scheme is generally viable, once the assembly size exceeds a certain threshold, it becomes burdensome.

Combined Path

The name of this mechanic is quite conditional.

One can somehow combine the previous two approaches — place part of the compiler, including possibly some library functions and other artifacts, so that the compiler gains access to them without ever referring to the OS, and place another part so that their usage, on one hand, does not depend on the compiler, and on the other hand, can be easily changed while maintaining operability.

For example, in the flow9 language, an interesting mechanic was applied.

The compiler serves as the foundation for product development, and all products are built around it. So, on the disk in the working folder, we see something like:

- flow
- product1
- product2 
...
- productN

and from any product folder, one can always call ../flow/bin/flowc main.flow, which will lead to product compilation and obtaining a distribution (I omitted a few details, but that’s essentially how it is).

Furthermore, if the code says import ds/list;, the compiler knows it needs to go to the ../lib/ds folder relative to its executable and grab the list.flow file from there.

Or, for example, in Go, there are environment variables GOROOT and GOPATH, defining, respectively, where the compiler’s binary files are located and where to look for files for compilation.

A similar function is performed by the combination of JAVA_HOME/CLASSPATH in Java. Although these are not the only variables that can be configured, these two substantially affect the compiler’s functionality.

At the same time, some files might be located in places with hard-coded paths. Thus, cached imported entities for ECL are stored in /home/user/.cache/common-lisp/ecl-23.9.9-9f27c69d-linux-x64/ and can be deleted at any time.

Rust, for example, stores imported modules in the ~/.cargo folder.

And so on.

How to Use This

What can the compiler use to gain access to the required entities?

• path to its own executable file
• contents of environment variables, GOPATH or CLASSPATH
• a config file located at a known address, e.g. ~/.sbclrc or %USERPROFILE%/my.cnf
• a config file defined within the compiler
• a config file defined in the project
• parameters passed via the command line
• parameters embedded in files during installation (Go does this, for example)
• dynamically changing parameters, if the system allows it

If several of the listed configuration methods are used, a priority system must definitely be established for them, meaning some should have higher «importance» than others, which will allow resolving conflicts if multiple ways of specifying settings are indicated simultaneously.

Conclusion

All of the above is intended so that when working on your next compiler, you have arguments for choosing the layout scheme for parts and libraries.

The list of used entities and configuration methods depends significantly on incoming requirements, and maximum flexibility is not always required, but limiting oneself to just one thing is usually not worth it either.