Common Lisp in Practice

Introduction

One of the things which has kept Common Lisp out of my day-to-day toolbox is a lack of clear instructions how to get up and running with it — not in the REPL, but building small utility programs that can be called from the shell. I tend to reach for Python or Emacs Lisp, since I have a good grasp and they’re readily available, but I’ve always felt that Common Lisp could be a potent tool for these applications.

Building a project in Lisp can be confusing, because Lisp itself works so differently to other languages. While there are fundamental similarities to other AOT languages, the mechanisms are very different than most people are used to.

After reading my friend Steve’s Road to Common Lisp, I was inspired to figure this out. Many thanks to him for helping me with the proofreading and aiding my understanding where it lacked.

Goals

The overarching goal is to explain how to put together a Common Lisp Program, by walking through it step by step. The end result should be a functional, well-understood program.

It isn’t a tutorial on Lisp programming, because there are already great resources for that.

It doesn’t rathole on every possible approach you could possibly use, but treads and illuminates the beaten path. It explains the less obvious nuts-and-bolts of building Common Lisp programs by example: Where to put source code, how to make a binary, how to use libraries.

Running the Examples

If you’d like to run the example code, you’ll need to install Steel Bank Common Lisp (SBCL) (for all of them) and Quicklisp (for the last two). The Quicklisp installation is unusual if you’re coming from other languages, so it’s also fine to read along and see what things are like before investing in a Lisp environment.

This is written in a literate programming style using org-babel. All program output should be very, very close to what you’d see if you ran those programs. The original Org document and source code extracted from it are available in my GitHub repo.

Background

A typical compiler is a standalone executable which runs one OS process for each file, producing a binary artifact corresponding to its input; a linker executable combines those into the final binary. Some compilers output an intermediate representation (IR), such assembly language, then spawn a separate executable to assemble them.

Each executable represents a capability boundary. Their functionality is only usable in one way: by executing them with no context except what’s provided on their commandline.

For example, if you wanted the compiler to create the assembler listing in memory, then have the assembler work on that (skipping the whole compiler-output/assembler-input dance), you’re out of luck; it can’t be done. Or if you wanted to go one better and have the compiler create an in-memory AST the assembler can work on directly, this is also impossible.

Another way to think about this is that most languages have a close relationship between programs and binaries, and typically a single binary can run a single program. This wall precludes tight integration of multiple programs, because they can only communicate across the I/O constructs afforded to OS processes.

Common Lisp does not work this way. Much like interpreted languages, everything is loaded into one environment, and all functionality is accessible in the same way — by calling a function. However, Lisp goes much further than other languages.

For example, Python 3 ships with five executables:

dpkg -L python3-minimal | grep -c /bin/

This includes the interpreter, a byte-compiler, another program to remove the files the byte-compiler creates, etc.

OpenJDK has 8:

dpkg -L openjdk-11-jre-headless | grep -c /bin/

GCC has 16:

dpkg -L gcc | grep -c /bin/

And in order to actually use GCC, you also need binutils, which has many more:

dpkg -L binutils | grep -c /bin/

Can you guess how many Steel Bank Common Lisp (SBCL) has?

dpkg -L sbcl | grep -c /bin/

Just one, /usr/bin/sbcl. Everything that can be done with SBCL is done inside of the environment it provides.

The Environment; Images

When Common Lisp starts, it initializes a Lisp environment in the computer’s memory, then evaluates a toplevel function. The environment contains the Lisp language and tools; the standard toplevel is the REPL. When code is typed into the REPL, or loaded from a file, it’s added to the environment and can be used by other programs inside it.

The state of the environment can be saved to disk in a Lisp image¹, and restored by giving that image to sbcl(1), or executing the image directly from the shell. When the image is saved, a different toplevel function can be specified.

These are the building blocks for making executables. Code is loaded into the environment, then an image is created from that state, with the toplevel set to the desired entry point.

Version 1: Quick & Dirty

With all that out of the way, it’s time to make a traditional "Hello, World" program. This program will:

Run from a shell.
Use the first argument given to it as the name of the person or thing to greet.

Starting from the ground up, the function to create the greeting:

(defun greet (whom)
  "Create a greeting message for WHOM."
  (format nil "Hello, ~A." whom))

Trying this in the REPL shows that it works:

(greet "World")

The Toplevel Function

To satisfy the first requirement, a toplevel function is needed — this will be evaluated when the image is restored, handling the command-line arguments and printing the greeting.

I named the toplevel function MAIN, but it can be called anything. Any function which accepts zero arguments can be used as a toplevel.

(defun main ()
  "Greet someone, or something."
  (write-line (greet (first (uiop:command-line-arguments))))

  (uiop:quit))

There are two functions in here that may be new to you.

The command-line arguments given to an executable aren’t turned into arguments for the toplevel function, as with other languages; they’re returned from UIOP:COMMAND-LINE-ARGUMENTS.

As one might expect, UIOP:QUIT terminates the process.

Neither of these functions are part of the Common Lisp standard; both come from UIOP, which bridges some of the gaps in the spec, and between implementations.

Packages

The next thing to get a handle on is packages. This terminology is different than in other languages, which use it in the sense of "package manager," meaning a downloadable library and/or mechanism to install them.

In Common Lisp, a package is a namespace which contains symbols. The symbols can be defined in the package, like MAIN and GREET, or they can be symbols inherited from other packages, like DEFUN and FORMAT (which both come from the COMMON-LISP package²).

The Hello World example should define a package, called HELLO.

Packages must be explicitly defined before they can be used. Many languages treat a statement like:

package hello

As definition and use, i.e. everything declared in this file is implicitly put in the hello package.

The similar-looking Lisp analogue of this is IN-PACKAGE:

(in-package :hello)

While the code is similar, the semantics are different. This only sets the active package, it won’t create :hello³ if it doesn’t exist. Creating the package is an explicit step which must be done first.

The DEFPACKAGE macro creates a package. It takes a symbol naming it:

(defpackage :hello)

In Lisp, newly-created packages are completely empty, and don’t even include core language functionality like DEFUN. Those symbols can be used to by qualifying them with the package name, but in most cases, it’s desirable to use them directly. Adding a (:use …) form inside DEFPACKAGE will copy other packages’ exported symbols into the package being defined:

(defpackage :hello
  (:use :common-lisp))

If you hypothetically wanted to use to more packages, their symbols would need to be added after :common-lisp⁴. Note that this should be used with care, since updates to those packages could collide with the symbols in :hello.

In the same way that defining and using a package are separate, loading and using a package are also completely separate operations. While many languages have an import mechanism which both loads and uses, Lisp doesn’t work this way; :foo and :bar must have been loaded already.

Exports

The last package-related topic to cover is exported symbols. When a symbol is exported, it may be used by packages; the set of exported symbols comprises the public API of a package. Non-exported symbols should only be used within the same package.

Many languages specify visibility symbol-by-symbol, at the point of definition:

public int hashCode()

Lisp declares exported symbols when the package containing them is defined, using the (:export …) form:

(defpackage :hello
  (:use :common-lisp)
  (:export :greet :main))

Tying it All Together

The complete source for Hello World now looks like this:

(defpackage :hello
  (:use :common-lisp)
  (:export :greet :main))

(in-package :hello)

(defun greet (whom)
  "Create a greeting message for WHOM."
  (format nil "Hello, ~A." whom))

(defun main ()
  "Greet someone, or something."
  (write-line (greet (first (uiop:command-line-arguments))))

  (uiop:quit))

Building an Image

Because the Common Lisp toolchain exists inside the Lisp environment, build scripts for Common Lisp project are written in, you guessed it, Lisp.

The first thing the build script should do is load the source of the program, which I’ve placed in hello.lisp:

(load "hello.lisp")

Then, tell Lisp to dump the image into an executable, which will call MAIN when invoked:

(sb-ext:save-lisp-and-die "hello"
 :toplevel 'hello:main
 :executable t)

I’m using SBCL for these examples, and SB-EXT:SAVE-LISP-AND-DIE is the SBCL way of doing this. The precise incantation will vary based on Lisp implementation, because it’s not part of the Common Lisp standard.

The call to SAVE-LISP-AND-DIE could be put at the end of hello.lisp for this example, but it’s is a poor separation of concerns for anything more complex than one-off scripts. Its rightful place is build.lisp.

The complete build script goes into build.lisp and looks like:

(load "hello.lisp")

(sb-ext:save-lisp-and-die "hello"
 :toplevel 'hello:main
 :executable t)

Executing the build script with sbcl(1) will produce the binary:

sbcl --non-interactive --load build.lisp

Running it shows the message:

./hello World

Passing in the name of the current user also works:

./hello $(whoami)

Now that the program works, and you hopefully understand why and how, it’s time to tear it down and rebuild it a few times.

Version 2: Package Structure

Having all the code in one file is fine for a toy, but larger programs benefit from more organization. If the core functionality is split from the CLI, other projects (or other parts of the same project) can reuse the greeting function without the CLI code.

Also, Lisp packages can span files, so it’s not good practice to put the package definition in one of the N files that represent its contents.

What this should look like is:

build.lisp
packages.lisp
- src/
  - greet.lisp
  - main.lisp

The organization is different, but the contents of the files are almost exactly the same.

The package definition is identical to v1, but moved into packages.lisp:

(defpackage :hello
  (:use :common-lisp)
  (:export :greet :main))

The greeting code is moved to src/greet.lisp. It’s identical, except it has to declare the package it belongs to.

(in-package :hello)

;; Unchanged from v1
(defun greet (whom)
  "Create a greeting message for WHOM."
  (format nil "Hello, ~A." whom))

The CLI code moves to src/main.lisp, and also declares what package it’s in:

(in-package :hello)

;; Unchanged from v1
(defun main ()
  "Greet someone, or something."
  (write-line (greet (first (uiop:command-line-arguments))))

  (uiop:quit))

The build.lisp script needs to load the new pieces in the correct order. Since packages must be defined before they’re used, packages.lisp needs to be loaded before either of the files in src/; since MAIN calls GREET, the file containing GREET must be loaded before the one with MAIN:

(load "packages.lisp")                  ; Load package definition
(load "src/greet.lisp")                 ; Load the core
(load "src/main.lisp")                  ; Load the toplevel

;; Unchanged from v1
(sb-ext:save-lisp-and-die "hello"
 :toplevel 'hello:main
 :executable t)

Building and running works the same way:

sbcl --non-interactive --load build.lisp
./hello World

Version 3: Systems

The next yak in the recursive shave is systems. Packages are part of the Lisp language specification, but systems are not; they’re provided by a library. The dominant systems library at the time of writing is ASDF, which means "Another System Definition Facility." ASDF is a de facto standard, and comes bundled with both SBCL and Quicklisp.

Systems and packages are orthogonal, but since they both deal with some of the same parts of the project, and the names often overlap, it can get confusing.

A package is a way of organizing the symbols of a project inside the Lisp environment. Lisp doesn’t have a convention for determining what package things belong to based on the path or filename. One package can be split across multiple files, or one file can contain multiple packages.

A system is a description of how to load part of a project into the environment. A system can load multiple packages, or it can load a subset of one package. Systems encapsulate the list and order of files needed to produce a usable package.

Further complicating things, one project can have multiple systems. A system is a view into part of a project, and different code may need different pieces. For example, test code will need the test library loaded, or may need to set state before loading the code to be tested, or may need to change values inside the package containing it. Having a separate system for tests allows these different usecases to be supported gracefully.

Defining the System

Systems are defined in an .asd file, using the DEFSYSTEM form. To maintain good separation of concerns, the Hello World project needs two systems: one for the core, and one for the CLI. For these examples, I’ll be using the CLI system to demonstrate. If someone wanted to reuse the core GREET code in their own program⁵, they’d use that system.

(defsystem :hello)

There are multiple strategies for loading code, but the easiest is to load components in the order they appear in the system definition. This is indicated with :serial t:

(defsystem :hello
  :serial t)

Then, the components need to be specified. These are the files and directories the make up the system:

(defsystem :hello
  :components ((:file "packages")
	       (:module "src"
			:serial t
			:components ((:file "greet")))))

Then a secondary system for the binary. The only new thing is :depends-on, which indicates that this system relies on the earlier one.

(defsystem :hello/bin
  :depends-on (:hello)      ; This system needs the core HELLO system…
  :components ((:module :src
		:components ((:file "main"))))) ; …and includes one
						; additional file.

Putting the two declarations together into hello.asd results in:

(defsystem :hello
  :components ((:file "packages")
	       (:module "src"
			:serial t
			:components ((:file "greet")))))


(defsystem :hello/bin
  :depends-on (:hello)      ; This system needs the core HELLO system…
  :components ((:module :src
		:components ((:file "main"))))) ; …and includes one
						; additional file.

Since the system defines the files and load order, the build script doesn’t need to replicate that anymore; it can lean on Quicklisp and ASDF instead:

(ql:quickload :hello/bin)

(sb-ext:save-lisp-and-die "hello"
 :toplevel 'hello:main
 :executable t)

ASDF needs to be told where to find the system definition, and all others it should be able to load. This is a complex topic, but the simplest approach is:

Use Quicklisp.
Make a symlink from Quicklisp’s local-projects directory, named after the project, which points to the source tree.

This is easily the grossest thing about this entire setup.

rm ~/quicklisp/local-projects/{hello,system-index.txt}
ln -sf $PWD/v3 ~/quicklisp/local-projects/hello

The rest of the source is unchanged from v2.

Running works the same way:

sbcl --non-interactive --load build.lisp
./hello World

Version 4: Using Libraries

The final step is to replace UIOP’s basic program arguments with a more full-featured library, unix-opts.

Common Lisp libraries are installed via Quicklisp, and loaded with ASDF. As with other Common Lisp tasks, actually installing the library is done from the REPL.

Quicklisp

Quicklisp is not a package manager like other languages have. There’s no project-specific setup, like with virtualenv or rbenv. There’s no node_modules.

Quicklisp is more of a caching mechanism than a package manager. Similar to Maven’s ~/.m2, a single copy of the code is stored in ~/quicklisp/dist/quicklisp/installed. ASDF looks there when asked to load systems into a Lisp environment.

As with other tooling, the primary interface for Quicklisp is the Lisp environment.

Installing unix-opts

The Quicklisp documentation discusses this, but I’m going to cover the essentials.

Quicklisp has QL:SYSTEM-APROPOS, which searches available libraries:

(ql:system-apropos "unix")

Installing is done with QL:QUICKLOAD. This downloads the library (if necessary) and loads its system:

(ql:quickload "unix-opts")

Defining the Systems

The only change to the system definitions is adding :unix-opts to the :depends-on form. Note that this refers to the system, not the package. Systems provide packages, and depend on other systems. Because build.lisp uses QL:QUICKLOAD to load the system, it’ll notice if unix-opts (or any other system in :depends-on) hasn’t been installed, and do that automatically.

The New `MAIN`

With the :unix-opts system loaded, the :unix-opts package is available for MAIN to use:

(in-package :hello)

(unix-opts:define-opts
  (:name :help
   :description "Print this help text"
   :short #\h
   :long "help"))

(defun main ()
  "Greet someone, or something."
  (multiple-value-bind (options free-args)
      (unix-opts:get-opts)
    (if (or (getf options :help) (/= (length free-args) 1))
	(unix-opts:describe
	 :prefix "A Hello World program."
	 :args "WHOM")
	(write-line (greet (first free-args)))))

  (uiop:quit))

Nothing needs to change in any of the other source files.

Building

For this to work, the Quicklisp local-projects symlink created in v3 needs to be updated to point here instead:

rm ~/quicklisp/local-projects/{hello,system-index.txt}
ln -sf $PWD/v4 ~/quicklisp/local-projects/hello

…and the ASDF registry cleared:

(asdf:clear-source-registry)

After building (using the same commands as previously), the new options parser is working:

sbcl --non-interactive --load build.lisp

./hello

./hello $(whoami)

Conclusion

At over four thousand words, this piece has been a lot more than I set out to write. The process of learning, organizing, and refining my own understanding has been wonderful. I hope you’ve been able to take away some of that, and will go forth with useful new tools.

Footnotes:

SBCL refers to them as "cores."

:CL is an alias for the :COMMON-LISP package, so (:use :cl) is a common equivalent.

You may note that I’ve written the name of the package as HELLO (which it is), but it’s in the code as :hello. For a deeper explanation on why this is the case, I recommend the chapter on Packages and Symbols from Programming in the Large. In the mean time, you’ll just have to trust that it’s right and I know what I’m doing.

⁴

:cl is an alias for :common-lisp, so it’s common to see that in code in the wild.

⁵

For example, to have their program greet the user when it starts, or to create a massively multiplayer online Hello World (MMOHW).

Atomized

Common Lisp in Practice

Introduction

Goals

Running the Examples

Background

The Environment; Images

Version 1: Quick & Dirty

The Toplevel Function

Packages

Exports

Tying it All Together

Building an Image

Version 2: Package Structure

Version 3: Systems

Defining the System

Version 4: Using Libraries

Quicklisp

Installing unix-opts

Defining the Systems

The New `MAIN`

Building

Conclusion

Further reading

Footnotes:

Common Lisp in Practice

Introduction

Goals

Running the Examples

Background

The Environment; Images

Version 1: Quick & Dirty

The Toplevel Function

Packages

Exports

Tying it All Together

Building an Image

Version 2: Package Structure

Version 3: Systems

Defining the System

Version 4: Using Libraries

Quicklisp

Installing unix-opts

Defining the Systems

The New MAIN

Building

Conclusion

Further reading

Footnotes:

The New `MAIN`