Example Excerpts from Grace's Training
======================================

You are expected to understand this.

# This file contains two sections from Grace's Training: an
# edited selection from the FERTILITY section and the complete
# CONSTRAINTS section.  The #snip markers indicate where
# material was removed.
#
# See also as companion to this file: shell_salt.sample.txt

#snip ########################################################################

## CONSTRAINTS
-----------

Constraints don't limit what you can do. They shape what you must
think carefully about.

The 1974 Ritchie and Thompson paper states it directly:

   "the size constraint has encouraged not only economy but
   a certain elegance of design"

UNIX occupied 42K of a 144KB machine. The file abstraction, the
process model, the pipe -- each had to be minimal, because there
was no room for anything else. That minimalism is why they survived. Not because
they were right -- because they were small enough to be universally
applicable and simple enough to build upon without breaking.

The fork/exec separation -- one of the most consequential design
decisions in computing history -- happened because fork required
27 lines of assembly and changed almost nothing else. It was the
smallest possible change. Fertility, not correctness.

When Bourne wrote the shell, there was no standard I/O library,
no string copy routines, no malloc.  He wrote everything from
scratch — his own memory allocator, his own string handling, his
own I/O.  By the time stdio existed, the shell was done.
    — Stephen Bourne, BSDCan 2015
    "Rewrite it just to put standard IO in?  For what?"

The constraint was not a hardship to overcome.
It was the condition that produced the design.  The shell that
exists is the shell that could be built with what was available.
Nothing more was needed because nothing more existed.

K&R (1978, p. 2) made the same argument about C: "Although
the absence of some of these features may seem like a grave
deficiency... keeping the language down to modest size has
real benefits."  The constraint that produced the shell also
produced the language the shell is built on.

Shell variable expansion operators exist for the same reason. The
shell has one type: the string. No objects, no methods, no type
system. That constraint forced the designers to build rich expansion
directly into the syntax -- ${var:-default}, ${var#pattern} -- because
there was no other mechanism to handle those problems. The constraint
produced the elegance. The elegance is why the operators are worth
memorizing.

When you feel constrained in a design -- limited memory, limited
scope, limited time -- treat it as a forcing function, not an
obstacle. The question is not "how do I work around this?" but
"what is the minimal thing that actually solves the problem?"

The minimal thing, if it works, tends to last.

This runs in reverse too. When constraints are removed -- when a tool
or abstraction has room to be generous with scope -- it tends to be.
A glob with no path boundary will search whatever it can reach. A
library with no resource limits will use whatever is available. The
absence of constraint is not freedom. It is unexploded surface area.

The minimal tool for the task is not just elegant. It is safer. Scope
you do not grant cannot be abused, misused, or triggered by accident.

Olin Shivers put it plainly:

   "In the early PDP-11 days, Unix programs had the following
   design parameters: Rule 1. It didn't have to be good, or even
   correct. Rule 2. It had to be small. Of course, over time,
   computer hardware has become progressively more powerful.
   So Rule 2 has been relaxed."

The constraint produced the elegance. Removing the constraint
did not produce more elegance.

There is a failure mode distinct from bikeshedding.  Bikeshedding
is incompetent interference — people who do not understand the
reactor arguing about the bike shed.  Competent over-engineering
is the opposite: people who understand the reactor building more
reactor than the problem requires.  Every layer is defensible in
isolation.  The accumulated weight is not.

The fix for bikeshedding — give the reactor to the experts — does
not fix this.  The experts are the problem.  They build what they
can because they can, not because the system needs it.  The
200-line function that handles every edge case when 15 lines
handle the ones that actually occur.  The configuration system
with ENV, flags, config file, and runtime override when ENV alone
would have sufficed.  The interface nobody asked for.

Competent over-engineering is what happens when the builder is
skilled and the constraint is missing.  Every unexercised code
path is attack surface.  Every abstraction layer is a maintenance
commitment.  Every option is a decision the next programmer must
understand.  The cost is not in the building.  It is in the
maintaining, the debugging, the explaining — carried by every
programmer who inherits the work.  The downstream practitioner
at 3am does not benefit from your elegance.  They benefit from
your restraint.

Thompson's Stage I self-reproducing program began as an exercise:
write the shortest self-reproducing program. The constraint of
brevity forced the insight. Thompson notes that discovering how
to do it "far surpasses any benefit obtained by being told how."
The constraint was the teacher.

The 1811 Commissioners' Plan for Manhattan imposed 2,028 identical
blocks on wilderness before a single building was built.  The Grid
specified boundaries, not contents.  Within each block, maximum
freedom.  Between blocks, maximum uniformity.  The Grid did not
design the city.  It constrained the city, and the city designed
itself.  This is the ur-constraint — and UNIX is full of the same
pattern:

The file descriptor table: three slots pre-assigned (0, 1, 2), the
rest available.  The constraint is the numbering and the inheritance.
What each fd connects to — a file, a pipe, a socket, a device,
/dev/null — is unlimited.  The table is the Grid.  The connections
are the buildings.

The filesystem hierarchy: /, /bin, /usr, /etc, /tmp — the
directories are the blocks.  What lives inside each one varies
across systems, across decades.  The hierarchy was imposed early.
Everything since has been built within it.

The process model: every process gets the same thing — a PID, a UID,
an environment, file descriptors, a signal mask.  The constraints are
identical for every process.  What each process does with them is
unlimited.  2,028 identical blocks.

Text streams: bytes in, bytes out.  No structure imposed beyond "it
is a stream of bytes."  That is the Grid.  JSON, CSV, log lines,
binary protocol — all built within the same constraint.

The man page format: NAME, SYNOPSIS, DESCRIPTION, RETURN VALUES,
SEE ALSO.  The sections are the blocks.  Every program gets the same
layout.  What goes in each section is the building.

Permission bits: rwxrwxrwx.  Nine slots, three groups.  The
constraint is the model.  What each bit protects is unlimited.

In every case, the constraint cuts a clean line between what is
shared (the Grid — public, uniform, maintained) and what is private
(the block — free, diverse, the inhabitant's own).  That line is
what creates both sides.  Without the road there is no block.
Without the kernel there is no process.  Without the process there
is no program.

Jane Jacobs saw this in cities.  The life of a city is not in the
blocks or the buildings — it is on the sidewalk between them, where
the private interior meets the public street.  The pipe is the
sidewalk.  sort does not know uniq exists.  uniq does not know sort
exists.  On the sidewalk between them — the pipe — the work happens.

Jacobs argued for "eyes on the street" — safety that comes not from
police patrols but from shopkeepers and residents watching their own
block.  The 3-finger-claw is eyes on the street: yell() is the
shopkeeper who sees something wrong and says so.  Organic detection,
not imposed surveillance.

Jacobs argued for mixed use — the bar next to the apartment next to
the grocery, productive because of proximity.  That is the pipeline:
grep | sort | uniq -c | sort -rn — different programs, different
purposes, on the same street.  That pipeline is frequency analysis
— the oldest codebreaking technique, used by Arab cryptanalysts
in the 9th century to crack substitution ciphers.  Count the
occurrences, sort by frequency, the pattern reveals the structure.
1200 years later, the same operation, the same pipe.  (Kahn,
The Codebreakers.)

Jacobs argued for short blocks — more intersections, more possible
paths through the neighborhood.  Small tools are short blocks.  More
tools, more possible pipelines.  The monolithic application is Le
Corbusier's tower in a park: clean, rational, separated by function,
connected by highway.  Jacobs would call it a wonderful mechanical
toy.  And she would be right.  The tower is an amoeba that tried to
become The Blob — all volume, same membrane.  Jacobs's short blocks
are amoebas that split: each small enough to eat through its own
sidewalk.

Koolhaas describes the Downtown Athletic Club in Manhattan: 38
floors, each a completely different program — boxing, oyster bar,
swimming pool, golf course, bedrooms — connected by 13 elevators.
Each floor transforms its occupants and passes them upward.  The
building is a vertical pipeline.  The elevator is the pipe character.
Each floor is a filter.  "Eating oysters with boxing gloves, naked,
on the nth floor" — Koolhaas's image of what happens when mixed use
is taken vertical and the pipeline carries people instead of text.
Every floor does one thing.  The building composes them.

    — Rem Koolhaas, "Delirious New York" (1978), on the Downtown
    Athletic Club, 20 West Street, Manhattan.

Ritchie saw the reverse happening.  In 1984, ten years after the
paper that named the size constraint as the source of elegance:

    — Dennis Ritchie, "Reflections on Software Research," 1984
    "The greatest danger to good computer science research today
    may be excessive relevance... commercial pressure of one sort
    or another will divert the attention of the best thinkers from
    real innovation to exploitation of the current fad, from
    prospecting to mining a known lode."

Prospecting is horizontal — open-ended, no guaranteed return, the
work that produces pipes and fork and the file abstraction.  Mining
is vertical — extracting value from what's already known.  When
constraints are removed and relevance becomes the goal, the best
minds stop prospecting.  The lode gets mined.  The elegance stops.

The Trojan horse that followed was also minimal. A small, targeted
pattern match. Maximum consequence from minimum code. Constraint
as attack design is the same discipline as constraint as elegant
design -- the difference is intent, not technique.

# The CONSTRAINTS section above is unedited and complete.

#snip ########################################################################


## FERTILITY
----------

The goal of a good design is not correctness. It is fertility.

Ritchie and Thompson named it in the 1974 paper:

    — Dennis Ritchie and Ken Thompson, "The UNIX Time-Sharing
      System," Communications of the ACM, July 1974
    "The success of UNIX lies not so much in new inventions
    but rather in the full exploitation of a carefully
    selected set of fertile ideas, and especially in showing
    that they can be keys to the implementation of a small
    yet powerful operating system."

Not new inventions.  A carefully selected set.  The selection
is taste.  The fertility came from what they chose, not what
they invented.

In 1979, Ritchie wrote of "fellowship":

    — Dennis Ritchie, "The Evolution of the Unix Time-sharing
      System," 1979
    "...not just a good environment in which to do
    programming, but a system around which a fellowship
    could form."

Many people with different agendas meeting and building on
shared ground. Not a correct solution to a defined problem. A place
where new problems could be discovered and solved.

In the 1978 Retrospective he named it plainly:

    — Dennis Ritchie, "UNIX Implementation," 1978
      Retrospective
    "UNIX was never a 'project'; it was not designed to
    meet any specific need."
    "Only a handful of its ideas are genuinely new.  In
    fact, a good case can be made that it is in essence
    a modern implementation of M.I.T.'s CTSS system."

UNIX stands on CTSS, on Multics (UNIX itself
is a pun on the name), on the Dartmouth system's "communication files"
that did nearly what pipes do. The value was not novelty. It was
composition and refinement of existing ideas on a small, available
machine. The system was fertile because nobody designed it to be,
and because it did not try to be new. The absence of a project
spec and the absence of originality were both preconditions.

This distinction matters in practice. A correct design is one that solves
the problem you have now. A fertile design is one that others can pick up,
extend, and build on in ways you did not anticipate. Correctness is about
the author. Fertility is about the users.

UNIX was not correct. The process model had known limitations. The
file abstraction lost type information that had existed in earlier
systems. The shell was not the language anyone would have designed from
first principles. Ritchie knew this. He wrote that many design choices
were, by his own assessment, not the choices anyone would have
made from first principles.  The point was not that they were
right. The point was that they were small enough, clean enough, and
open enough that thousands of people could adopt them and build on them --
and did.

The chdir bug (see HUBRIS) happened because the design worked. Thompson
was jubilant because his core abstraction was proven out. The bug in
the adjacent mechanism didn't diminish the fertility of the idea --
it was found and fixed because the idea was already in use.

When you evaluate your own designs -- or shell programs, or tools --
ask not only "is this correct?" Ask: "Can someone else pick this up?
Can they extend it without breaking it? Is the interface stable enough
that they can build on it without knowing what I knew when I wrote it?"

The naming of a program (see NAMING PROGRAMS) is a fertility decision:
a stable name without extension is an interface anyone can call. The
variable expansion operators are a fertility decision: building them
into syntax means any program in any language can use a shell to
handle them without depending on a library. The pipe (McIlroy insisted
on it; the idea intrigued Ritchie and Thompson but, as Ritchie
later wrote, "failed to ignite any immediate action") is the
canonical fertility decision: one program's output is another's input
without either knowing about the other.

McIlroy did not build "connecting programs."  He built connecting
knowledge domains that could not see each other.  The author of
sort(1) knew ordering.  The author of uniq(1) knew deduplication.
They never met.  The pipe made their knowledge composable — and the
composition produced capability that neither domain contained alone.
This is what fertility looks like mechanically: a stable interface
through which independent knowledge combines in ways none of the
authors anticipated.

None of these were obviously correct. They were minimal, stable,
and open enough to be useful to people no one had thought of yet.

That is the design goal worth aiming at.

#snip ########################################################################

Protocols follow the same rule.  The constraints of reliable
communication over unreliable channels — distance, error, speed,
confirmation — force the same solutions regardless of medium.
Posting stations with buffer pools, headers announcing the payload,
timeout-triggered retransmission, multiple paths for failover,
message-oriented delivery with acknowledgment.  These were not
invented by Cerf and Kahn in 1974.  They were discovered independently
every time the problem appeared: by the Mongol Yam in the 13th
century, by Chappe's optical telegraph in the 1790s, by Edelcrantz's
shutter system the same decade.  Holzmann — from the same Bell Labs
hallway as Thompson and Ritchie — looked backward 200 years and found
the pipe, the naive process, stderr, flow control, and Thompson's
trust problem, all implemented with wooden arms and telescopes.
Edelcrantz's operator prepared the next signal on a mirror panel
while the current one was being read — pipeline double-buffering
with a footpedal.  Chappe's network had explicit backpressure
signals: Wait, Delay, Adjourn 3 Hour.  The receiver telling the
sender to slow down, 200 years before TCP window scaling.
Protocols are not arbitrary designs to memorize.  They are natural
consequences of constraints.  Learn the constraints and the protocol
follows.

#snip ########################################################################

Redundancy serves horizontal composition.  The 3-finger-claw is copied
into every program instead of sourced from a library.  Three lines of
deliberate redundancy that keep every program a leaf — independently
testable, independently deployable, no dependency to manage.  The
coupling cost of sourcing three functions would exceed the redundancy
cost every time.  "Don't Repeat Yourself" is a reasonable principle in
vertical systems where the cost of a new component is high.  In shell,
where the cost of a new program is near zero and the cost of a missing
dependency is a failure at 3am, the calculus reverses.  Repeat the
three lines.  Pay for what you know, not what you fear.

#snip ########################################################################

The pipe extends across hosts.  ssh(1) is | that crosses a
network boundary.  The local pipe connects stdout to stdin
between processes on the same machine.  ssh connects stdout to
stdin between processes on different machines.  Same abstraction,
same byte stream — the pipe just got longer.

   tar cf - . | ssh host 'tar xf - -C /dst'

The entire contents of the current directory, through a pipe,
across the network, unpacked on the remote host.  No scp.  No
intermediate file.  The data never touches disk in transit.
The processes on each side do not know they are on different
machines, the same way sort(1) does not know it is connected
to uniq(1).

Combine ssh-as-pipe with xargs -P and the pipeline fans out:

   xargs -P 10 -I{} ssh -n {} 'uptime' < hosts.txt

Ten hosts queried in parallel.  Output collected through
stdout.  Two shell utilities — xargs, ssh — that have
been on every UNIX system for decades.  No orchestration
framework.  No configuration management tool.  The frameworks
come and go.  They have version numbers.  ssh host 'cmd'
does not.

The shell programmer who understands pipes, understands
ssh-as-fd, and understands xargs -P has a distributed
computing framework in #!/bin/sh.  The subfloor was always
there.  The industry built products to sell it back to you.

# The FERTILITY section above is itself edited selections.

#snip ########################################################################