1.1   Authors

1.2   Intent

This document is currently maintained in a bzr branch with the intention of:

• automatically-generating an HTML rendering of it for the web.
• packaging the document such that it is available on any Ubuntu system.

1.3   Purpose

The purpose of this document is multi-faceted. It is intended as:

• A gentle introduction to Upstart.
• A Cookbook of recipes and best-practises for solving common and not so common problems.
• An extended guide to the configuration syntax of Upstart.

It attempts to explain the intricacies of Upstart with worked examples and lots of details.

Note that the reference documentation for Upstart will always be the manual pages, this is merely a supplement to them.

1.4   Coverage

It is important to understand that Upstart exists in a number of "guises". At the time of writing, there are essentially two major versions of Upstart:

• The upstream version

  This is the pure, or "vanilla" version.

• The Ubuntu-packaged version [2].

  This is a "debianised" version of Upstart (in other words, a version packaged for Debian and derivatives). It includes a number of enhancements specifically for running Upstart on an Ubuntu system.

Upstart is relied upon by millions of system across a number of different Operating Systems (including Ubuntu, Google's Chromium OS and Google's Chrome OS).

This document is written with Ubuntu in mind, but will attempt to identify Ubuntu-specific behaviour where appropriate.

1.5   Audience

This document is targetted at:

• Users interested in learing about Upstart.
• System Administrators looking to make the most of the capabilities of Upstart.
• Developers and Packagers who wish to package their application to work with Upstart.

1.6   Warning

This document aims to aid understanding of Upstart and identify some hopefully useful "canned" solutions and advice to common problems and questions.

The Authors have taken as much care as possible in the preparation of this document, but you are still advised strongly to exercise caution extreme caution when changing critical system facilities such as the init daemon. Most situations are recoverable and advice is provided in this document, but if your system explodes in a ball of fire or becomes unusable as a result of a suggestion from this document, you alone have the intellectual pleasure of fixing your systems.

2.1   Commands and configuration stanzas

Throughout this document a fixed font such as this will be used to denote commands, brief command output and configuration stanzas.

2.2   User Input and Command Output

An indented block will be used to denote user input and command output.

$ echo hello
hello

# whoami
root

2.3   Configuration Examples

An indented block is also used to show examples of job configuration:

script
  # a config file
end script

3.1   What is Upstart?

Quoting from http://upstart.ubuntu.com/,

Upstart is an event-based replacement for the /sbin/init daemon which handles starting of tasks and services during boot, stopping them during shutdown and supervising them while the system is running.

The "init" or "system initialisation" process on Unix and Linux systems has process ID (PID) "1". That is to say, it is the first process to start when the system boots (ignoring the initrd/initramfs). As the quote shows, Upstart is an "init" replacement for the traditional Unix "System V" "init" system. Upstart provides the same facilities as the traditional "init" system, but surpasses it in many ways.

4.1   Job

A "unit of work" - either a "task" or a "service".

4.2   Event

A notification sent by Upstart to all interested parties (either jobs or other events). They can generally be thought of as "signals", "methods", or "hooks" [7], depending on how they are emitted and/or consumed.

Events are emitted (created and then broadcast) to the entire Upstart system. Note that it is not possible to stop any other job or event from seeing an event when it is emitted.

If there are no jobs which have registered an interest in an event in either their start on or stop on conditions, the event has no effect on the system.

Events can be created by an administrator at any time using:

# initctl emit <event>

Note that some events are "special". See the upstart-events(7) manual page for a list.

Note also that an event name with the same name as a job is allowed.

Jobs are started and stopped by a special set of events:

starting

This event is emitted by Upstart when a job has been scheduled to run and is about to start executing.

started

This event is emitted by Upstart when a job is now running. Note that a job does not have to have an associated program or script to "running" does not necessarily imply that any additional process is executing.

stopping

This event is emitted by Upstart when a job is about to be stopped.

stopped

This event is emitted by Upstart when a job has completed (successfully or otherwise).

See Job Lifecycle for further details.

To help reinforce the difference, consider how Upstart itself starts:

1. It performs its internal initialization.
2. Upstart itself emits a single event called startup (7). This event triggers the rest of the system to initialize. Note that there is no "startup" job (and hence no /etc/init/startup.conf file).
3. init(8) runs the mountall job (as defined in /etc/init/mountall.conf) since the startup(7) event satisfies mountall's requirement, "start on startup".
4. The mountall(8) job in turn emits a number of events (including local-filesystems(7) and all-swaps(7)). See upstart-events(7) for further details.

Upstart provides three different types of Events.

# initctl emit --no-wait mysignal

# initctl emit mymethod

start on mymethod
task
exec /usr/bin/myapp $ACTION

# initctl emit mymethod ACTION=do_something
[ $? -ne 0 ] && { echo "ERROR: myapp failed"; exit 1; }

4.3   Job Lifecycle

4.4   Order in which Events are Emitted

As a general rule, you cannot rely upon the the order in which events will be emitted. Your system is dynamic and Upstart responds to changes as-and-when they occur (for example hot-plug events).

That said, most systems which use Upstart provide a number of "well-known" events which you can rely upon.

For example on Ubuntu, these are documented in the upstart-events(7) man page.

4.5   Order in Which Jobs Which start on the Same Event are Run

Assume you have three jobs like this:

• /etc/init/X.conf

  start on event-A
  

• /etc/init/Y.conf

  start on event-A
  

• /etc/init/Z.conf

  start on event-A
  

Question: If event event-A is emitted, which job will run first?

Answer: It is not possible to say, and indeed you should not make any assumptions about the order in which jobs with the same conditions run in.

5.1   start on

start on (local-filesystems and net-device-up IFACE!=lo)

start on runlevel [2345]

start on started network-services

start on started other-service

start on starting other-service

start on starting apache2
stop on stopped apache2
respawn

exec /usr/sbin/memcached

5.2   stop on

stop on runlevel [016]

stop on stopping network-services

stop on stopping other-service

start on starting other-service

5.3   respawn

Absent this stanza, a job that exits quietly transition into the stop/waiting state, no matter how it exitted. With this stanza, whenever the main script/exec exits without the goal of the job having been changed to 'stop', the job will be started again. This includes running the pre-start and post-start stanzas.

There are a number of reasons why you may or may not want to use this. For most traditional network services this makes good sense. If the tracked process exits for some reason that wasn't the administrator's intent, you probably want to start it back up again.

Likewise, for tasks, (see below), respawning means that you want that task to be retried until it exits with zero(0) as its exit code.

One situation where it may seem like respawn should be avoided, is when a daemon doesn' respond well to SIGTERM for stopping it. You may believe that you need to send the service its shutdown command without upstart being involved, and therefore, you don't want to use respawn because upstart will keep trying to start your service back up when you told it to shutdown.

However, the appopriate way to handle that situation is a pre-stop which runs this shutdown command. Since the job's goal will already be 'stop' when a pre-stop is run, you can shutdown the process through any means, and the process won't be respawned (even with the respawn stanza).

5.4   task

In concept, a task is just a short lived job. In practice, this is accomplished by changing how the transition from a goal of "stop" to "start" is handled.

Without the 'task' keyword, the events that cause the job to start will be unblocked as soon as the job is started. This means the job has emitted a 'starting' event, run its pre-start, begun its script/exec, and post-start, and emitted its 'started' event.

With task, the events that lead to this job starting will be blocked until the job has completely transitioned back to stopped. This means that the job has run up to the previously mentioned 'started' event, and has also completed its post-stop, and emitted its 'stopped' event.

Typically, task is for something that you just want to run and finish completely when a certain event happens.:

# pre-warm-memcache

start on started memcached

task

exec /path/to/pre-warm-memcached

So you can have another job that starts your background queue worker once the local memcached is pre-warmed:

# queue-worker

start on stopped pre-warm-memcache
stop on stopping memcached

respawn

exec /usr/local/bin/queue-worker

The key concept demonstrated above is that we 'start on stopped pre-warm-memcache'. This means that we don't start until the task has completed. If we were to use started instead of stopped, we would start our queue worker as soon as /path/to/pre-warm-memcached had been started running.

We could also accomplish this without mentioning the pre-warm in the queue-worker job by doing this:

# queue-worker

start on started memcached
stop on stopping memcached

respawn

exec /usr/local/bin/queue-worker

# pre-warm-memcache

start on starting queue-worker
task
exec /path/to/pre-warm-memcache

If we did not use 'task' in the above example, queue-worker would be allowed to start as soon as we executed /path/to/pre-warm-memcache, which means it might potentially start before the cache was warmed.

5.5   kill timeout

The number of seconds Upstart will wait before killing a process. The default is 5 seconds.

5.6   normal exit

Used to change Upstarts idea of what a "normal" exit status is. Conventially, processes exit with status "0" (zero) to denote success and non-zero to denote failure. If your application can exit with exit status "13" and you want Upstart to consider this as an normal (successful) exit, then you can specify:

normal exit 0 13

You can even specify signals. For example, to consider exit codes "0", "13" as success and also to consider the program to have completed successfully if it exits on signal "SIGUSR1" and "SIGWINCH", specify:

normal exit 0 13 SIGUSR1 SIGWINCH

5.7   expect

Upstart will keep track of the process ID that it thinks belongs to a job (or multiple if it has instances)

If you do not give an expect line, then Upstart will track the life cycle of the exact pid that it executes. However, many Unix services will "daemonize", meaning that they will create a new process that lets go of the terminal and other bits, and exits.

In this case, Upstart must have a way to track it, so you can use expect fork, or expect daemon.

5.8   pre-start

Use this stanza to prep the environment for the job. Clearing out cache/tmp dirs is a good idea, but any heavy logic is discouraged, as usptart job files should read like configuration files, not so much like complicated software.:

pre-start script
  [ -d "/var/cache/squid" ] || squid -k
end script

Another possibility is to cancel the start of the job for some reason. One good reason is that its clear from the system configuration that a service is not needed:

pre-start script
  if ! grep -q 'parent=foo' /etc/bar.conf ; then
    stop ; exit 0
  fi
end script

Note that the 'stop' command did not receive any arguments. This is a shortcut available to jobs where the 'stop' command will look at the current environment and determine that you mean to stop the current job.

5.9   post-start

Use this stanza when the job executed needs to be waited for before being considered "started". An example is mysql.. after executing it, it may need to perform recovery operations before accepting network traffic. Rather than start dependent services, you can have a post-start like this:

post-start script
  while ! mysqladmin ping localhost ; do sleep 1 ; done
end script

5.10   pre-stop

Stopping a job will involve sending SIGTERM to it. If there is anything that needs to be done before SIGTERM, do it here. Arguably, services should handle SIGTERM very gracefully, so this shouldn't be necessary. However, if the service takes > kill timeout seconds (default, 5 seconds) then it will be sent SIGKILL, so if there is anything critical, like a flush to disk, and raising kill timeout is not an option, pre-stop is not a bad place to do it. [5]

You can also use this stanza to cancel the stop, in a similar fashion to the way one can cancel the start in the pre-start.

5.11   post-stop

There are times where the cleanup done in pre-start is not enough. Ultimately, the cleanup should be done both pre-start and post-stop, to ensure the service starts with a consistent environment, and does not leave behind anything that it shouldn't.

exec / script

If it is possible, you'll want to run your daemon with a simple exec line. Something like this:

exec /usr/bin/mysqld

If you need to do some scripting before starting the daemon, script works fine here. Here is one example of using a script stanza that may be non-obvious:

# statd - NSM status monitor

description   "NSM status monitor"
author                "Steve Langasek <steve.langasek@canonical.com>"

start on (started portmap or mounting TYPE=nfs)
stop on stopping portmap

expect fork
respawn

env DEFAULTFILE=/etc/default/nfs-common

pre-start script
    if [ -f "$DEFAULTFILE" ]; then
        . "$DEFAULTFILE"
    fi

    [ "x$NEED_STATD" != xno ] || { stop; exit 0; }

    start portmap || true
    status portmap | grep -q start/running
    exec sm-notify
end script

script
    if [ -f "$DEFAULTFILE" ]; then
        . "$DEFAULTFILE"
    fi

    if [ "x$NEED_STATD" != xno ]; then
        exec rpc.statd -L $STATDOPTS
    fi
end script

Because this job is marked "respawn", an exit of 0 is "ok" and will not force a respawn (only exitting with a non-0 exit or being killed by an unexpected signal causes a respawn), this script stanza is used to start the optional daemon rpc.statd based on the defaults file. If NEED_STATD=no is in /etc/default/nfs-common , this job will run this snippet of script, and then the script will exit with 0 as its return code. Upstart will not respawn it, but just gracefully see that it has stopped on its own, and return to 'stopped' status. If, however, rpc.statd had been run, it would stay in the 'start/running' state and be tracked normally.

5.12   instance

Sometimes you want to run the same job, but with different arguments. The variable that defines the unique instance of this job is defined with 'instance'.

Lets say that once memcached is up and running, we want to start a queue worker for each directory in /var/lib/queues:

# queue-workers

start on started memcached

task

script
  for dir in `ls /var/lib/queues` ; do
    start queue-worker QUEUE=$dir
  done
end script

And now:

# queue-worker

stop on stopping memcached

respawn

instance $QUEUE

exec /usr/local/bin/queue-worker $QUEUE

In this way, Upstart will keep them all running with the specified arguments, and stop them if memcached is ever stopped.

5.13   manual

Added in upstart v0.6.7

This stanza will tell Upstart to ignore the start on / stop on stanzas. It is useful for keeping the logic and capability of a job on the system while not having it automatically start at bootup.

6.1   Really understanding start on and stop on

(Note: This section focusses on start on, but the information also applies to stop on unless explicitly specified).

The start on stanza needs careful contemplation. Consider this example:

start on started mysql

The syntax above is actually a short-hand way of writing:

start on started JOB=mysql

Remember that started(7) is an event which Upstart emits automatically when the mysql job has started to run. The whole start on stanza can be summarized as:

start on <event> [<vars_to_match_event_on>]

Where <vars_to_match_event_on> is optional, but if specified comprises one or more variables.

A slight variation of the above:

start on started JOB=mydb DBNAME=foobar

This example shows that the fictitious job above would only be started when the mydb database server brings the foobar database online. Correspondingly, file /etc/init/mydb.conf would need to specify "export DBNAME" and be started like this:

start mydb DBNAME=foobar

Looking at a slightly more complex real-life example:

# /etc/init/alsa-mixer-save.conf
start on starting rc RUNLEVEL=[06]

This job says,

"Run when the rc job emits the starting event, but only if the

environment variable RUNLEVEL equals either 0 (halt) or 6 (reboot)".

If we again add in the implicit variable it becomes clearer:

# /etc/init/alsa-mixer-save.conf
start on starting JOB=rc RUNLEVEL=[06]

But where does the RUNLEVEL environment variable come from? Well, variables are exported in a job configuration file to related jobs. Thus, the answer is The rc Job.

If you look at this job configuration file, you will see, as deduced:

export RUNLEVEL

# /etc/init/rc.conf
start on runlevel [0123456]
stop on runlevel [!$RUNLEVEL]

export RUNLEVEL
export PREVLEVEL

console output
env INIT_VERBOSE

task

exec /etc/init.d/rc $RUNLEVEL

stop on runlevel [!$RUNLEVEL]

stop on runlevel [!2]

6.2   Environment Variables

Upstart allows you to set environment variables which will be accessible to the jobs whose job configuration files they are defined in.

For example:

# /etc/init/env.conf
env TESTING=123

script
  # prints "TESTING='123'" to system log
  logger -t $0 "TESTING='$TESTING'"
end script

Further, we can pass environment variables defined in events to jobs. Assume we have two job configuration files, A.conf and B.conf:

# /etc/init/A.conf
start on wibble
export foo

# /etc/init/B.conf
start on A
script
  logger "value of foo is '$foo'"
end script

If we now run the following command, both jobs A and B will run, causing B to write "value of foo is 'bar'`" to the system log:

# initctl emit wibble foo=bar

Note that job configuration files do not have access to a users environment variables, not even the superuser. This is not possible since all job processes created are children of init which does not have a users environment.

However, using the technique above, it is possible to inject a variable from a users environment into a job indirectly:

# initctl emit wibble foo=bar USER=$USER

As a final example of environment variables, consider this job configuration file [5]:

env var=bar
export var

pre-start script
  logger "pre-start: before: var=$var"

  var=pre-start
  export var

  logger "pre-start: after: var=$var"
end script

post-start script
  logger "post-start: before: var=$var"

  var=post-start
  export var

  logger "post-start: after: var=$var"
end script

script
  logger "script: before: var=$var"

  var=main
  export var

  logger "script: after: var=$var"
end script

post-stop script
  logger "post-stop: before: var=$var"

  var=post-stop
  export var

  logger "post-stop: after: var=$var"
end script

This will generate output in your system log as follows (the timestamp and hostname have been removed, and the output formatted to make it clearer):

logger: pre-start:  before: var=bar
logger: pre-start:   after: var=pre-start

logger: post-start: before: var=bar
logger: post-start:  after: var=post-start

logger: script:     before: var=bar
logger: script:      after: var=main

logger: post-stop:  before: var=bar
logger: post-stop:   after: var=post-stop

As shown, every script section receives the value of $var as bar, but if any script section changes the value, it only affects that particular script sections copy of the variable. To summarize:

A script section cannot modify the value of a variable defined in a job configuration file for other script sections.

7.1   D-Bus Service Activation

As of D-Bus version 1.4.1-0ubuntu2, you can have Upstart start a D-Bus service rather than D-Bus. This is useful because it is then possible to create Upstart jobs that start or stop when D-Bus services start.

See Run a Job When a User Logs in for an example.

8.1   Utilities

8.2   Bridges

Bridges react to events from some other (non-Upstart) source and create corresponding Upstart events.

<subsystem>-device-<action>

for subsystem in /sys/class/*
do
  for action in added changed removed
  do
    echo "${subsystem}-device-${action}"
  done
done

# udevadm info --export-db

$ udevadm monitor --environment

start on (graphics-device-added or drm-device-added)

start on (graphics-device-added or drm-device-added)
script
  echo "`env`" > /dev/.initramfs/job-A.log
end script

DEV_LOG=3
DEVNAME=/dev/fb0
UPSTART_INSTANCE=
ACTION=add
SEQNUM=1176
MAJOR=29
KERNEL=fb0
DEVPATH=/devices/platform/efifb.0/graphics/fb0
UPSTART_JOB=job-A
TERM=linux
SUBSYSTEM=graphics
PATH=/usr/local/sbin:/usr/local/bin:/usr/bin:/usr/sbin:/sbin:/bin
MINOR=0
UPSTART_EVENTS=graphics-device-added
PWD=/
PRIMARY_DEVICE_FOR_DISPLAY=1

ID_BUS=pci
UDEV_LOG=3
UPSTART_INSTANCE=
ID_VENDOR_FROM_DATABASE=Realtek Semiconductor Co., Ltd.
ACTION=add
SEQNUM=1171
MATCHADDR=52:54:00:12:34:56
IFINDEX=2
KERNEL=eth0
DEVPATH=/devices/pci0000:00/0000:00:03.0/net/eth0
UPSTART_JOB=job-A
TERM=linux
SUBSYSTEM=net
ID_MODEL_ID=0x8139
PATH=/usr/local/sbin:/usr/local/bin:/usr/bin:/usr/sbin:/sbin:/bin
ID_MM_CANDIDATE=1
ID_MODEL_FROM_DATABASE=RTL-8139/8139C/8139C+
UPSTART_EVENTS=net-device-added
INTERFACE=eth0
PWD=/
MATCHIFTYPE=1
ID_VENDOR_ID=0x10ec

DEV_LOG=3
DEVNAME=/dev/input/event12
UPSTART_INSTANCE=
ACTION=add
SEQNUM=2689
XKBLAYOUT=gb
MAJOR=13
ID_INPUT=1
KERNEL=event12
DEVPATH=/devices/pci0000:00/0000:00:1d.0/usb2/2-1/2-1.2/input/input33/event12
UPSTART_JOB=test_camera
TERM=linux
DEVLINKS=/dev/char/13:76 /dev/input/by-path/pci-0000:00:1d.0-event
SUBSYSTEM=input
PATH=/usr/local/sbin:/usr/local/bin:/usr/bin:/usr/sbin:/sbin:/bin
MINOR=76
DISPLAY=:0.0
ID_INPUT_KEY=1
ID_PATH=pci-0000:00:1d.0
UPSTART_EVENTS=input-device-added
PWD=/

9.1   List All Jobs

To list all jobs on the system along with their states, run:

$ initctl list

9.2   Create an Event

To create, or "emit" an event, use initctl(8) specifying the emit command.

For example, to emit the hello event, you would run:

# initctl emit hello

This event will be "broadcast" to all Upstart jobs.

If you are creating a job configuration file for a new application, you probably do not need to do this though, since Upstart emits events on behalf of a job whenever the job changes state.

A simple configuration file like that shown below may suffice for your application:

# /etc/init/myapp.conf
description "run my app under Upstart"
task
exec /path/to/myapp

9.3   Create an Event Alias

Say you have an event, but want to create a different name for it, you can simulate a new name by creating a new job which:

• has a start on that matches the event you want to "rename"
• is a task
• emits the new name for the event

For example, if you wanted to create an alias for a particular flavour of the runlevel event called "shutdown" which would be emitted when the system was shutdown, you could create a job configuration file called /etc/init/shutdown.conf containing:

start on runlevel RUNLEVEL=0
task
exec initctl emit shutdown

Note that this isn't a true alias since:

• there are now two events which will be generated when the system is shutting down:
  • runlevel RUNLEVEL=0
  • shutdown
• the two events will be delivered by Upstart at slightly different times (shutdown will be emitted just fractionally before runlevel RUNLEVEL=0).

However, the overall result might suffice for your purposes such that you could create a job configuration file like the following which will run (and complete) just before your system changes to runlevel 0 (in other words halts):

start on shutdown
task
exec backup_my_machine.sh

9.4   Synchronisation

Upstart is very careful to ensure when a condition becomes true that it starts all relevant jobs in sequence (see Order in Which Jobs Which start on the Same Event are Run). However, although Upstart has started them one after another they might still be running at the same time. For example, assume the following:

• /etc/init/X.conf

  start on event-A
  script
    echo "`date`: $UPSTART_JOB started" >> /tmp/test.log
    sleep 2
    echo "`date`: $UPSTART_JOB stopped" >> /tmp/test.log
  end script
  

• /etc/init/Y.conf

  start on event-A
  
  script
    echo "`date`: $UPSTART_JOB started" >> /tmp/test.log
    sleep 2
    echo "`date`: $UPSTART_JOB stopped" >> /tmp/test.log
  end script
  

• /etc/init/Z.conf

  start on event-A
  
  script
    echo "`date`: $UPSTART_JOB started" >> /tmp/test.log
    sleep 2
    echo "`date`: $UPSTART_JOB stopped" >> /tmp/test.log
  end script
  

Running the following will cause all the jobs above to run in some order:

# initctl emit event-A

Here is sample output of /tmp/test.log:

Thu Mar 31 10:20:44 BST 2011: Y start
Thu Mar 31 10:20:44 BST 2011: X start
Thu Mar 31 10:20:44 BST 2011: Z start
Thu Mar 31 10:20:46 BST 2011: Y stop
Thu Mar 31 10:20:46 BST 2011: Z stop
Thu Mar 31 10:20:46 BST 2011: X stop

There are a few points to note about this output:

• All jobs start "around the same time" but are started sequentially.
• The order the jobs are initiated by Upstart cannot be predicted.
• All three jobs are running concurrently.

It is possible with a bit of thought to create a simple framework for synchronisation. Take the following job configuration file /etc/init/synchronise.conf:

manual

This one-line Abstract Job configuration file is extremely interesting in that:

• Since it includes the manual keyword, a job created from it can only be started manually.
• Only a single instance of a job created from this configuration can exist (since no instance stanza has been specified).

What this means is that we can use a job based on this configuration as a simple synchronisation device.

The astute reader may observe that synchronise has similar semantics to a POSIX pthread condition variable.

Now we have our synchronisation primitive, how do we use it? Here is an example which we'll call /etc/init/test_synchronise.conf:

start on stopped synchronise

# allow multiple instances
instance $N

# this is not a service
task

pre-start script
  # "lock"
  start synchronise || true
end script

script
  # do something here, knowing that you have exclusive access
  # to some reasource that you are using the "synchronise"
  # job to protect.
  echo "`date`: $UPSTART_JOB ($N) started" >> /tmp/test.log
  sleep 2
  echo "`date`: $UPSTART_JOB ($N) stopped" >> /tmp/test.log
end script

post-stop script
  # "unlock"
  stop synchronise || true
end script

For example, to run 3 instances of this job, run:

for n in $(seq 3)
do
  start test_synchronise N=$n
done

Here is sample output of /tmp/test.log:

Thu Mar 31 10:32:20 BST 2011: test_synchronise (1) started
Thu Mar 31 10:32:22 BST 2011: test_synchronise (1) stopped
Thu Mar 31 10:32:22 BST 2011: test_synchronise (2) started
Thu Mar 31 10:32:24 BST 2011: test_synchronise (2) stopped
Thu Mar 31 10:32:25 BST 2011: test_synchronise (3) started
Thu Mar 31 10:32:27 BST 2011: test_synchronise (3) stopped

The main observation here:

• Each instance of the job started and stopped before any other instance ran.

Like condition variables, this technique require collaboration from all parties. Note that you cannot know the order in which each instance of the test_synchronise job will run.

Note too that it is not necessary to use instances here. All the is required is that your chosen set of jobs all collaborate in their handling of the "lock". Instances make this simple since you can spawn any number of jobs from a single "template" job configuration file.

9.5   Stop a Job from Running if A pre-start Condition Fails

If you wish a job to not be run if a pre-start condition fails:

pre-start script
  # main process will not be run if /some/file does not exist
  test -f /some/file || { stop ; exit 0; }
end script

script
  # main process is run here
end script

9.6   Run a Job Only When an Event Variable Matches Some Value

By default, Upstart will run your job if the start on condition matches the events listed:

start on event-A

But if event-A provides a number of environment variables, you can restrict your job to starting only when one or more of these variables matches some value. For example:

start on event-A FOO=hello BAR=wibble

Now, Upstart will only run your job if all of the following are true:

• the event-A is emitted
• the value of the $FOO variable in event-A's environment is "hello".
• the value of the $BAR variable in event-A's environment is "wibble".

9.7   Run a Job when an Event Variable Does Not Match Some Value

Upstart supports negation of environment variable values such that you can say:

start on event-A FOO=hello BAR!=wibble

Now, Upstart will only run your job if all of the following are true:

• the event-A is emitted
• the value of the $FOO variable in event-A's environment is "hello".
• the value of the $BAR variable in event-A's environment is not "wibble".

9.8   Run a Job as Soon as Possible After Boot

(Note: we ignore the initramfs in this section).

To start a job as early as possible, simply "start on" the startup event. This is the first event Upstart emits and all other events and jobs follow from this:

start on startup

9.9   Run a Job When a User Logs in Graphically (possibly Ubuntu-specific)

Assuming a graphical login, this can be achieved using a start on condition of:

start on desktop-session-start

This requires the display manager emit the event in question. See the upstart-events (7) man page on an Ubuntu system for the 2 events a Display Manager is expected to emit. If your Display Manager does not emit these event, check its documentation to see if it allows scripts to be called at appropriate points and then you can easily conform to the reference implementations behaviour:

# A user has logged in
/sbin/initctl -q emit desktop-session-start \
  DISPLAY_MANAGER=some_name USER=$USER

# Display Manager has initialised and displayed a login screen
# (if appropriate)
/sbin/initctl -q emit login-session-start \
  DISPLAY_MANAGER=some_name

9.10   Run a Job When a User Logs in

This makes use of D-Bus Service Activation.

1. Add "UpstartJob=true" to file "/usr/share/dbus-1/system-services/org.freedesktop.ConsoleKit.service".

2. Create a job configuration file corresponding to the D-Bus service, say /etc/init/user-login.conf [1]:

   start on dbus-activation org.freedesktop.ConsoleKit
   exec /usr/sbin/console-kit-daemon --no-daemon
   

3. Ensure that the D-Bus daemon ("dbus-daemon") is started with the --activation=upstart option (see /etc/init/dbus.conf).

Now, when a user logs in, D-Bus will emit the dbus-activation event, specifying the D-Bus service started. You can now create other jobs that start on user-login.

UPSTART_INSTANCE=
DBUS_STARTER_BUS_TYPE=system
UPSTART_JOB=user-login
TERM=linux
PATH=/usr/local/sbin:/usr/local/bin:/usr/bin:/usr/sbin:/sbin:/bin
SERVICE=org.freedesktop.ConsoleKit
DBUS_SYSTEM_BUS_ADDRESS=unix:path=/var/run/dbus/system_bus_socket,guid=e86f5a01fbb7f5f1c22131090000000a
UPSTART_EVENTS=dbus-activation
PWD=/
DBUS_STARTER_ADDRESS=unix:path=/var/run/dbus/system_bus_socket,guid=e86f5a01fbb7f5f1c22131090000000a

9.11   Run a Job For All of a Number of Conditions

If you have a job configuration file like this:

start on (event-A or (event-B or event-C))

script
  echo "`date`: ran in environment: `env`" >> /tmp/myjob.log
end script

Upstart will run this job when any of the following events is emitted:

• event-A
• event-B
• event-C

You cannot know the order in which the events will arrive in, but the specified start on condition has told Upstart that any of them will suffice for your purposes. So, if event-B is emitted first, Upstart will run the job and only consider re-running the job if and when the job has finished running. If event-B is emitted and the job is running and then (before the job finishes running) event-A is emitted, the job will not be re-run.

However, what if you wanted to run the script for all the events? If you know that all of these events will be emitted at some point, you could change the start on to be:

start on (event-A and (event-B and event-C))

Here, the job will only run at the time when the last of the three events is received.

Is it possible to run this job for each event as soon as each event arrives? Yes it is:

start on (event-A or (event-B or event-C))

instance $UPSTART_EVENTS

script
  echo "`date`: ran in environment: `env`" >> /tmp/myjob.log
end script

By adding the instance keyword, you ensure that whenever any of the events listed in your start on condition is emitted, an instance of the job will be run. Therefore, if all three events are emitted very close together in time, three jobs instances will now be run.

See the Instance section for further details.

9.12   Run a Job Before Another Job

If you wish to run a particular job before some other job, simply make your jobs start on condition specify the starting event. Sine the starting event is emitted just before the job in question starts, this provides the behaviour you want since your job will be run first.

For example, assuming your job is called job-B and you want it to start before job-A, in /etc/init/job-B.conf you would specify:

start on starting job-A

9.13   Run a Job After Another Job

If you have a job you wish to run after job "job-A", your start on condition would need to make use of the stopped event like this:

start on stopped job-A

9.14   Run a Job Before Another Job and Stop it After that Job Stops

If you have a job you wish to be running before job "job-A" starts, but which you want to stop as soon as job-A stops:

start on starting job-A
stop on stopped job-A

9.15   Run a Job Only If Another Job Fails

To have a job start only when job-A fails, use the $RESULT variable from the stopped (7) event like this:

start on stopped job-A RESULT=ok

9.16   Run a Job Only If Another Job Succeeds

To have a job start only when job-A succeeds, use the $RESULT variable from the stopped (7) event like this:

start on stopped job-A RESULT=failed

Note that you could also specify this condition as:

start on stopped job-A RESULT!=ok

9.17   Run a Job Only If One Job Succeeds and Another Fails

This would be a strange scenario to want, but it is quite easy to specify. Assuming we want a job to start only if job-A succeeds and if job-B fails:

start on stopped job-A RESULT=ok and stopped job-B RESULT=failed

9.18   Run a Job If Another Job Exits with a particular Exit Code

Imagein you have a database server process that exits with a particular exit code (say 7) to denote that it needs some sort of cleanup process to be run before it can be re-started. To handle this you could create /etc/init/mydb-cleanup.conf with a start on condition like this:

start on stopped mydb EXIT_STATUS=7

script
  # handle cleanup...

  # assuming the cleanup was successful, restart the server
  start mydb
end script

9.19   Use Details of a Failed Job from Another Job

Although you cannot see the exact environment another job ran in, you can access some details. For example, if your job specified /etc/init/job-B.conf as:

start on stopped job-A RESULT=fail

script
  exec 1>>/tmp/log.file
  echo "Environment of job $JOB was:"
  env
  echo
end script

The file /tmp/log.file might contain something like this:

UPSTART_INSTANCE=
EXIT_STATUS=7
INSTANCE=
UPSTART_JOB=B
TERM=linux
PATH=/usr/local/sbin:/usr/local/bin:/usr/bin:/usr/sbin:/sbin:/bin
PROCESS=main
UPSTART_EVENTS=stopped
PWD=/
RESULT=failed
JOB=A

Here, job-B can see that:

• job-A exited in its "main" process. This is a special name for the script section. All other script sections are named as expected. For example, if the pre-start section had failed, the PROCESS variable would be set to pre-start, and if in post-stop, the variable would have been set to post-stop.
• job-A exited with exit code 7.
• job-A only had 1 instance (since the INSTANCE variable is set to the null value.
• job-A ran in the root ("/") directory.
• UPSTART_JOB is the name of the job running the script (ie job-B).
• JOB is the name of the job that we are starting on (here job-A).
• UPSTART_EVENTS is a list of the events that caused UPSTART_JOB (ie job-B) to start. Here, the event is starting showing that job-B started as a result of job-A being sent the stopped event.

9.20   Stop a Job when Another Job Starts

If we wish job-A to stop when job-B starts, specify the following in /etc/init/job-A.conf:

stop on starting job-B

# /etc/init/job-A.conf
start on stopped job-B

script
  # do something when job-B is stopped
end script

# /etc/init/job-B.conf
start on stopped job-A

script
  # do something when job-A is stopped
end script

# start job-A

9.21   Run a Job Periodically

This cannot currently be handled by Upstart directly. However, the "Temporal Events" feature is being worked on now will address this.

Until Temporal Events are available you should either use cron (8), or something like:

# /etc/init/timer.conf

instance $JOB_TO_RUN

script
  for var in SLEEP JOB_TO_RUN
  do
    eval val=\${$var}
    if [ -z "$val" ]
    then
      logger -t $0 "ERROR: variable $var not specified"
      exit 1
    fi
  done

  eval _sleep=\${SLEEP}
  eval _job=\${JOB_TO_RUN}

  while [ 1 ]
  do
    stop  $_job || true
    sleep $_sleep
    start $_job || true
  done
end script

Note well the contents of the while loop. We ensure that the commands that might fail are converted into expressions guaranteed to pass. If we did not do this, timer.conf would fail, which would be undesirable. Note too the use of instance to allow more than one instance of the timer job to be running at any one time.

9.22   Migration from System V initialization scripts

With SysV init scripts, the Administrator decides the order that jobs are started in by assigning numeric values to each service. Such a system is simple, but non-optimal since:

• The SysV init system runs each job sequentially.

  This disallows running jobs in parallel, to make full use of system resources. Due to the limited nature of the SysV system, many SysV services put services that take a long time to start into the background to give the illusion that the boot is progressing quickly. However, this makes it difficult for Administrators to know if a required service is running by the time their later service starts.

• The Administrator cannot know the best order to run jobs in.

  Since the only meta information encoded for services is a numeric value used purely for ordering jobs, the system cannot optimize the services since it knowns nothing about the requirements for each job.

In summary, the SysV init system is designed to be easy for the Administrator to use, not easy for the system to optimize.

In order to migrate a service from SysV to Upstart, it is necessary to change your mindset somewhat. Rather than trying to decide which two services to "slot" your service between, you need to consider the conditions that your service needs before it can legitimately be started.

So, if you wished to add a new service that traditionally started before cron(8) or atd(8) you do not need to change the configuration files cron.conf or atd.conf. You can make "insert" your new service by specifying a simple:

# /etc/init/my-service.conf
start on (starting cron or starting atd)

In English, this says,

"start the "my-service" service just before either the cron or the atd services start".

Whether crond or atd actually start first is not a concern for my-service: Upstart ensures that the my-service service will be started before either of them. Even if cron normally starts before atd but for some reason one day atd starts first, Upstart will ensure that my-service will be started before atd.

Note therefore that introducing a new service should not generally require existing job configuration files to be updated.

9.23   How to Establish a Jobs start on and stop on Conditions

How do you establish what values you should specify for a jobs start on and stop on conditions?

This isn't a question that can be answered easily since each job requires a specific and possibly unique set of conditions before it can run. However, this section attempts to provide some guidelines.

Firstly, read the upstart-events manual page (Ubuntu only) carefully as it provides a standard set of "HOOK POINTS" WHICH your job can make use of (atleast for an Ubuntu system).

Now a few questions...

• Does your job need to read from a configuration or data file? If so, where does the file live?:

  • local Filesystem
  • virtual Filesystem
  • remote Filesystem

  What partition does the file live in?:

  • /etc?
  • /var?

• Does your job need to write to a data file or a log file? If so, where will the file live?:

  • local Filesystem
  • virtual Filesystem
  • remote Filesystem

  What partition wll the file live in?:

  • /etc?
  • /var?

• Should a particular set of services already be running when your job starts?

• Should a particular set of services not be running when your job starts?

• What runlevel(s) should your job run in?

• Does your service require a network connection? If so, what type of network? Local networking? Ethernet?

• Should your service only start when a client network connection is initiated? If so, use the socket event (emitted by the upstart-socket-bridge (8)). See the socket-event (7) man page for details.

• Does your application provide a D-Bus service which you want to start when some sequence of Upstart events are emitted? If so, use the D-Bus service activation facility.

Other questions relating to other stanzas:

• What should happen if your job fails to start?
• What should happen if your job fails after some period of time?
• Do you want Upstart to restart the job if it exits? If so, use the respawn stanza.
• Does your job use non-standard exit codes to denote success and failure? If so, use the normal exit stanza.
• Is your job a daemon? If so, how many times does it call fork(2)?

9.24   Guarantee that a job will only run once

If you have a job which must only be run once, but which depends on multiple conditions, the naive approach won't necessarily work:

task
start on (A or B)

If event 'A' is emitted, the task will run. But assuming the task has completed and event 'B' is then emitted, the task will run again.

task
start on startup and (A or B)

9.25   Stop a Job That is About to Start

Upstart will start a job when its "start on" condition becomes true.

Although somewhat unusual, it is quite possible to stop a job from starting when Upstart tries to start it:

start on starting job-A

script
  stop $JOB
end script

9.26   Stop a Job When Some Other Job is about to Start

Here, we create /etc/init/job-C.conf which will stop job-B when job-A is about to start:

start on starting job-A

script
  stop job-B
end script

9.27   Start a Job when a Particular Filesystem is About to be Mounted

Here, we start a job when the /apps mountpoint is mounted read-only as an NFS-v4 filesystem:

start on mounting TYPE=nfs4 MOUNTPOINT=/apps OPTION=ro

Here's another example:

start on mounted MOUNTPOINT=/var/run TYPE=tmpfs

Another example where a job would be started when any non-virtual filesystem is mounted:

start on mounted DEVICE=[/UL]*

The use of the $DEVICE variable is interesting. It is used here to specify succinctly any device that:

• is a real device (starts with "/" (to denote a normal "/dev/..." mount)).
• is a device specified by its filesystem:
  • label (starts with "L" (to denote a "LABEL=" mount)).
  • UUID (starts with "U" (to denote a "UUID=" mount)).

Another example where a job is started when a non-root filesystem is mounted:

start on mounting MOUNTPOINT!=/ TYPE!=swap

9.28   Start a Job when a Device is Hot-Plugged

Hotplug kernel events create udev (7) events under Linux and Upstart events are created from udev events by the upstart-udev-bridge (8).

Added to this the ifup and ifdown commands are run at boot when network devices are available for use.

start on net-device-added INTERFACE=eth0

start on net-device-up IFACE=eth0

9.29   Stopping a Job if it Runs for Too Long

To stop a running job after a certain period of time, create a generic job configuration file like this:

# /etc/init/timeout.conf
stop on stopping JOB=$JOB_TO_WAIT_FOR
kill timeout 1
manual

export JOB_TO_WAIT_FOR
export TIMEOUT

script
  sleep $TIMEOUT
  initctl stop $JOB_TO_WAIT_FOR
end script

Now, you can control a job using a timeout:

start myjob
start timeout JOB_TO_WAIT_FOR=myjob TIMEOUT=5

This will start job myjob running and then wait for 5 seconds. If job "myjob" is still running after this period of time, the job will be stopped using the initctl(8) command. Note the stop on stanza which will cause the timeout job not to run if the job being waited for has already started to stop.

9.30   Run a Job When a File or Directory is Created/Deleted

If you need to start a Job only when a certain file is created, you could create a generic job configuration file such as the following:

# /etc/init/wait_for_file.conf
instance FILE_PATH
export   FILE_PATH

script
  while [ ! -e "$FILE_PATH" ]
  do
    sleep 1
  done

  initctl emit file FILE_PATH="$FILE_PATH"
end script

Having done this, you can now make use of it. To have another job start if say file /var/run/foo.dat gets created, you first need to create a job configuration file stating this:

# /etc/init/myapp.conf
start on file FILE_PATH=/var/run/foo.dat

script
  # ...
end script

Lastly, kick of the process by starting an instance of wait_for_file:

start wait_for_file FILE_PATH=/var/run/foo.dat

Now, when file /var/run/foo.dat is created, the following will happen:

1. The myapp job will emit the file event, passing the path of the file which you just specified in that events environment.
2. Upstart will see that the start on condition for the myapp job configuration file is satisfied.
3. Upstart will create a myapp job, and start it.

You can modify this strategy slightly to run a job when a file is:

• modified
• deleted
• contains certain content
• et cetera

See test(1), or your shells documentation for available file tests.

Note that this is very simplistic. A better approach would be to use inotify(7).

9.31   Run a Job Each Time a Condition is True

This is the default way Upstart works when you have defined a task:

# /etc/init/myjob.conf
task
exec /some/program
start on (A or B)

Job "myjob" will run every time either event 'A' or event 'B' are emitted. However, there is a corner condition: if event 'A' has been emitted and the task is currently running when event 'B' is emitted, job "myjob" will not be run. To avoid this situation, use instances:

# /etc/init/myjob2.conf
task
instance $SOME_VARIABLE
exec /some/program
start on (A or B)

Now, as long variable $SOME_VARIABLE is defined with a unique value each time either event 'A' or 'B' is emitted, Upstart will run job "myjob2" multiple times.

9.32   Run a Job on System Startup/Shutdown

9.33   Pass State From a Script Section to its Job Configuration File

Assume you have a job configuration file like this:

script
   # ...
end script

exec /bin/some-program $ARG

How can you get the script section to set $ARG and have the job configuration file use that value in the "exec" stanza? This isn't as easy as you might imagine for the simple reason that Upstart runs the script section in a new process. As such, by the time Upstart gets to the exec stanza the process spawned to handle the script section has now ended. This implies they cannot communicate directly.

A way to achieve the required goal is as follows:

# set a variable which is the name of a file this job will use
# to pass information between script sections.
env ARG_FILE="/var/myapp/myapp.dat"

# make the variable accessible to all script sections (ie sub-shells)
export ARG

pre-start script
   # decide upon arguments and write them to
   # $ARG_FILE, which is available in this sub-shell.
end script

script
  # read back the contents of the arguments file
  # and pass the values to the program to run.
  ARGS="$(cat $ARG_FILE)"
  exec /bin/some-program $ARGS
end script

9.34   Pass State From Job Configuration File to a Script Section

To pass a value from a job configuration file to one of its script sections, simply use the env stanza:

env CONF_FILE=/etc/myapp/myapp.cfg

script
  exec /bin/myapp -c $CONF_FILE
end script

This example is a little pointless, but the following slightly modified example is much more useful:

start on an-event
export CONF_FILE

script
  exec /bin/myapp -c $CONF_FILE
end script

By dropping the use of the env stanza we can now pass the value in via an event:

# initctl emit an-event CONF_FILE=/etc/myapp/myapp.cfg

This is potentially much more useful since the value passed into myapp.conf can be varied without having to modify the job configuration file.

9.35   Run a Job as a Different User

Upstart has no special syntax for this. Simply use su:

script
su -c command myuser
end script

9.36   Disabling a Job from Automatically Starting

With Upstart 0.6.7, to stop Upstart automatically starting a job, you can either:

• Rename the job configuration file such that it does not end with ".conf".
• Edit the job configuration file and comment out the "start on" stanza using a leading '#'.

To re-enable the job, just undo the change.

With newer versions of Upstart, you can make use of override files and the manual stanza to achieve the same result in a simpler manner:

echo "manual" >> /etc/init/myjob.override

Note that you could achieve the same effect by doing this:

echo "manual" >> /etc/init/myjob.conf

However, using the override facility means you can leave the original job configuration file untouched.

To revert to the original behaviour, either delete or rename the override file (or remove the manual stanza from your ".conf" file).

9.37   Changing the Default Runlevel

Traditionally, the default runlevel was encoded in file /etc/inittab. However, with Upstart, this file is no longer used (it is supported by Upstart, but its use is deprecated).

To change the default runlevel, modify the variable DEFAULT_RUNLEVEL in file /etc/init/rc-sysinit.conf. For example, to make the system boot by default to single user mode, set:

env DEFAULT_RUNLEVEL=1

If you want to change the default runlevel for a single boot, rather than making the change permanent by modify the rc-sysinit.conf file, simply append the variable to the kernel command line:

DEFAULT_RUNLEVEL=1

It is worth noting that Unix and Linux systems are normally confined to only a few runlevels (7 is common), but Upstart allows any number of runlevels to be defined.

9.38   Jobs that "Run Forever"

To create a job that runs continuously from the time it is manually started(7) until the time it is manually stopped(7), create a job configuration file without any process definition (exec and script) or event definition (start on for example) stanzas:

# /etc/init/runforever.conf
description "job that runs until stopped manually"

This job can only be started by the administrator running:

# start runforever

The status of this job will now be "start/running" until the administrator subsequently runs:

# stop runforever

These types of jobs have other uses as covered in other parts of this document.

9.39   Run a Java Application

Running a Java application is no different to any other, but you may wish to define some variables to simplify the invocation:

env HTTP_PORT=8080
env USER=java_user
env JAVA_HOME=/usr/lib/jvm/java-6-openjdk
env JVM_OPTIONS="-Xms64m -Xmx256m"
env APP_OPTIONS="--httpPort=$HTTP_PORT"
env LOGFILE=/var/log/myapp.log

script
  exec su -c "$JAVA_HOME/bin/java $JVM_OPTIONS \
    -jar $ROOT/myjar.jar $APP_OPTIONS > $LOGFILE 2>&1" $USER
end script

exec start-stop-daemon --start --exec $JAVA_HOME/bin/java \
  -- $JAVA_OPTS -jar $SOMEWHERE/file.war

9.40   Ensure a Directory Exists Before Starting a Job

This is a good use of the pre-start stanza:

env DIR=/var/run/myapp
env USER=myuser
env GROUP=mygroup
env PERMS=0755

pre-start script
  mkdir $DIR             || true
  chmod $PERMS $DIR      || true
  chown $USER:$GROUP DIR || true
end script

9.41   Run a GUI Application

To have Upstart start a GUI application, you first need to ensure that the user who will be runnig it has access to the X display. This is achieved using the xhost command.

Once the user has access, the method is the same as usual:

env DISPLAY=:0.0
exec xclock -update 1

9.42   Run an Application through GNU Screen

If you want Upstart to create a GNU Screen (or Byobu) session to run your application in, this is equally simple:

exec su myuser -c "screen -D -m -S MYAPP java -jar MyApp.jar"

9.43   Run Upstart in a chroot Environment

Upstart jobs cannot currently be started in a chroot(2) environment [6] because Upstart acts as a service supervisor, and processes within the chroot are unable to communicate with the Upstart running outside of the chroot (Bug:430224). This will cause some packages that have been converted to use Upstart jobs instead of init scripts to fail to upgrade within a chroot.

Users are advised to configure their chroots with /sbin/initctl pointing to /bin/true, with the following commands run within the chroot:

dpkg-divert --local --rename --add /sbin/initctl
ln -s /bin/true /sbin/initctl

9.44   Record all Jobs and Events which Emit an Event

For example, if you want to record all jobs which emit a started event:

# /etc/init/debug.conf
start on started
script
  exec 1>>/tmp/log.file
  echo "$0:$$:`date`:got called. Environment of job $JOB was:"
  env
  echo
end script

You could also log details of all jobs (except the debug job itself) which are affected by the main events:

# /etc/init/debug.conf
start on ( starting JOB!=debug \
  or started JOB!=debug \
  or stopping JOB!=debug \
  or stopped JOB!=debug )
script
  exec 1>>/tmp/log.file
  echo -n "$UPSTART_JOB/$UPSTART_INSTANCE ($0):$$:`date`:"
  echo    "Job $JOB/$INSTANCE $UPSTART_EVENTS. Environment was:"
  env
  echo
end script

Note that the $UPSTART_JOB and $UPSTART_INSTANCE environment variables refer to the debug job itself, whereas $JOB and $INSTANCE refer to the job which the debug job is triggered by.

9.45   Integrating your New Application with Upstart

Integrating your application into Upstart is actually very simple. However, you need to remember that Upstart is NOT "System V" (aka "SysV"), so you need to think in a different way.

With SysV you slot your service script between other service scripts by specifying a startup number. The SysV init system then runs each script in numerical order. This is very simple to understand and use, but highly inefficient in practical terms since it means the boot cannot be parallelised and thus cannot be optimized.

9.46   Block Another Job Until Yours has Started

It is common that a particular piece of software, when installed, will need to be started before another. The logical conclusion is to use the 'starting' event of the other job:

start on starting foo

This will indeed, block foo from starting until our job has started.

But what if we have multiple events that we need to delay:

start on starting foo or starting network-services

This would seem to make sense. However, if we have a timeline like this:

starting foo
starting our job
starting network-services
started network-services

Network-services will actually NOT be blocked. This is because upstart only blocks an event if that event causes change in the goal of the service. So, we need to make sure upstart waits every time. This can be done by using a "wait job":

# myjob-wait
start on starting foo or starting network-services
stop on started myjob or stopped myjob
instance $JOB
normal exit 2
task
script
  status myjob | grep -q 'start/running' && exit 0
  start myjob || :
  sleep 3600
end script

This is a bit of a hack to get around the lack of state awareness in Upstart. Eventually this should be built in to upstart. The job above will create an instance for each JOB that causes it to start. It will try and check to see if its already running, and if so, let the blocked job go with exit 0. If its not running, it will set the ball in motion for it to start. By doing this, we make it very likely that the stopped or started event for myjob will be emitted (the only thing that will prevent this, is a script line in 'myjob' that runs 'stop'). Because we know we will get one of those start or stopped events, we can just sleep for an hour waiting for upstart to kill us when the event happens.

9.47   Controlling Upstart using D-Bus

Upstart contains its own D-Bus server which means that initctl and any other D-Bus application can control Upstart. The examples below use dbus-send, but any of the D-Bus bindings could be used.

$ dbus-send --system --print-reply --dest=com.ubuntu.Upstart /com/ubuntu/Upstart com.ubuntu.Upstart0_6.GetAllJobs

$ job=myjob
$ dbus-send --system --print-reply --dest=com.ubuntu.Upstart /com/ubuntu/Upstart/jobs/${job}/_ org.freedesktop.DBus.Properties.GetAll string:''

# job=myjob
# dbus-send --system --print-reply --dest=com.ubuntu.Upstart /com/ubuntu/Upstart/jobs/${job} com.ubuntu.Upstart0_6.Job.Start array:string: boolean:true

# job=myjob
# dbus-send --system --print-reply --dest=com.ubuntu.Upstart /com/ubuntu/Upstart/jobs/${job} com.ubuntu.Upstart0_6.Job.Stop array:string: boolean:true

# job=myjob
# dbus-send --system --print-reply --dest=com.ubuntu.Upstart /com/ubuntu/Upstart/jobs/${job} com.ubuntu.Upstart0_6.Job.Restart array:string: boolean:true

9.48   Establish Blocking Job

Image you have just run the following command and it has "blocked" (appeared to hang):

# initctl emit event-A

The reason for the block is that the event-A event changes the goal of "some job", and until the goal has changed, the initctl command will block.

But which job is being slow to change goal? It is now possible to hone in on the problem using initctl show-config in a script such as this:

#!/bin/sh
# find_blocked_job.sh

[ $# -ne 1 ] && { echo "ERROR: usage: $0 <event>"; exit 1; }
event="$1"

# obtain a list of jobs (removing instances)
initctl list | awk '{print $1}' | sort -u | while read job
do
  initctl show-config -e "$job" |\
    egrep "(start|stop) on \<event\>" >/dev/null 2>&1
  [ $? -eq 0 ] && echo $job
done

This will return a list of jobs, one per line. One of these will be the culprit. Having identified the problematic job, you can debug using techniques from the Debugging section.

10.1   Cannot Start a Job

If you have just created or modified a job configuration file such as /etc/init/myjob.conf, but start gives the following error when you attempt to start it:

start: Unknown job: myjob

The likelihood is that the file contains a syntax error. The easiest way to establish if this is true is by running the following command:

$ init-checkconf /etc/init/myjob.conf

If you are wondering why the original error couldn't be more helpful, it is important to remember that the job control commands (start, stop and restart) and initctl communicate with Upstart over D-Bus. The problem here is that Upstart rejected the invalid myjob.conf, so attempting to control that job over D-Bus is nonsensical - the job does not exist.

10.2   Strange Error When Running start/stop/restart or initctl emit

If you attempt to run a job command, or emit an event and you get a D-Bus error like this:

$ start myjob
start: Rejected send message, 1 matched rules; type="method_call", sender=":1.58" (uid=1000 pid=5696 comm="start) interface="com.ubuntu.Upstart0_6.Job" member="Start" error name="(unset)" requested_reply=0 destination="com.ubuntu.Upstart" (uid=0 pid=1 comm="/sbin/init"))

The problem is caused by not running the command as root. To resolve it, either "su -" to root or use a facility such as sudo:

# start myjob
myjob start/running, process 1234

The reason for the very cryptic error is that the job control commands (start, stop and restart) and initctl communicate with Upstart over D-Bus.

10.3   The initctl command shows "the wrong PID"

The likelihood is that you have mis-specified the type of application you are running in the job configuration file. Since Upstart traces or follows fork(2) calls, it needs to know how many forks to expect. If your application forks once, specify the following in the job configuration file:

expect fork

However, if your application forks twice (which all daemon processes should do), specify:

expect daemon

If you do not know how many times the application forks, you may wish to run it using a tool such as strace(1).

See also Alternative Method.

10.4   Symbolic Links don't work in /etc/init

Upstart does not monitor files which are symbolic links since it needs to be able to guarantee behaviour and if a link is broken or cannot be followed (it might refer to a filesystem that hasn't yet been mounted for example), behaviour would be unexpected, and thus undesirable. As such, all system job configuration files must live in or below /etc/init (although user jobs can live in other locations).

12.1   Obtaining a List of Events

To obtain a list of events that have been generated by your system, do one of the following:

grep init: /var/log/syslog

# : same as "``--verbose``"
# initctl log-priority info

# : same as "``--debug``"
# initctl log-priority debug

12.2   See the Environment a Job Runs In

To get a log of the environment variables set when Upstart ran a job you can add simple debug to the appropriate script section. For example:

script
  echo "DEBUG: `set`" >> /tmp/myjob.log

  # rest of script follows...
end script

Alternatively you could always have the script log to the system log:

script
  logger -t "$0" "DEBUG: `set`"

  # rest of script follows...
end script

Or, have it pop up a GUI window for you:

env DISPLAY=:0.0

script
  env | zenity --title="got event $UPSTART_EVENTS" --text-info &
end script

12.3   Obtaining a log of a Script Section

This relies on a trick relating to the early boot process on an Ubuntu system. On the first line below script stanza, add:

exec 2>>/dev/.initramfs/myjob.log
set -x

This will ensure that /bin/sh will log its progress to the file named. The location of this file is special in that /dev/.initramfs/ will be available early on in the boot sequence (before the root filesystem has been mounted read-write).

12.4   Checking a Job Configuration File for Syntax Errors

To ensure that you haven't misused the Upstart syntax, use the init-checkconf command:

$ init-checkconf myjob.conf

12.5   Check a Script Section for Errors

Upstart runs your job using /bin/sh -e for safety reasons: scripts running as the root user need to be well-written!

To check that you haven't made a (shell) syntax error in your script section, you can use sed like this:

$ /bin/sh -n <(sed -n '/^script/,/^end script/p' myjob.conf)

Or for a pre-start script section:

$ /bin/sh -n <(sed -n '/^pre-start script/,/^end script/p' myjob.conf)

No output indicates no syntax errors.

Alternatively, you could wrap this into a script like this:

#!/bin/sh
# check-upstart-script-sections.sh

[ $# -ne 1 ] && { echo "ERROR: usage: $0 <conf_file>"; exit 1; }
file="$1"

[ ! -f "$file" ] && { echo "ERROR: file $file does not exist" >&2; exit 1; }

for v in pre-start post-start script pre-stop post-stop
do
  if egrep -q "\<${v}\>" $file
  then
    sed -n "/^ *${v}/,/^ *end script/p" $file | \
      sh -n || echo "ERROR in $v section"
  fi
done

And run it like this to check all possible script sections for errors:

$ check-upstart-script-sections.sh myjob.conf

12.6   Debugging a Script Which Appears to be Behaving Oddly

If a script section appears to be behaving in an odd fashion, the chances are that one of the commands is failing. Remember that Upstart runs every script section using /bin/sh -e. This means that if any simple command fails, the shell will exit. For example, if file /etc/does-not-exist.cfg does not exist in the example below the script will exit before the shell runs the `if` test:

script
  grep foo /etc/does-not-exist.cfg >/dev/null 2>&1
  if [ $? -eq 0 ]
  then
    echo ok
  else
    echo bad
  fi
end script

In other words, you will get no output from this script if the file grep is attempting to operate on does not exist.

The common idiom to handle possible errors of this type is to convert the simple expression into an expression guaranteed to return true:

script
  # ensure this statement always evaluates to true
  command-that-might-fail || true

  # ditto
  another-command || :
end script

See man sh for further details.

13.1   Boot into Recovery Mode

Select the "recovery" option in the Grub boot menu

This assumes that Upstart (init(8) itself) is usable.

Note that you need to hold down the SHIFT key to see the Grub boot menu.

13.2   Boot to a shell directly

If Upstart (init(8)) itself has broken, you'll need to follow the steps below. By specifying an alternate "initial process" (here a shell) it is possible to repair the system.

1. Hold down SHIFT key before the splash screen appears (this will then display the grub menu).

2. Type, "e" to edit the default kernel command-line.

3. Use the arrow keys to go to the end of the line which starts "linux /boot/vmlinuz ...".

4. Press the END key (or use arrows) to go to end of the line.

5. Add a space followed by "init=/bin/sh".

6. If the line you are editing contains "quiet" and/or "splash", remove them.

7. Press CONTROL+x to boot with this modified kernel command line.

8. When the shell appears you will need to remount the root filesystem read-write like this:

   # mount -oremount,rw /
   

You can now make changes to your system as necessary.

14.1   Restarting Jobs with Complex Conditions

The and and or operators allowed with start on and stop on do not work intuitively: operands to the right of either operator are only evaluated once and state information is then discarded. This can lead to jobs with complex start on or stop on conditions not behaving as expected when restarted. For example, if a job specifies the following condition:

start on A and (B or C)

When "A`" and "B" become true, the condition is satisfied so the job will be run. However, if the job ends and subsequently "A" and "C" become true, the job will not be re-run even though the condtion is satisfied.

14.2   Bugs

See:

• Upstart bugs

  https://bugs.launchpad.net/upstart

• Ubuntu-specific Upstart bugs

  https://launchpad.net/ubuntu/+source/upstart/+bugs

15.1   Manual Pages

• man 5 init

  Configuration syntax reference.

• man 8 init

  Options for running the Upstart init daemon.

• man 8 initctl

  Explanation of the Upstart control command.

• man 7 upstart-events [3]

  Comprehensive summary of all "well-known" Upstart system events. Available by default on Ubuntu systems only.

15.2   Web Sites

• http://upstart.ubuntu.com/

  Main Ubuntu page for Upstart.

• http://launchpad.net/upstart

  The main Upstart Bazaar project page.

• http://upstart.at

  The New Upstart Blog site.

• http://netsplit.com/category/tech/upstart/

  Scotts Original Upstart blog with useful overviews of features and Concepts.

• https://wiki.ubuntu.com/ReplacementInit

  Original Specification.

15.3   Mailing Lists

• https://lists.ubuntu.com/mailman/listinfo/upstart-devel