Monday, September 10, 2012

out: an output utility

Intro

Whilst writing the tests for the Job Logging feature in Upstart, I discovered that I needed some very specific job behaviour that was not easy to produce using any simple existing utility I could find.

The Problem

What I wanted was a utility that would produce specified amounts of data to either standard output, standard error or direct to the terminal without using shell redirection. The reason for eschewing the shell being that Upstart is clever and intelligently determines whether a shell should be involved. From a user perspective, that's fantastic, but for testing purposes, I needed to force Upstart down particular code paths where it would not for example automatically pass the job through a shell. In fact the constraint was even more restrictive; what I really wanted to do was this:
Produce output (including null bytes) to stdout, stderr or the terminal without using any shell meta characters.
Unless I used some extremely esoteric shell, that effectively meant "no shells" ;-) ksh and zsh actually came close as their printf/print shell built-ins allow output to a specified file descriptor, but not to the terminal. Anyway, by this stage, I'd decided I wanted a generic utility, not some specific shell feature.

The Pragmatic Solution

For the Upstart tests, I ended up using various different command-line utilities for different tests that could produce the data I wanted. However, I couldn't help thinking there should be a better way. Yes, I could have written a bespoke C program or Python/Perl script to achieve my goal, but in the spirit of Larry Walls view that "Easy things should be easy", I thought that there should be a simpler method.

The Improved Solution


In the end this niggling problem got the better of me so I decided to write a tool to satisfy the above constraint. What I came up with is a small utility which sits somewhere between echo(1) and printf(1) (with just a dash of seq(1) thrown in) which, since it produces output, I've called simply "out".

Here's an example of how you could print "hello" direct to the terminal (the redirection here is just to show that you still get output even though we've discarded stdout and stderr output):
$ out -t hello >/dev/null 2>&1
hello

This command below will echo "hello" to stdout, the terminal and stderr:
$ out hello -t -r 1 -e -r 1
Along with allowing output redirection to any file descriptor or the terminal without the support of shell redirection, it supports the standard C-style escape sequences (like '\n' and '\a'). It also allows you to redefine the escape character.

Additionally, out supports printf(1)-style Unicode escape sequences such as '\uFFFFFFFF' . After adding the Unicode support, I thought it would be rather fun to add ranges (similar to bash "sequence expressions") so out also supports some new sequences that allow sets of characters to be generated. These are great fun for exploring Unicode. Here's an example of generating the alphabet in lower-case:
$ out '\{a..z}\n'
abcdefghijklmnopqrstuvwxyz
How about some Greek?
$ out "\{α..ω}\n"
αβγδεζηθικλμνξοπρςστυφχψω

Or we could specify this using Unicode characters to get the same result:


$ out "\{\u03b1..\u03c9}\n"
αβγδεζηθικλμνξοπρςστυφχψω
And here's how you could generate the Unicode braille block:
$ out '\{\u2800..\u28FF}\n'
⠀⠁⠂⠃⠄⠅⠆⠇⠈⠉⠊⠋⠌⠍⠎⠏⠐⠑⠒⠓⠔⠕⠖⠗⠘⠙⠚⠛⠜⠝⠞⠟⠠⠡⠢⠣⠤⠥⠦⠧⠨⠩⠪⠫⠬⠭⠮⠯⠰⠱⠲⠳⠴⠵⠶⠷⠸⠹⠺⠻⠼⠽⠾⠿⡀⡁⡂⡃⡄⡅⡆⡇⡈⡉⡊⡋⡌⡍⡎⡏⡐⡑⡒⡓⡔⡕⡖⡗⡘⡙⡚⡛⡜⡝⡞⡟⡠⡡⡢⡣⡤⡥⡦⡧⡨⡩⡪⡫⡬⡭⡮⡯⡰⡱⡲⡳⡴⡵⡶⡷⡸⡹⡺⡻⡼⡽⡾⡿⢀⢁⢂⢃⢄⢅⢆⢇⢈⢉⢊⢋⢌⢍⢎⢏⢐⢑⢒⢓⢔⢕⢖⢗⢘⢙⢚⢛⢜⢝⢞⢟⢠⢡⢢⢣⢤⢥⢦⢧⢨⢩⢪⢫⢬⢭⢮⢯⢰⢱⢲⢳⢴⢵⢶⢷⢸⢹⢺⢻⢼⢽⢾⢿⣀⣁⣂⣃⣄⣅⣆⣇⣈⣉⣊⣋⣌⣍⣎⣏⣐⣑⣒⣓⣔⣕⣖⣗⣘⣙⣚⣛⣜⣝⣞⣟⣠⣡⣢⣣⣤⣥⣦⣧⣨⣩⣪⣫⣬⣭⣮⣯⣰⣱⣲⣳⣴⣵⣶⣷⣸⣹⣺⣻⣼⣽⣾⣿
out also allows arbitrary delays to be introduced, repeats, basic random character generation and a few other tidbits. Here is the man page including a lot of examples:


OUT(1)         OUT(1)



NAME
       out - manual page for out


SYNOPSIS
       out [OPTION]... [STRING]...

DESCRIPTION
       Echo strings to specified output stream(s).

OPTIONS
       -a, --intra-char=<char>
       Insert  specified  character  (which may be a 1-character escape
       character) between all output characters.

       -b, --intra-pause=<delay>
       Pause between writing each character.

       -e, --stderr
       Write subsequent strings to standard error (file descriptor 2).

       -h, --help
       This help text.

       -i, --interpret
       Interpret escape characters (default).

       -l, --literal
       Write literal strings only (disable escape characters).

       -o, --stdout
       Write subsequent strings to standard output (file descriptor 1).

       -p, --prefix=<prefix>
       Use <prefix> as escape prefix (default='\').

       -r, --repeat=<repeat>
       Repeat previous value <repeat> times.

       -s, --sleep=<delay>
       Sleep for <delay> amount of time.

       -t, --terminal
       Write subsequent strings directly to terminal.

       -u, --file-descriptor=<fd> Write to specified file descriptor.

       -x, --exit=<num>
       Exit with value <num>.

ESCAPE CHARACTERS
       out recognises C-style escape sequences as  used  by  printf(1) .   By
       default an  escape  sequence  is introduced by the backslash character
       ('\'), however this may be changed with the -p option.  out  also  sup‐
       ports some additional sequences:

       \0     - nul byte (hex value 0x00)

       \a     - alert (bell)

       \b     - backspace

       \c     - no further output

       \e     - escape character (used for changing terminal attributes)

       \f     - form feed

       \g     - generate pseudo-random printable character

       \n     - newline

       \oNNN  - byte with octal value NNN (1 to 3 digits)

       \r     - carriage return

       \t     - horizontal tab

       \uNNNN -  2  byte Unicode (ISO/IEC 10646) character with hex value NNNN
       (4 digits)

       \UNNNNNNNN
       - 4 byte Unicode  (ISO/IEC  10646)  character  with  hex  value
       NNNNNNNN (8 digits)

       \v     - vertical tab

       \xNN   - byte with hexadecimal value NN (1 to 2 digits)

RANGE ESCAPES
       out also supports range escapes which allow a range of characters to be
       specified in a compact format.

       \{N..N}
       - specify a range by two 1-byte literal characters.

       \{oNNN..oNNN}
       - specify a range by two 3-byte octal values.

       \{uNNNN..uNNNN}
       - specify a range by two 2-byte Unicode values.

       \{UNNNNNNNN..UNNNNNNNN}
       - specify a range by two 4-byte Unicode values.

       \{xNN..xNN}
       - specify a range by two 2-byte hex values.

       Note that ranges take two values of the same type and the maximum width
       for that type must be specified.

NOTES
       ·   Arguments are processed in order.

       ·   With the exception of '-x', arguments may be repeated any number of
    times.

       ·   All output will be sent to standard output until  an  output  redi‐
    rection  option is specified that changes the output stream (namely
    -e or -t (or their  long-option  equivalents),  or  if  output  has
    already been redirected -o (or its long-option equivalent)).

       ·   If  <str>  is  the empty string ("" or '') it will be treated as \0
    such that a nul byte will be displayed.

       ·   To cancel the effect of -a, specify a null string: -a ''.

       ·   If <repeat> is '-1', repeat forever.

       ·   Replace the 'Z' in the range formats  above with  the  appropriate
    characters.

       ·   Ranges can be either ascending or descending.

       ·   <delay>  can  take  the  following  forms where <num> is a positive
    integer:

    <num>ns : nano-seconds (1/1,000,000,000 second)
    <num>us : micro-seconds (1/1,000,000 second)
    <num>ms : milli-seconds (1/1,000 second)
    <num>cs : centi-seconds (1/100 second)
    <num>ds : deci-seconds (1/10 second)
    <num>s  : seconds
    <num>m  : minutes
    <num>h  : hours
    <num>h  : days
    <num>   : seconds

    If <num> is -1, wait until any signal is received.
    If signal is SIGNUM continue, else exit immediately.

       ·   Generated printable random characters may not  display  unless  you
    are using an appropriate font.

EXAMPLES
 # Print "foofoofoo" to stderr, followed by "barbar" to stdout.
 out "foo" -r 2 -o "bar" -r 1

 # Write 50 nul bytes direct to the terminal.
 out -t "" -r 49

 # Write continuous stream of nul bytes direct to the terminal,
 # 1 per second.
 out -b 1s -t '' -r -1

 # Display a greeting slowly (as a human might type)
 out -b 20cs "Hello, $USER.\n"

 # Display a "spinner" that loops 4 times.
 out -b 20cs -p % "%r|%r/%r-%r\%r" -r 3

 # Display all digits between zero and nine with a trailing
 # newline.
 out "\{0..9}\n"

 # Display slowly the lower-case letters of the alphabet,
 # backwards without a newline.
 out -b 1ds "\{z..a}"

 # Display upper-case 'ABC' with newline.
 out '\u0041\u0042\u0043\n'

 # Display 'foo' with newline.
 out '\o146\u006f\x6F\n'

 # Clear the screen.
 out '\n' -r $LINES

 # Write hello to stdout, stderr and the terminal.
 out 'hello' -t -r 1 -e -r 1

 # Display upper-case letters of the alphabet using octal
 # notation, plus a newline.
 out "\{\o101..\o132}"

 # Display 'h.e.l.l.o.' followed by a newline.
 out -a . "hello" -a '' "\n"

 # Display upper-case and lower-case letters of the alphabet
 # including the characters in-between, with a trailing newline.
 out "\{A..z}\n"

 # Display lower-case alphabet followed by reversed lower-case alphabet
 # with the digits zero to nine, then nine to zero on the next line.
 out "\{a..z}\{z..a}\n\{0..9}\{9..0}\n"

 # Display lower-case Greek letters of the alphabet.
 out "\{α..ω}"

 # Display cyrillic characters.
 out "\{Ѐ..ӿ}"

 # Display all printable ASCII characters using hex range:
 out "\{\x21..\x7e}"

 # Display all printable ASCII characters using 2-byte UTF-8 range:
 out "\{\u0021..\u007e}"

 # Display all printable ASCII characters using 4-byte UTF-8 range:
 out "\{\U00000021..\U0000007e}"

 # Display all braille characters.
 out "\{\u2800..\u28FF}"

 # Display 'WARNING' in white on red background.
 out '\e[37;41mWARNING\e[0m\n'

 # Generate 10 random characters.
 out '\g' -r 9


AUTHOR
       Written by James Hunt <james.hunt@ubuntu.com>

COPYRIGHT
       Copyright © 2012 James Hunt <james.hunt@ubuntu.com>

LICENSE
       GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html>.
       This  is  free  software:  you  are free to change and redistribute it.
       There is NO WARRANTY, to the extent permitted by law.

SEE ALSO
       echo(1) printf(1)



User Commands     2012-09-10    OUT(1)

Ideas

The TODO file in the distribution contains a number of ideas to enhance outs abilities.

Code

See Also








No comments:

Post a Comment