History

(From Wikipedia and TLDP.org)

1971

Thompson shell

• IO redirection compact syntax
• pipes added later

1975 to 1977

… clear that the Thompson shell was inadequate for most serious programming tasks.

PWB shell, aka Mashey shell.

• internal if-then-else
• switch
• while
• $v - one letter variables and $ sigill

circa 1977

Bourne shell is distributed, replacement for Mashey shell.

• complete rewrite
• environment variables
• eliminated goto
• command substitution

Later

• built in test
• # comment
• Functions & return
• Job control

1989

bash beta v0.99 released

• superset of the Bourne shell command syntax
• completion
• command history
• $(…) command substitution
• $((math)) calculations
• <(process substitution)

1996

bash 2 adds array variables

2004

bash 3.0 adds built in regular expressions

if [[ "$variable" =~ "your-regex" ]]

2009

bash 4.0

• associative arrays
• coproc
• optional ** globbing
• command_not_found_handle

Status for 2015

(Personal opinion)

• bash rules
• zsh is used sometimes
• rest is crap

Why shell scripting?

• DSL
• power
• simplicity (*)

Shell

echo '…' >/etc/apt/sources.list.d/no-product-ads-here.list

Want to try that in Java? (from Stack Overflow, edited)

FileUtils.writeStringToFile(
new File("/etc/apt/sources.list.d/no-product-ads-here.list"),
"…");

Why shell scripting?

Nginx example - popular alternatives

Nginx example - alternative 1

Nginx Chef cookbook (as of 2015-07, commit e36944b)

• 55M downloads, CentOS, Fedora, Debian, Ubuntu, RedHat, Amazon, …
• Lines: readme - 521, recipies - 1577, attributes - 592, templates - 953

Nginx example - alternative 2

Puppet Nginx module (as of 2015-07, commit d1b0908)

• 230K downloads, Debian, RedHat, Ubuntu
• Lines: readme - 286, Puppet code - 2517, templates - 892

Nginx example - bonus slide

Think disregarding the context of this presentation

- Let's try Nginx but I think we will switch to Apache later.

- I know! We need abstraction cookbook!

Is this the right solution?

Nginx example - bonus slide

Nginx example - the sane way

Scripted. Idempotent code.

• Lines: docs - 72, code - 56, templates - 329
• Templates contain project-specific configuration
• Code includes functionality not in alternatives above:
  • DH file generation (8 lines)
  • Handle socket leak on reload (22 lines)

Nginx example - summary

The overlap of provided (Chef and Puppet) and required functionality is roughly zero.

Nginx example - how are you screwed

… when using Chef or Puppet … because life

if lsof -n | grep -q 'nginx.* 300u';then
  …
  service nginx restart
  …
fi

if ! pgrep -f 'nginx: master' >/dev/null;then
  …
  service nginx start
  …
fi

OK, scripting. Why not Ruby or Python or …?

What is a shell language?

My definition

Shell language

A language that is used by a shell.

Why not language X?

Python example

Ever wished the compactness of shell scripts be put into a real programming language? -- plumbum project

from plumbum import local
ls = local["ls"]

ls()

vs

ls

Why not language X?

Python example

ls["-a"] | grep["-v", "\\.py"] | wc["-l"]

vs

ls -a | grep -v '\.py' | wc -l

Why not language X?

Non-shell languages are not fit for the job

• Shell script should be able to contain copy+paste parts from the CLI
• I'm not ready to type much more in CLI, I need to get the job done

What's wrong with current shell languages?

• DSL for changed domain
• Missing data structures
• Syntax (because of compatibility issues)

So, which language?

A real programming language of course

Should be DSL

Vision: 21st century UI and UX

… kind of

The shell - stuck UI (wrong)

Except for editors, how many interactive programs do you really use? We have:

# dd if=/dev/hda … | ssh …
_

… and you're blocked.

The shell - non blocking UI (right)

Should be:

10:23 [56%, 20MB/s, ETA 20m] dd if=/dev/hda … | ssh …
# _

The shell - progress indication

dd if=/dev/hda …

Possible heuristic: Has only one open file (except for 0,1,2) to read from, position in /proc/PID/fdinfo/FD

… and add API for progress and status already!

The shell - completion

Wrong way: simple instead of right

# apt-cache search ^bash completion
bash - …
bash-completion - …
python-argcomplete - …
python3-argcomplete - …
# apt-get install <TAB><TAB>
Display all 54316 possibilities? (y or n)

Also: why not navigate and pick?

The shell - pipe (pre)view

• Preview when composing
• Snoop on running jobs

The shell - history

Context / state. Currently only timestamp.

• Directory
• Values of used variables
• Globbing results
• Exit code
• Output
• more …

The shell - UI mock

Vision: syntax

syntax

No

ping_output=`$MYADMIN ping 2>&1`; ping_alive=$(( ! $? ))
…
if [ "$1" = "check_alive"  -a  $ping_alive = 1 ] ||
…
echo -e "… resulted in\n$ping_output\n"

Yes

ping=$($MYADMIN ping 2>&1)
…
if mode == 'check_alive' and ping or
…
echo("… resulted in \n${ping}\n")

syntax

syntaxes actually

defg get_all_instances(filters) {
	if 'VPCID' in ENV {
		filters.push(['vpc-id', ENV['VPCID']])
	}
	f = mkfilters(filters)
	instances = ``aws ec2 describe-instances $*f``
		.Reservations.Instances.flatten()
}

(bad naming stolen from boto)

Vision: Cloud

• Auto-groups, auto-split
• Syntax, not escaping
• Stop SSH-ate-my-input madness

Vision: DSL

DSL

if $(-f /etc/debian_version) → if $(test -f /etc/debian_version)

Code from stdlib

defg Bool(p:Process) { p.status == 'exited' and p.code == 0 }
…
defg '$()'(*args) {
	guard args.len() > 1
	guard startsWith(args[0], '-')
	args = ['test'] + args
	# todo: better syntax for this:
	($())(*args)
}

optimized for common cases

.Reservations.Instances (code from stdlib)

defg '.'(a:Array, attr:String) { map(a, (.), attr) }
						

syntactic sugar for common cases - @

@ wraps the code on the right with lambda

val @  expr  →  val.map(F(X=null, Y=null, Z=null) { expr })

val @? expr  →  val.filter(F(X=null, Y=null, Z=null) { expr })

    @  expr  →  F(X=null, Y=null, Z=null) { expr }

syntactic sugar for common cases - @

Example: val @ expr

echo(5 @ X*2) → echo(5.map(F(x) {x*2}))

outputs

[0, 2, 4, 6, 8]

syntactic sugar for common cases - @

Example: @ expr

Code from stdlib

defg flatten(s:Seq) {
	ret = []
	s.each(@X.each(@ret.push(X)))
	ret
}

syntactic sugar for common cases - returns

if x==1 { return y }

is

x==1 returns y

Vision: consistency

filling the type/method matrix

code from stdlib + demo

defg map(n:Number, mapper:F, *args) {
	ret = []
	for(i=0;i<n;i=i+1) {
		ret.push(mapper(i, *args))
	}
	ret
}

echo(10.map(F(x) {x*2}))

filling the type/method matrix

code from stdlib

defg Number(b:Bool) { if b {1}{0} }

consistent syntax

Example 1

my_func(*args)

arr = [1, 2, *args, 3, 4]

Example 2

my_func(**kwargs)

opts = {'my': 'default', **options_i_got, 'my2': 'override'}