Overview

This document describes the py_mini_sh Python package which provides a number of utility functions that allows developers to create shell-like scripts in Python. This saves having to learn platform-specific shell syntax (Windows CMD shells and Linux sh or bash). Then, when the going gets tough, you have the complete power of the Python programming language to do the really hard tasks.

One of the unfortunate tendencies in Open Source projects is for the project page to have lengthy instructions for how to install dependencies and to build the project. It should be obvious that if you can write the instructions to describe the required steps, why not just write a script to do this automatically!

The problem usually is because the instructions differ only slightly between Windows and Linux, and there is no scripting language that comes natively on both of these platforms. This package proposes that Python is the natural language in which to write these install, build and configuration scripts, and the section below describes how we can solve the problem of bootstrap into Python as quickly as possible.

License

This software is made available under the MIT License (see LICENSE.txt).

Usage

In the meantime, let’s look at an example py-mini-sh script, which describes the setup and build for a Pyramid-based Python project.

"""
Bootstrap a python build environment for a Pyramid-based project.
"""
from py_mini_sh import exec_, pipe, run, is_f
from py_mini_sh import parse_pylint_output, readall, writeall, CommandError

# Project defines:
PYLINT_OUT = 'my_pyramid_project-pylint'
PYTEST_COVER_ARGS = '--cov=my_pyramid_project --cov-report=html'
PYLINT_OUT_WORSE = '{PYLINT_OUT}-worse'


def bootstrap():
    # Update core stuff in our virtualenv
    exec_('{PYTHON_EXEC} -m pip install -U pip setuptools wheel')

    # Install the project in editable mode with its testing requirements.
    exec_('{VENV_BIN}pip install -e ".[testing]"')

    # build the sphinx doc
    exec_('{VENV_BIN}python -m sphinx -b html -d ../build/doctrees source ../SHIP-{ALLVARIANTS}/docs/html',
          cwd="docs")

    # run tests.
    exec_('{VENV_BIN}pytest {PYTEST_COVER_ARGS}')

    # run pylint, and check that its score is not worse than the last committed one
    pylint_data = pipe('{VENV_BIN}pylint pyramid_weekly_reports', ignore_error=True)
    e, w, c, r = parse_pylint_output(pylint_data)
    if is_f('{PYLINT_OUT}'):
        E, W, C, R = parse_pylint_output(readall('{PYLINT_OUT}'))
        logging.info("""
            Pylint results:
                Was Error: %d, Warning: %d, Convention: %d, Refactor: %d
                Now Error: %d, Warning: %d, Convention: %d, Refactor: %d
        """,
        E, W, C, R, e, w, c, r)
        if e > E or w > W or c > C or r > R:
            writeall_('{PYLINT_OUT_WORSE}', pylint_data)
            raise CommandError("Pylint results worse than before")
        writeall_('{PYLINT_OUT}', pylint_data)
    del_('{PYLINT_OUT_WORSE}')


if __name__ == '__main__':
    sys.exit(run(bootstrap, globals()))

As can be seen in this example, a py-mini-sh scripts is just a Python script that imports the functions from the Python package py_mini_sh, which makes things that little bit easier when writing shell-like scripts.

The main py_mini_sh utility functions shown here are the exec_, pipe, readall, writeall and run, all of which are described in the API documentation. The entry-point function is run and it will execute the supplied script function in the context of the supplied variables. The flow of the script is determined by the Python language itself, and the commands that are executed utilize the Python format function heavily in dealing with “shell variables”.

Bootstrapping Python

We now have a chicken-and-egg situation. We like the fact that we can write platform-independent shell-like scripts in Python, but we’re using a language that is not universally available on all platforms! To solve this we do have to resort to a bit of boiler-plate platform-specific scripts to jump-start our way into the Python world. Thankfully this package has some example scripts (bat file for Windows and bash script for Linux), to help you do this.

The bat file listed below is an example file for Windows that can be used to bootstrap your way into Python. Copy and paste it into your project, naming it with the name of your task. For example, if you want to call your script buildall, call this bat file buildall.bat, copy the Linux boostrap example file (described below) as buildall, and create your Python mini-sh file as buildall.py. This will allow you to run:

buildall

on either platform in exactly the same way.

Virtualenv

This Python package demands that it is used in a virtualenv. This is basically a philosophical opinion on how Python should be used. The example platform-specific scripts create a local virtualenv based on the convention of the virtualenv called pyenv??, where the ?? denotes the version of Python being used, for example pyenv35 or pyenv36. Either hardwire these scripts to use a particular version, or allow the version to be specified on the command line.

Windows

Let’s look at a possible Windows bootstrapping file first which will run the buildall.py example shown above.

@echo off
REM This build script checks that the argument "34" or "36" is given, which
REM indicates whether the software should be built with Python 3.4 or with
REM 3.6.
REM
REM A virtualenv is created in this folder if one doesn't exist already.
REM
REM It installs the py_mini_sh package into this virtualenv.
REM
REM It then runs the buildall.py file using the this virtualenv.

set VENV=
set PY_LOC=
set PYVER=
set PY_BITS=

IF "%1"=="" (
    GOTO :BADARG
) ELSE IF "%1"=="34" (
    set PYVER=3.4
) ELSE IF "%1"=="36" (
    set PYVER=3.6
) ELSE (
    GOTO :BADARG
)
set VENV=pyenv%1

REM Let's see if Python 3.4 or 3.6 has been installed elsewhere before.
for /F "usebackq tokens=*" %%P in (`c:\Windows\py.exe -%PYVER% -c "import sys,os; print(os.path.dirname(sys.executable))"`) do set PY_LOC=%%P

IF "%PY_LOC%"=="" (
    echo Failed to find python version %PYVER%. Install this first.
    GOTO :FAILED
)

echo Using the Python in %PY_LOC%

REM Build the virtualenv if it doesn't exist already.
IF exist %VENV% (
    echo Virtualenv %VENV% exists.
) ELSE (
    "%PY_LOC%\python" -m venv %VENV%
    REM Install the py_mini_sh package
    %VENV%\Scripts\python -m pip install -U py_mini_sh
)

REM Delegate everything else to the buildall.py script.
%VENV%\Scripts\python.exe buildall.py %*

REM Activate this shell with the created virtualenv
call %VENV%\Scripts\activate.bat

GOTO DONE

:BADARG
ECHO Specify either 34 or 36 as an argument
:FAILED
ECHO Failed to build
set ERRORLEVEL=1
:DONE
EXIT /B %ERRORLEVEL%

This is a bit of an eyeful, principally because the bat format is not pretty. The REM comments at the top of the file describe what this script is trying to achieve. The most important thing that it does is create a Python virtualenv in the current folder with the py_mini_sh package installed into it, and then it runs the required Python script that can use this Python package.

Note that due to line 51 the virtualenv is activated in the calling shell, which is for the convenience of the user.

Linux

With a Linux bootsrapping script, things are a little easier. An equivalent Linux script to the Windows one above is shown below using bash, which is pretty much universally available on all Linux machines.

#!/bin/bash
# Bootstrap into Python to run the buildall.py script.
# Usage: buildall 27|34|35|36

set -e

if [ "$#" -ne 1 ]; then
  echo "Usage: $0 27|34|35|36" >&2
  exit 1
fi

export P=$1
export PY_VER="${P:0:1}.${P:1:1}"
export VENV=pyenv${P}

# Check that this version of Python exist
if python${PY_VER} --version >&2 /dev/nul; then
    echo "Using Python version $P"
else
    echo "Cannot find Python version $P"
    exit 1
fi

# Create the virtualenv based on the requested Python
export PY_LOC="/usr/bin/python${PY_VER}"
if [ -d ${VENV} ]; then
    echo "Using virtualenv ${VENV}"
else
    echo "Building virtualenv ${VENV}"
    if virtualenv -p ${PY_LOC} ${VENV}; then
        echo "Done."
        # Install the py_mini_sh package
        ${VENV}/bin/pip install -U py_mini_sh
    else
        exit 1
    fi
fi

# Now run the buildall.py file with the virtualenv's python
${VENV}/bin/python buildall.py