A Better Way to a Jenkins Service

The typical way to install a Jenkins service on a Windows system is to download the JNLP file and then execute the file with Java Web StartTM - Oracle: be quite afraid. This process manufactures an excess of anxiety and annoyance for the user due to dialog boxes and their controls appearing only upon successful completion of mysterious (magical) events. Upon failure all you can do is puzzle about how ye flask may not be got.

Thanks to me ye need not wonder, for I will show thou how to get ye flask. Executing javaws on the Jenkins JNLP file triggers several actions:

  • agent.jar is downloaded
  • WinSW.exe is downloaded
  • If the JavaWS window appears, the File menu appears within that window, and you click on Install as Windows Service in that menu, WinSW.exe is renamed to the name of the Jenkins service, which is then installed as a proper Windows service
  • The new service is started using the WinSW.exe as the service executable taking its config from a file generated by Jenkins (agent.jar I'm guessing)
  • The javaws process is terminated

At first you might reason as I did that since the actual Java command line is configured in the agent config file, that you can just run it directly in a PowerShell script and install that script as a service. However, Windows does not allow services to be defined by script files, only by platform-native executable files. One solution to this problem is represented by the popular NSSM. I find Jenkins's own Windows service wrapper to be much more elegant as it does not use an intermediary process. WinSW merely takes on the name of the service you want to run by executing the command line provided in its config file.

Here is the correct way to automate that whole process (guaranteed to get thou ye flask without any effort):

{%- set agent = {
    'service_name': 'JenkinsAgent',
    'service_long_name': 'Jenkins Agent (powered by WinSW)',
    'service_desc': 'This service runs an agent for Jenkins',
    'service_home': 'C:\\Jenkins',
    'winsw_uri': 'https://github.com/kohsuke/winsw/releases/download/winsw-v2.2.0/WinSW.NET4.exe',
    'winsw_sha512': 'f485dec639155528d804bbeddc69a0d9fe77c44444821a5b6cf557ec0b8f9932153a0411bab0524a47f133d2ee5bf3c03830f32b36c77b83cc7f0b1c5a108b33',
    'jar_sha512': '6lojuezjazurpmarwmzymthvo0aukur9fvgbiro8gdkkbv7t1ps2nhupdixjcqt1uhzqdzjswojkahulfqbeo5fepuc8ordezt5siugmelxncjwnmggmsvepjufomotk',
} %}


# Install {{ agent['service_name'] }}.jar
service-downloaded:
  file.managed:
    - name: {{ agent['service_home'] }}\\{{ agent['service_name'] }}.jar
    - source: https://jenkins.mycorp.com/jnlpJars/agent.jar
    - source_hash: {{ agent['jar_sha512'] }}
    - makedirs: True

# Setup {{ agent['service_name'] }}.xml
service-configured:
  file.managed:
    - name: {{ agent['service_home'] }}\\{{ agent['service_name'] }}.xml
    - contents: |
        <configuration>
          <id>{{ agent['service_name'] }}</id>
          <name>{{ agent['service_long_name'] }}</name>
          <description>{{ agent['service_desc'] }}</description>

          <executable>java.exe</executable>  <!-- Should be AdoptOpenJDK version -->
          <arguments>-Xrs -jar "{{ agent['service_home'] }}\\{{ agent['service_name'] }}.jar" -jnlpUrl https://jenkins.mycorp.com/computer/{{ grains['id'] }}/slave-agent.jnlp -secret {{ pillar[grains['id']]['jnlp_secret_key'] }}</arguments>

          <logmode>rotate</logmode>
          <onfailure action="restart" />
        </configuration>

# Download, install, and start {{ agent['service_name'] }}.exe
service-created:
  file.managed:
    - name: {{ agent['service_home'] }}\\{{ agent['service_name'] }}.exe
    - source: {{ agent['winsw_uri'] }}
    - source_hash: {{ agent['winsw_sha512'] }}
  module.run:
    - service.create:
      - name: {{ agent['service_name'] }}
      - bin_path: {{ agent['service_home'] }}\\{{ agent['service_name'] }}.exe
      - start_type: auto
      - account_name: '.\{{ pillar['robo_account']['name'] }}'
      - account_password: '{{ pillar['robo_account']['pass'] }}'
    - unless:  # Any changes to the data in this state will require a manual service reinstall because the requisite only checks for existence
      - 'if (Get-Service {{ agent['service_name'] }} -ErrorAction SilentlyContinue)
         {
             Write-Error "Do not execute because service {{ agent['service_name'] }} already exists"
         }'
  service.running:
    - name: {{ agent['service_name'] }}
    - enable: True

Using jq with SaltStack Output

Many of the windows-oriented or windows-themed exec and state modules in SaltStack return large, indivisible units of data back to the command line. I know the "real solution" here is to plug salt into an advanced/complex returner mechanism with it's own DB and UI, but not everybody is on the outscaling growth trajectory and destined for legions of minions. For the rest of us small-time ops/admins, there's a nice, obscure tool called jq that will filter the salt data for you right inline: a real unix paradigm conformer. Suppose you want to know only about which of your windows minions are set to auto update and watch for updates for other Microsoft products:

salt -G 'kernel:Windows' win_wua.get_wu_settings --out=json | jq -r 'keys[] as $k | "\($k):\n  level:     \(.[$k] | ."Notification Level")\n  MS update: \(.[$k] | . "Microsoft Update")"'
jerry:
  level:     1
  MS update: false
kevin:
  level:     4
  MS update: true
stuart:
  level:     4
  MS update: false
dave:
  level:     2
  MS update: false
carl:
  level:     3
  MS update: true

Also characteristic of traditional command line utilities, jq's syntax is not amenable to quick comprehension or mastery. Indeed here are the resources I used to construct the above query:

  • jq Manual: Well-appearing but mostly designed for someone with much more open and/or flexible time than I. Perhaps you would read this if you were a college student with no friends or family during winter break.
  • Selecting object values: found via jq select -jquery.

Obliquely topical obligatory XKCD (Bonus: 4!):

PowerShell Discard Empty Lines

Lately I have found myself needing to normalize the whitespace in a line-delimited list of strings and have found the following construct very useful for this need:

PS C:\Users\Dude\Code\MyRepo> git log -n 19 --format='%B' `  # Let's trim the empty lines from this git output
>> | % { $_.Trim() } `                                       # Trim any pre/post-whitespace
>> | ? { $_ }                                                # Discard empty lines

Visual Studio Power Environment: Jenkins Pipeline Edition

Rather than bother with custom groovy to hack the Jenkins declarative pipeline environment, let's use a structural paradigm that has been successful for me because of its simplicity (the only cost being a reusable custom groovy function) and readability (the function wrapper demonstrates what it does in salutary clarity).

int withVSEnvironment(String label, Boolean returnStatus, String script) {
    ret = powershell(
        label: label,
        returnStatus: returnStatus,
        script: """
            foreach (\$_ in cmd /c "`"%VS140COMNTOOLS%`"..\\..\\vc\\bin\\vcvars32.bat > nul 2>&1 & SET") {
                if (\$_ -match '^([^=]+)=(.*)') {
                    [System.Environment]::SetEnvironmentVariable(\$matches[1], \$matches[2])
                }
            }

            $script
        """
    )
    if (returnStatus) {
        return ret
    }
}


pipeline {
    agent any
    stages {
        stage('Stage Left') {
            steps {
                withVSEnvironment(
                    'Building with PowerShell!!!',
                    False,  // returnStatus
                    '''
                    # No weak cmd script here
                    MSBuild.exe Source/Answer/The_Solution.sln /m /t:Clean,Rebuild
                    '''
                )
            }
        }
    }
}

Visual Studio Power Environment

It is sometimes interesting to speculate about how unusual feature patterns in Microsoft projects may reflect the traditions or prejudices of the teams involved with creating and evolving those projects. Today's example is "%VS140COMNTOOLS%"..\..\vc\bin\vcvars32.bat, that wily little batch script that mucks your command environment enough to enable most Visual Studio declarative-like thingies to happen (easily) on the command prompt.

So PowerShell has been around for like 17 years now (if you include the Monad proto period). That's enough time for an effective rewrite or two of the Visual Studio codebase and yet we're still stuck with this inconvenient script written in possibly the worst shell still in popular use today. How can we get it to work with possibly the best shell in popular use today? Unfortunately there is no single command that will dump the environment of a cmd subprocess into the parent powershell environment. I found several hacks online proposing to do this, the most correct and elegant being Invoke-Environment.

Note

[Digression] reStructureText's link formatting is egregiously inferior.

What I really wanted was some Groovy code that will dynamically modify the environment of a declarative pipeline with the results of vcvars32.bat. Jenkins pipeline allows for setting environment variables dynamically, but the dynamism packaged into this feature only allows for arbitrary values for a set of a priori variable names. The vcvars32.bat script sets an indeterminate set of variable names. There may still be a way to do this, but my Groovy-fu does not show me how to edit the env object at sufficient introspection.

Here is the essential functionality of Invoke-Environment applied to vcvars32.bat. This is what I used to prime the PowerShell environment for Visual Studio work by pasting the code block directly into the powershell() step before executing Visual Studio commands. This is not the most elegant, but it is only a few more lines than directly sourcing the batch script and seems to be the plainest way to absorb its environment.

foreach ($_ in cmd /c "`"%VS140COMNTOOLS%`"..\\..\\vc\\bin\\vcvars32.bat > nul 2>&1 & SET") {
    if ($_ -match '^([^=]+)=(.*)') {
        [System.Environment]::SetEnvironmentVariable($matches[1], $matches[2])
    }
}

Note

If you are using this in a Jenkins pipeline, all $'s will need to be escaped, \$, to signify that they signal PowerShell variables and not Groovy variables. Also ` is PowerShell's escape character, so `" is an escaped quote.

How To Mount LUKS

Warning

This code is meant to be a trenchant example of the essential actions necessary. You are crazy and I am not responsible if you copy and paste.

  • Mount LUKS:
    apt install cryptsetup
    LUKS_BLKDEV=$(blkid | grep crypto_LUKS | cut -d ':' -f 1)
    cryptsetup luksOpen ${LUKS_BLKDEV} root
    # Enter passphrase
    mkdir /mnt/root && mount /dev/mapper/root /mnt/luks_root  # Skip this step if `/dev/mapper/root` is an LVM partition.
    
  • Mount LVM:

    Note

    Ubuntu seems to always use LVM inside of LUKS, so this is how to setup the LVM layer once LUKS has been decrypted.

    apt install lvm2
    VOL_GRP=$(vgscan | tail -n 1 | cut -d '"' -f 2)
    vgchange -ay ${VOL_GRP}
    mkdir /mnt/root && mount /dev/${VOL_GRP}/root /mnt/root
    

Changing the Package Your BlueJeans Came in

Recently my company decided to move from WebEx to BlueJeans for conferencing. While they do provide linux support, such support is limited to RPM-based systems.

I can sympathize with this because although RPM spec is an antique format with plenty of arcane customs and traditions, if you start the day with a goal to create an RPM and your app is not too complex, you will have an RPM at the end of the day if not sooner. Replace RPM with deb and you will likely find yourself lost in a world you never knew existed. At the end of the day if you have a deb it means you're already a debian monk or are ready to become such.

The BlueJeans app itself 'seems' to be just another electron app or similar. Thus, adapting it to a deb-based system turned out to be trivial with a little alien magic. Here's the salt state I used:

{%- set bj_full_ver = '1.37.22' %}
{%- set bj_part_ver = bj_full_ver.rsplit('.', 1)[0] %}
{%- set bj_rpm = 'bluejeans-{}.x86_64.rpm'.format(bj_full_ver) %}

bluejeans-deps:
  pkg.installed:
    - name: BlueJeans deps
    - pkgs:
      - libgconf-2-4
      - libudev-dev
  file.symlink:
    - name: /lib/x86_64-linux-gnu/libudev.so.0
    - target: /lib/x86_64-linux-gnu/libudev.so
    - require:
      - pkg: bluejeans-deps

bluejeans-get:
  cmd.run:
    - name: wget https://swdl.bluejeans.com/desktop/linux/{{ bj_part_ver }}/{{ bj_full_ver }}/{{ bj_rpm }} --output-document /tmp/{{ bj_rpm }}
    - unless:
      - dpkg -l bluejeans
bluejeans-convert:
  cmd.run:
    - name: alien --keep-version --scripts /tmp/{{ bj_rpm }}
    - cwd: /tmp
    - onchanges:
      - cmd: bluejeans-get
bluejeans-install:
  cmd.run:
    - name: dpkg --install /tmp/bluejeans_{{ bj_full_ver }}-1_amd64.deb
    - onchanges:
      - cmd: bluejeans-get
    - require:
      - cmd: bluejeans-convert
bluejeans-clean:
  cmd.run:
    - name: rm --force /tmp/bluejeans-*
    - onchanges:
      - cmd: bluejeans-get
    - require:
      - cmd: bluejeans-install

While this gets the job done, it's not very elegant, and in these modern times one expects packages to either bundle into the base package repo that comes with the distro or come out of a gpg-signed app repo. A bare package URL is kind of amateur especially considering how easy it is to setup a repo (createrepo ftw). Also a debian repo is not too hard to setup once you invest the time to learn. But I'm willing to admit to being a DevOps engineer with smug opinions about how the world should function.

Python ArgParse Columns

Have you ever been bothered by cramped help text formatted by Python's argparse module, especially for its affinity for enthusiastic left indents? It has that distinctive format universally discernible for features good and unfortunate. Today we are going to explore and annihilate one of these features of misfortune. Like pip, pytest, and despite what the authors may claim in effect, for it's abstruse Java-celebrating recursions into metastatic blandness: also the logging module--they all depress onto the user some burden of monolithic inflexibility in a way that feels as if the user shall be prevented from getting the job done by the nonblocking opinion of aloof, indifferent code. In the best case, the job is done with extra provisions for these superior opinions that provide comedic commentary written in code on such opinions. In the case of argparse, the formatting customization is limited to 3 simplistic choices. Subclassing is not an option because the way these classes affect argparse output behavior is hidden in source code (obfuscated from sane external minds as that unwholsome logic called implementation detail). However, though learning how awfully the code you routinely employ is hampered by irritating internal rigidity, this particular black box needn't inevitably stumble anyone since the columns conundrum itself yields readily to a few lines of boilerplate.

Problem

Observe:

#/usr/bin/env python3
# my_script.py
import argparse


def _get_opts():
    """
    Setup options for my_script
    """
    ap = argparse.ArgumentParser(description='My script does something, I guess', formatter_class=argparse.ArgumentDefaultsHelpFormatter)
    ap.add_argument('-a', '--an-argument-with-a-really-long-help-msg', help='The intent of this help string is to demonstrate how unreadable a hard line wrap at a narrow column span looks when there is an expansive, capacious field of space to the right of this text available in the terminal.  The environment variable os.environ[\'COLUMNS\'] does not inherit the parent process\'s (a bash shell in this case) $COLUMNS value.')
    return vars(ap.parse_args())


def my_action():
    """
    Run the action of my script
    """
    opts = _get_opts()
    raise NotImplementedError('Really, I don\'t plan to do anything at all here')

if __name__ == '__main__':
    my_action()
$ python3 my_script.py --help
usage: my_script.py [-h] [-a AN_ARGUMENT_WITH_A_REALLY_LONG_HELP_MSG]

My script does something, I guess

optional arguments:
  -h, --help            show this help message and exit
  -a AN_ARGUMENT_WITH_A_REALLY_LONG_HELP_MSG, --an-argument-with-a-really-long-help-msg AN_ARGUMENT_WITH_A_REALLY_LONG_HELP_MSG
                        The intent of this help string is to demonstrate how
                        unreadable a hard line wrap at a narrow column span
                        looks when there is an expansive, capacious field of
                        space to the right of this text available in the
                        terminal. The environment variable
                        os.environ['COLUMNS'] does not inherit the parent
                        process's (a bash shell in this case) $COLUMNS value.
                        (default: None)

Solution

Since 3.3, the shutil module provides a way to insert the terminal's geometry into your script in a way that argparse comprehends:

import shutil
import argparse

# Necessary boilerplate to imbue the argparse help output with essential readability
os.environ['COLUMNS'] = str(shutil.get_terminal_size().columns)
$ python3 my_script.py --help
usage: my_script.py [-h] [-a AN_ARGUMENT_WITH_A_REALLY_LONG_HELP_MSG]

My script does something, I guess

optional arguments:
  -h, --help            show this help message and exit
  -a AN_ARGUMENT_WITH_A_REALLY_LONG_HELP_MSG, --an-argument-with-a-really-long-help-msg AN_ARGUMENT_WITH_A_REALLY_LONG_HELP_MSG
                        The intent of this help string is to demonstrate how unreadable a hard line wrap at a narrow column span looks when there is an expansive, capacious field of space to the right of this text available in the terminal. The environment variable os.environ['COLUMNS'] does not inherit the parent
                        process's (a bash shell in this case) $COLUMNS value. (default: None)

Shell Function to Execute a Command and Return the Output

'Shell', whatever that means is one or other species of a collection of ancient language families complicated by an indefinite permutation of complementary, supplementary, and conflictiary forward, backward, and sideward compatibility relations. Lately shell usually means bash, although I know there will be disputes even here. The primary argument for this convention is that all major distros are setup with bash as the default shell. Being an ancient language, its quirks and compatibility knots are more intrusive than what one would desire coming from experience with modern languages, so any idiom that is well completed, clear, and expressive feels like an achievement worth recording.

Here is a function that I created but, after code review, discovered I did not need, but also want to remember. It executes its first argument and 'returns' its output into its second argument. Whether this is actually 'POSIX compliant' I do not know, but that it works on GNU Bash 4.1+ seems to be careful enough to certify my uncertainty constraints.

function EXECUTE()
{
    # $1: Command to execute
    # $2: Capture the output

    if [ -n "${MY_SCRIPT_TEST}" ] ; then
        LOG "${FMT_BLUE}" '' "Would have executed: ${1}"
    else
        LOG "${FMT_MAGENTA}" '' "Executing: ${1}"
        # Set arg 2 to the output resulting from the execution of arg 1
        eval "$2='$(eval ${1})'"
    fi
}

Here's the function in the context of some other useful functions that altogether does something that could be useful and well-defined.

function USAGE()
{
    printf -- '%s\n' "This is my script that performs some actions"
    printf -- '%s\n' "set MY_SCRIPT_COLORS=1 to get colorized output and"
    printf -- '%s\n' "set MY_SCRIPT_TEST=1 to pretend"
}


function SETUP_LOG()
{
    local LOG_DIR=${1}

    VALIDATE_DIR "${LOG_DIR}" 'logging directory'

    # Global variables ########################################################
    PRINTF_HAS_FMT_B=''
    ###########################################################################
    printf '%b%s' 'str' &> /dev/null
    [ "$?" = "0" ] && PRINTF_HAS_FMT_B='yes'
}


function LOG()
{
    local MESSAGE_COLOR=$1
    local PROCESS_ESCAPES=$2
    local MESSAGE=$3
    local TIME_STAMP=$(date +'%Y-%m-%dT%H:%M:%S')

    # Process escape sequences in message if requested and available
    if [ -n "${PROCESS_ESCAPES}" ] ; then
        if [ -n "${PRINTF_HAS_FMT_B}" ] ; then
            MESSAGE=$(printf '%b%s' "${MESSAGE}")
        fi
    fi

    # Log to stdout
    if [ -n "${MY_SCRIPT_COLORS}" ] ; then
        printf -- '\e[00;%sm%s\e[0m\n' "${MESSAGE_COLOR}" "${MESSAGE}"
    else
        printf -- '%s\n' "${MESSAGE}"
    fi

    # Log to log file
    if [ -f "${LOG_FILE}" ] ; then
        printf -- '[%s] %s\n' "${TIME_STAMP}" "${MESSAGE}" >> "${LOG_FILE}"
    fi
}


function VALIDATE_DIR()
{
    local DIR=$1
    local DESC=$2

    if [ -z "${DIR}" ] ; then
        LOG "${FMT_RED}" '' "Please provide the ${DESC}"
        exit 1
    fi

    mkdir -p "${DIR}"
    if [ ! -d "${DIR}" ] ; then
        LOG "${FMT_RED}" '' "Cannot ensure directory ${DIR}"
        exit 1
    fi
}


function VALIDATE_ARGS()
{
    # Global variables ########################################################
    FMT_BLACK=30
    FMT_RED=31
    FMT_GREEN=32
    FMT_YELLOW=33
    FMT_BLUE=34
    FMT_MAGENTA=35
    FMT_CYAN=36
    FMT_WHITE=37
    FMT_EXTENDED=38
    FMT_DEFAULT=39

    LOG_DIR='/tmp/my_script'
    LOG_FILE="${LOG_DIR}/my_script.log"
    ###########################################################################

    # Display help message
    local HELP=''
    case "${1}" in
        '-h') HELP='yes' ;;
        '--help') HELP='yes' ;;
        'help') HELP='yes' ;;
        *) ;;
    esac
    if [ -n "${HELP}" ] ; then
        USAGE
        exit 0
    fi

    SETUP_LOG "${LOG_DIR}"
    LOG "${FMT_CYAN}" '' "Executing command line: ${0} ${*}"
}


function EXECUTE()
{
    # $1: Command to execute
    # $2: Capture the output

    if [ -n "${MY_SCRIPT_TEST}" ] ; then
        LOG "${FMT_BLUE}" '' "Would have executed: ${1}"
    else
        LOG "${FMT_MAGENTA}" '' "Executing: ${1}"
        # Set arg 2 to the output resulting from the execution of arg 1
        eval "$2='$(eval ${1})'"
    fi
}


function PERFORM_SOME_ACTIONS()
{
    local TMP_FILE_COUNT=''
    EXECUTE 'find /tmp -type f | wc -l' TMP_FILE_COUNT
    LOG "Temporary file count is ${TMP_FILE_COUNT}"
}


VALIDATE_ARGS $@
PERFORM_SOME_ACTIONS

Whose Proxy?

There are many diagrams and word salads presented in cause of explaining forward and reverse (backward? I like linguistic symmetries) proxies. Munificent logic and inholistic reason thus spewed confounded me in my simple desire to know. Fortunately for you, in a dream I had last night I explained it to a dream person thus, and when I had spoken it I knew it was the reductive idea purged of all contrarian confounding encumberment; clairvoyance having carried it forward until now. There is no need to fret about the client or server not know about the presence of the proxy in either case.

In a forward proxy, the server communicates with the proxy as a client. In a reverse proxy, the client communicates with the proxy as a server.