Category: Bash

  • Bash Snippet: Trim Function

    Bash Snippet: Trim Function

    Occasionally I find myself wanting to removing leading and/or trailing occurrences of a character or string from a larger string in bash. A couple common uses are removing quotes, or stripping an unknown number of spaces leading into the data of interest. This can be done pretty easily with other methods such as awk or sed, but with the disadvantage of requiring external commands. This a pure bash implementation using only shell built-ins. It might be better to rewrite this with $BASH_REMATCH, but this is what I currently have.

    
    
    #!/bin/bash

    trim () {
        str="$1"
        match="$2"
        # trim spaces by default
        if [ -z "$match" ]; then
            match=" "
        fi
        # trim leading
        while [ "${str:0:${#match}}" == "$match" ];
        do
            str="${str:${#match}:${#str}}"
        done
        # trim tailing
        while [ "${str:$((${#str}-${#match}))}" == "$match" ];
        do
            str="${str:0:$((${#str} - ${#match}))}"
        done
        echo "$str"
    }

    #Remove leading and trailing spaces
    example="    Hello     "
    echo "`trim  "$example" " "`"

    #The second parameter is optional when removing spaces
    example="    Hello     "
    echo "`trim  "$example"`"


    #Remove leading and trailing occurences of "word"
    example="wordHelloword"
    echo "`trim  "$example" "word"`"
  • Zombie Process: Killing the Undead

    Is your Ubuntu MOTD warning you of a zombie process?

    Welcome to Ubuntu 11.10 (GNU/Linux 3.0.0-20-server x86_64)

    * Documentation: https://help.ubuntu.com/11.10/serverguide/C

    System information as of Thu Jun 28 18:36:57 EDT 2012

    System load: 0.0 Processes: 94
    Usage of /: 68.2% of 1.79TB Users logged in: 1
    Memory usage: 29% IP address for eth0: 10.0.0.10
    Swap usage: 8%

    => There is 1 zombie process.

    What’s the scoop with that last line “There is 1 zombie process“, is my operating system getting caught up in this current climate of zombie infatuation? Well no, sadly it’s more boring than that. A zombie process occurs when a child process ends, but the parent doesn’t “reap” it. For a much better run down on what a zombie process is check out the Wikipedia article: Zombie process.

    Here is a quick run down on some terminology. A process is just a fancy name for a running instance of a program. A child process (or just “child”) is a process started by another process. A process that starts another process is the “parent process” of the process it starts.

    The ‘ps’ command shows processes we are running.

    user@host:~$ ps
    PID TTY TIME CMD
    5828 pts/4 00:00:00 bash
    6122 pts/4 00:00:00 ps

    The ‘pstree’ command can show a family tree (of sorts) for processes, parents, their children, the children of their children, etc. Our shell is bash, and as we can see in the output from ‘ps’ above, the process ID number (pid) of our bash prompt is 5828.

    user@host:~$ pstree -Gpl 5828
    bash(5828)───pstree(6123)

    Here we can see that bash is the parent process of the pstree command itself when we run it from the bash prompt. The pstree command exits shortly after it displays this information, and bash will go back to being childless. If we run another instance of bash from inside of the current bash prompt, the new bash instance will be a child of the first.

    user@host:~$ bash
    user@host:~$ pstree -Gpl 5828
    bash(5828)───bash(6124)───pstree(6389)

    So you can see our original bash process with the pid number 5828 has begotten our new child bash process of 6124. The new bash process is where we are running the ‘pstree’ command from, so pstree is a child of 6124.

    For an interesting look at your systems family tree, try running ‘pstree -Gpl 1‘.

    Hopefully you have a good handle on the whole parent/child thing. Now we’ll go zombie hunting. The system has told us that there is a zombie, but we know nothing about it. The ps command has options that will print the status of a process in a column of its output.

    root@host:~# ps aux |grep Z
    USER       PID %CPU %MEM    VSZ   RSS TTY      STAT START   TIME COMMAND
    root      2925  0.0  0.0   9256   880 pts/2    S+   18:40   0:00 grep Z
    root     28766  0.0  0.0      0     0     ?    Z    Jun06   0:00 [apt] <defunct

    Here we’ve used the ‘grep’ command to search for a pattern “Z”. Because there is a “Z” in the “VSZ” column header we can also see the ‘ps’ header we were talking about. Over in the “STAT” column we can see that something called “[apt]” has the mark of the zombie (Z).

    At this point you might be thinking about using the ‘kill’ command to kill this zombie dead. The problem with killing a zombie is that by definition they are already dead. Unlike motion pictures, the way to kill a Linux zombie isn’t by shooting it in its head, but by killing its parent (maybe we should call them vampires instead?).

    Kill and kill -9 are futile at killing zombies.

    root@host:~# kill 28766
    root@host:~# ps aux |grep Z
    USER       PID %CPU %MEM    VSZ   RSS TTY      STAT START   TIME COMMAND
    root      2925  0.0  0.0   9256   880 pts/2    S+   18:40   0:00 grep Z
    root     28766  0.0  0.0      0     0     ?    Z    Jun06   0:00 [apt] <defunct
    root@host:~# kill -9 28766
    root@host:~# ps aux |grep Z
    USER       PID %CPU %MEM    VSZ   RSS TTY      STAT START   TIME COMMAND
    root      2925  0.0  0.0   9256   880 pts/2    S+   18:40   0:00 grep Z
    root     28766  0.0  0.0      0     0     ?    Z    Jun06   0:00 [apt] <defunct

    So what gives? The ‘kill’ command is for killing processes, so what good is it if we can’t kill these processes? Zombie processes have ended, they are no more. Ghosts might be a more fitting term, traces of them exist in the system, but they are no longer functioning, they are waiting to gasp their last breath of exit code to their parent and have their memory wiped from the face of the system. The problem is the parent isn’t cooperating. It’s conceivable that this ignorance of its child’s death is intentional, but it is rare for a zombie condition to persists by design. If you see a zombie process and it doesn’t clear itself up in a moment, there is a good chance you’ll need to take matters into your own hands.. or you know, just ignore it.

    Using pstree we can find child pids if we know the parent, but how can we find the parent pid number from a child pid?

    Finding your own parent PID is easy in bash, it’s stored in the PPID variable.

    user@host:~$ echo $PPID
    5828

    What about finding the parent of an arbitrary process ID? If you have the proc file system (you probably do), you can see lots of information about a given process including the parent pid by looking at the ‘stat’ file for that pid.

    root@host:~# cat /proc/6124/stat
    6124 (bash) S 5828 6124 5828 34820 6398 4202496 1166 3867 0 0 4 0 0 1 20 0 1 0 7883109 25640960 617 18446744073709551615 4194304 5111244 140736553088608 140736553087152 140358224629054 0 65536 3686404 1266761467 18446744071579277349 0 0 17 0 0 0 0 0 0

    The 4th value in the ‘stat’ file is ppid, or the “parent pid” of the process.

    The ‘stat’ file for any pid in a procfs enabled system can be found in /proc/[pid]/stat, where [pid] is replaced with the pid number you are interested in. For a description of the ‘stat’ file format search for ‘/proc/[pid]/stat’ at the URL below:
    http://www.kernel.org/doc/man-pages/online/pages/man5/proc.5.html

    To see just the pid number and ignore the other information we’re not currently interested in we can use the ‘awk’ command to select only the 4th field.

    root@host:~# awk ‘{print $4}’ /proc/6124/stat
    5828

    Armed with the information above, I’ve created a quick little zombie hunting script for use in the cron scheduler, or command line. The script first tries to alert the parent process to reap its child using the SIGCHLD signal. When SIGCHLD fails SIGKILL is used next.

    Zombie Hunter

    #!/bin/bash
    zombies=(`ps ax |awk ‘{print $3" "$1}’ |grep -e ^‘Z ‘ |sed ‘s/Z //1’`)
    for zombie in ${zombies[@]}
    do
        echo "Found a zombie process "`awk ‘{print $2}’ /proc/$zombie/stat`" [pid:$zombie]"
        parent="`awk ‘{print $4}’ /proc/$zombie/stat`"
        echo "Asking parent process "`awk ‘{print $2}’ /proc/$parent/stat`" [pid:$parent] to come quietly…"
        kill -SIGCHLD $parent
        sleep 10 # This seems awfully patient
        if [ -f /proc/$parent/stat ]; then
            echo "Asking not so nicely"
            kill -9 $parent
        fi
        sleep 1
        if ! [ -f /proc/$zombie/stat ]; then
            echo "Zombie vanquished"
        fi
    done
    root@host:~# ./zombie-hunter
    Found a zombie process (apt) [pid:28766]
    Asking parent process (run-parts) [pid:28763] to come quietly…
    Asking not so nicely
    Zombie vanquished
  • Access an APC AP5456 IP Gateway for Analog KVM in Linux

    Access an APC AP5456 IP Gateway for Analog KVM in Linux

    I recently wrote about running an ActiveX component without Internet Explorer. I used that technique to come up with a shell script front-end for downloading, unpacking and running an executable in Wine for accessing an APC IP KVM (model AP5456). Here is the results of that effort.

    At a minimum the script requires Wine and curl, but to do everything without manually downloading and extracting the cabinet file yourself, you’ll also need cabextract. Some less exotic stuff from coreutils: basename and dirname are needed, but you probably have those already.

    You can download the script here.

    #!/bin/bash

    # A script to access an APC brand model AP5456 IP Gateway for Analog KVM
    # In Linux, using Wine

    USERNAME="apc"
    PASSWORD=""
    KVM_SERVER=""

    VIEWER_PATH=""

    while [ -z "$KVM_SERVER" ]
    do
        echo -n "FQDN of KVM Server (not including https://): "
        read KVM_SERVER
    done

    while [ -z "$USERNAME" ]
    do
            echo -n "Username: "
            read USERNAME
    done

    while [ -z "$PASSWORD" ]
    do
            echo -n "Password: "
            tput invis
            read PASSWORD
            tput sgr0
    done

    get_cab_name () {
        while read line
        do
            if [ "${line/’OBJECT CODEBASE’/}" != "$line" ]; then
                cab_name="${line##*'<OBJECT CODEBASE="’}"
                cab_name="${cab_name%%’"’*}"
                echo "$cab_name"
            fi
        done
    }

    get_cmd_args () {
        while read line
        do
            if [ "${line/PARAM/}" != "$line" ]; then
                ipaddr="${line##*'<PARAM NAME="ipaddr" VALUE="’}"
                ipaddr="${ipaddr%%’"’*}"

                sessionkey="${line##*'<PARAM NAME="sessionkey" VALUE="’}"
                sessionkey="${sessionkey%%’"’*}"

                encryptionkey="${line##*'<PARAM NAME="encryptionkey" VALUE="’}"
                encryptionkey="${encryptionkey%%’"’*}"
            fi
        done
        if [ -n "$sessionkey" ] && [ -n "$encryptionkey" ] && [ -n "$ipaddr" ]; then
            echo "-k $sessionkey -e $encryptionkey $ipaddr"
        fi

    }

    wine="`which wine`"
    if [ -z "$wine" ]; then
        echo "Wine was not found, but is required.  Can not continue without it." >&2
        exit 1
    fi

    cd "`dirname $0`"
    #If a path to vpclient.exe wasn’t specified, and we can’t find it, use /tmp
    #as a place to download it to
    if [ -z "$VIEWER_PATH" ] && ! [ -f "vpclient.exe" ]; then
        mkdir -p "/tmp/vpclient"
        VIEWER_PATH="/tmp/vpclient"
    fi

    #If a full path wasn’t specified, prepend current working directory
    if [ "${VIEWER_PATH:0:1}" != "/" ]; then
        VIEWER_PATH="`pwd`/$VIEWER_PATH"
    fi

    if ! [ -f "$VIEWER_PATH/vpclient.exe" ]; then

        cab_url="`curl -s –insecure –user "$USERNAME:$PASSWORD" "https://$KVM_SERVER/launch.asp" |get_cab_name`"
        cab_url="https://$KVM_SERVER/$cab_url"

        cabextract="`which cabextract`"
        if [ -z "$cabextract" ]; then
            echo "vpclient.exe was not found in $VIEWER_PATH" >&2
            echo "cabextract was not found either, so we can’t download">&2
            echo -e "and extract vpclient.exe from it’s CAB archive.\n" >&2
            echo "Error: user intervention required." >&2
            echo "———————————-" >&2
            echo "Download $cab_url" >&2
            echo "Unpack `basename $cab_url` into $VIEWER_PATH" >&2
            echo " — OR — " >&2
            echo "Install cabextract" >&2
            exit 1
        fi

        if ! [ -d "$VIEWER_PATH" ]; then
            echo "WARNING: Viewer Path specified is not valid." >&2
            echo "Path: $VIEWER_PATH" >&2
            echo -e "——————————————-\n" >&2
            echo -n "Would you like me to create it? (y/N)?" >&2
            read answer
            shopt -s nocasematch
            if [ "$answer" = "y" ]; then
                mkdir -p "$VIEWER_PATH"
                if ! [ -d "$VIEWER_PATH" ]; then
                    echo "Couldn’t create: $VIEWER_PATH" >&2
                    exit 1
                fi
            else
                exit
            fi
            shopt -u nocasematch
           
        fi

        cd "$VIEWER_PATH"
            curl -s –insecure –user "$USERNAME:$PASSWORD" "$cab_url" >"`basename $cab_url`"
            $cabextract -q "`basename $cab_url`" 2>/dev/null
    fi


    cmd_args="`curl -s –insecure –user "$USERNAME:$PASSWORD" "https://$KVM_SERVER/launch.asp" |get_cmd_args`"

    cd "$VIEWER_PATH"

    #Sanity check
    if [ -f "./vpclient.exe" ]; then
        $wine "./vpclient.exe" $cmd_args
    else
        echo "Failed, do it yourself: $wine vpclient.exe $cmd_args"
    fi
  • Bash Snippet: URL Encoding

    Bash Snippet: URL Encoding

    One approach would be to encode everything, but the approach I took was to just encode things I thought might be problematic to pass over the query string.

    URL (a.k.a percent) encoding of a string in bash:

    urlencode () {
            tab="`echo -en "\x9"`"
            i="$@"
            i=${i//%/%25}  ; i=${i//’ ‘/%20} ; i=${i//$tab/%09}
            i=${i//!/%21}  ; i=${i//’"’/%22}  ; i=${i//#/%23}
            i=${i//\$/%24} ; i=${i//\&/%26}  ; i=${i//\’/%27}
            i=${i//(/%28}  ; i=${i//)/%29}   ; i=${i//\*/%2a}
            i=${i//+/%2b}  ; i=${i//,/%2c}   ; i=${i//-/%2d}
            i=${i//\./%2e} ; i=${i//\//%2f}  ; i=${i//:/%3a}
            i=${i//;/%3b}  ; i=${i//</%3c}   ; i=${i//=/%3d}
            i=${i//>/%3e}  ; i=${i//\?/%3f}  ; i=${i//@/%40}
            i=${i//\[/%5b} ; i=${i//\\/%5c}  ; i=${i//\]/%5d}
            i=${i//\^/%5e} ; i=${i//_/%5f}   ; i=${i//\`/%60}
            i=${i//\{/%7b} ; i=${i//|/%7c}   ; i=${i//\}/%7d}
            i=${i//\~/%7e}
            echo "$i"
            i=""
    }

    urlencode "The sun is hot, 42 / 0 = undefined,  & 1 + 1 = 2…"