You connect into the server and type “w” to see who else is on. You see a connection from an IP you don’t recognize!
USER TTY FROM LOGIN@ IDLE JCPU PCPU WHAT
root pts/0 203.0.113.17 Wed10 2days 0.31s 0.31s -bash
root pts/1 192.168.1.42 14:43 0.00s 0.28s 0.01s w
Who is that? What are they up to?
Yes, there are many logs you should probably start looking through at this point, and many seasoned system administrators might shout strace! (a venerable and invaluable tool), but this connection has been idle for days, strace will probably be fairly boring at the moment.
Should we kill the process (nicely) in an attempt to get it to write to the bash history file? What if it is a malicious user who has already unset $HISTFILE? What if it is another system administrator who might get cranky about having their terminal killed?
Ah-ha, so we should fire up our favorite debugger and connect to the running process, right?
Sure, if your debugger of choice happens to be installed on the system and you are comfortable with using it, but a simpler, more accessible option may also be available.
root 28095 0.0 0.0 7148 3564 pts/0 Ss+ Mar21 0:00 -bash
root@host:~# ./last_bash.sh
Usage: ./last_bash.sh <PID of bash process>
root@host:~# ./last_bash.sh 28095
Was the last command... ./add_email_alias.sh ?
Ah, it’s just the new junior administrator, who added a new email alias and evidently forgot to logout. Phew!
We did all this with a bash script? How?
OK, truth be told, bash is not the best choice for something like this, and it only works because we make some unreliable assumptions (hence the “Was the last command … ?” remark), but what I’m about to show you may still provide you with useful insights into otherwise tricky SysAdmin problems.
Before I break it down, let me further pique your interest with another example.
Your good friend Joe in the I.T. department is setting up the CEO’s new desktop. The plan was to migrate the user profile from the CEO’s laptop, but the CEO decided last minute (as they do) that he was going to keep on using the laptop and promptly went away on an important trip. Joe has no idea what the CEO’s email password is, but wants desperately to have the new desktop ready to go for when the CEO returns.
OK, there are a number of troubling presumptions with this scenario, both technical and ethical, but assuming this does not violate company policy, and that we can trust Joe with the CEO’s password, while completely ignoring the burning questions of why we have no backups of the user profile, or whether we consider it a wise and reasonable idea to store such an important password in an email client to begin with, how do we go about recovering the password?
We can’t change the password without disrupting the CEO’s email access from his laptop. With the CEO out of the office and otherwise unreachable this could be a major no-no.
Ah, how about tcpdump?
Sure that’s a good idea.. in the mid-1990s. Even though we scoff at security by saving easily recoverable passwords on the hard disks of traveling laptops, we draw the line at allowing employees to check email without encryption from any old coffee shop WiFi connection. We’ve got SSL/TLS in play, tcpdump is a no go.
OK, so we setup socat or .. something.. use our same certificate chain and.. nevermind, sorry Joe you’re on your own.
But wait, the logs indicate the CEO is currently accessing IMAP!
Fri Mar 23 14:52:14 EDT 2018
root@host# grep ceo /var/log/mail.log |tail -n1
Mar 23 14:51:40 mail dovecot: imap-login: Login: user=<ceo@example.org>, method=PLAIN, rip=198.51.100.54, lip=10.0.0.10, mpid=26689, TLS
root@host# netstat -punta |grep ESTA |grep 198.51.100.54
tcp 0 0 10.0.0.10:993 198.51.100.54:51021 ESTABLISHED 1986/imap-login
root@host:# ./get_passwd.sh ceo
Waiting for ceo to send login credentials...
Password: T0pS3cretz
OK, so yeah I replaced all the sensitive data in these examples, but they are all real working solutions, and they were all achieved with simple bash scripts that make use of the proc filesystem to read process memory.
Time to look behind the curtain. Lets go back to our first example, viewing the last command typed in someone else’s bash shell. I want to start here because there are a few less steps than the IMAP example, and it’s easier to understand.
Our target bash shell was at process ID 28095. We can get access to the memory of this process by reading /proc/28095/mem, but we can do better than that. The kind of interesting dynamic stuff we want is in the “heap” portion of the process memory, and thankfully we’ve got a treasure map to get right down to the good stuff (edited for brevity):
00495000-00498000 r-xp 00000000 fc:00 16252982 /lib/i386-linux-gnu/libdl-2.15.so
007ad000-007cd000 r-xp 00000000 fc:00 16252955 /lib/i386-linux-gnu/ld-2.15.so
007da000-00979000 r-xp 00000000 fc:00 16252974 /lib/i386-linux-gnu/libc-2.15.so
0097c000-0097f000 rw-p 00000000 00:00 0
009e7000-009e8000 rw-p 0000a000 fc:00 16253022 /lib/i386-linux-gnu/libnss_nis-2.15.so
00b7c000-00b7d000 rw-p 00007000 fc:00 16253005 /lib/i386-linux-gnu/libnss_compat-2.15.so
00c58000-00c59000 r-xp 00000000 00:00 0 [vdso]
00d4d000-00d4e000 rw-p 0001d000 fc:00 16252980 /lib/i386-linux-gnu/libtinfo.so.5.9
00e11000-00e12000 rw-p 0000b000 fc:00 16253013 /lib/i386-linux-gnu/libnss_files-2.15.so
00e64000-00e65000 rw-p 00016000 fc:00 16252997 /lib/i386-linux-gnu/libnsl-2.15.so
00e65000-00e67000 rw-p 00000000 00:00 0
08125000-0812a000 rw-p 000dc000 fc:00 7340076 /bin/bash
0812a000-0812f000 rw-p 00000000 00:00 0
098e4000-09aa2000 rw-p 00000000 00:00 0 [heap]
b755b000-b7562000 r--s 00000000 fc:00 58988402 /usr/lib/i386-linux-gnu/gconv/gconv-modules.cache
b7562000-b7762000 r--p 00000000 fc:00 58985651 /usr/lib/locale/locale-archive
b7762000-b7764000 rw-p 00000000 00:00 0
bfd29000-bfd4a000 rw-p 00000000 00:00 0 [stack]
See it? The heap is in the memory range between 0x098e4000 and 0x09aa2000.
We want to read that memory space, but Bash is kind of crummy for working with non-printable characters.
First of all, how do we read just that portion of memory? We can use dd.
The dd command allows you to specify a number of blocks to skip (using “skip=“), as well as a number of blocks to read (using “count=“). We’ll use “ibs=1” to specify that we want to work with a block size of 1 byte.
But we have a small problem, dd wants these values in decimal, but we have a range of hexadecimal numbers. No problem, we can convert with bash itself.
160317440
You might be lucky enough to have the strings program (part of binutils) available on your system. Sadly this isn’t always a given, but for this purpose we’ll assume you do. You have other options of course, but this is just easiest.
We can pipe the output of dd into strings and read all the human friendly bits.
In Bash our last command is stored in the environment variable “$_”
root
root@host:~# echo $_
whoami
Knowing this we can pipe the output of dd into strings into grep and look for a line containing “$_“. Of course there is no reason why the heap might not contain many occurrences of “$_”, but as it turns out, we seem to get lucky here.
Now that we understand what is happening, lets see how the sausage is made.
#!/bin/bash
if [ -z "$1" ]; then
echo "Usage: $0 <PID of bash process>" >&2
exit 1
fi
if ! [ -d "/proc/$1/" ]; then
echo "Couldn't find PID #$1" >&2
exit 1
fi
skip=$(grep -e '\[heap\]'$ /proc/$1/maps)
skip="${skip%%'-'*}"
skip="$((16#$skip))"
echo -n "Was the last command... "
last_cmd="$(dd if=/proc/$1/mem bs=1 skip=$skip 2>/dev/null |strings |grep -m1 -e ^'_=')"
echo "${last_cmd#*'_='} ?"
Here’s a quick walk-through of the above script:
We start with some basic usage checking, then grab the “heap” line from our map.
Using some bash we grab just the starting value of the heap range, and then we convert that from hex to decimal.
Next we dd, not caring to bother specifying an upper limit, pipe the output to grep, where we pass the “-m1” switch which tells it to stop after the first occurrence (this helps us cut down on false positives, and makes it easier to script).
That’s it, short and sweet. With the possible exception of “strings” these commands are available on probably most of the Linux systems you work with.
How about that IMAP example?
Exactly the same, but with more parsing of the output. Here we’re using the IMAP server dovecot with a MySQL database containing the hashed user credentials and settings. We want to look at the memory of any dovecot auth-workers talking to MySQL. The areas of the heap that we are interested in involve Dovecot’s “PASSV” request. Instead of looking for “$_” we look for “PASSV” and we repeat this for all auth-workers, and we keep doing this until we get our answer.
#!/bin/bash
target_user="$1"
if [ -z "$target_user" ]; then
echo "Usage: $(basename "$0") <username of target>" >&2
echo -e "\tAttempts to recover password for specified user" >&2
exit 1
fi
echo "Waiting for $target_user to send login credentials..."
while [ "$?" == "0" ]
do
ps aux |grep 'dovecot/auth' |grep -v grep |awk '{print $2}' |while [ "$?" == "0" ] && read pid
do
if [ -d "/proc/$pid/" ]; then
skip=$(grep -e '\[heap\]'$ /proc/$pid/maps)
skip="${skip%%'-'*}"
skip="$((16#$skip))"
dd if=/proc/$pid/mem ibs=1 skip=$skip 2>/dev/null |strings |grep PASSV |while read line
do
line="${line#*"PASSV"}"
line="${line#*$'\t'}"
line="${line#*$'\t'}"
mailpass="${line%%$'\t'*}"
line="${line#*$'\t'}"
mailuser="${line%%$'\t'*}"
mailuser="${mailuser##*'='}"
mailuser="${mailuser%@*}"
#echo "$mailuser / $mailpass"
if [ "$mailuser" == "$target_user" ]; then
echo "Password: $mailpass"
exit 1
fi
done
fi
done
done
Properly parsing the heap requires better understanding of the program and identifying the pointers that reference those areas of memory of interest to us. We can’t generically use offsets of the heap address range because these might change with the flow of the program execution, and even with the same inputs will vary from system to system depending on the machine architecture, compiler, and other environmental factors.
Forget proper; by clumsily grabbing for bits of human-friendly text we might manage to actually come up with a fairly reliable and reusable script.
In many cases this blind luck approach is good enough, and can save us the significant effort involved in chasing these values down through gdb or another debugger (at the cost of reliability in our results).
TL;DR when a quick strace fails, think about strings’ing the heap, you might be surprised at how helpful it can be.
