Glimpsing the Heap

You connect into the server and type “w” to see who else is on. You see a connection from an IP you don’t recognize!

Who is that? What are they up to?

Yes, there are many logs you should probably start looking through at this point, and many seasoned system administrators might shout strace! (a venerable and invaluable tool), but this connection has been idle for days, strace will probably be fairly boring at the moment.

Should we kill the process (nicely) in an attempt to get it to write to the bash history file? What if it is a malicious user who has already unset $HISTFILE? What if it is another system administrator who might get cranky about having their terminal killed?

Ah-ha, so we should fire up our favorite debugger and connect to the running process, right?

Sure, if your debugger of choice happens to be installed on the system and you are comfortable with using it, but a simpler, more accessible option may also be available.

Ah, it’s just the new junior administrator, who added a new email alias and evidently forgot to logout. Phew!

We did all this with a bash script? How?

OK, truth be told, bash is not the best choice for something like this, and it only works because we make some unreliable assumptions (hence the “Was the last command … ?” remark), but what I’m about to show you may still provide you with useful insights into otherwise tricky SysAdmin problems.

Before I break it down, let me further pique your interest with another example.

Your good friend Joe in the I.T. department is setting up the CEO’s new desktop. The plan was to migrate the user profile from the CEO’s laptop, but the CEO decided last minute (as they do) that he was going to keep on using the laptop and promptly went away on an important trip. Joe has no idea what the CEO’s email password is, but wants desperately to have the new desktop ready to go for when the CEO returns.

OK, there are a number of troubling presumptions with this scenario, both technical and ethical, but assuming this does not violate company policy, and that we can trust Joe with the CEO’s password, while completely ignoring the burning questions of why we have no backups of the user profile, or whether we consider it a wise and reasonable idea to store such an important password in an email client to begin with, how do we go about recovering the password?

We can’t change the password without disrupting the CEO’s email access from his laptop. With the CEO out of the office and otherwise unreachable this could be a major no-no.

Ah, how about tcpdump?

Sure that’s a good idea.. in the mid-1990s. Even though we scoff at security by saving easily recoverable passwords on the hard disks of traveling laptops, we draw the line at allowing employees to check email without encryption from any old coffee shop WiFi connection. We’ve got SSL/TLS in play, tcpdump is a no go.

OK, so we setup socat or .. something.. use our same certificate chain and.. nevermind, sorry Joe you’re on your own.

But wait, the logs indicate the CEO is currently accessing IMAP!

OK, so yeah I replaced all the sensitive data in these examples, but they are all real working solutions, and they were all achieved with simple bash scripts that make use of the proc filesystem to read process memory.

Time to look behind the curtain. Lets go back to our first example, viewing the last command typed in someone else’s bash shell. I want to start here because there are a few less steps than the IMAP example, and it’s easier to understand.

Our target bash shell was at process ID 28095. We can get access to the memory of this process by reading /proc/28095/mem, but we can do better than that. The kind of interesting dynamic stuff we want is in the “heap” portion of the process memory, and thankfully we’ve got a treasure map to get right down to the good stuff (edited for brevity):

See it? The heap is in the memory range between 0x098e4000 and 0x09aa2000.

We want to read that memory space, but Bash is kind of crummy for working with non-printable characters.

First of all, how do we read just that portion of memory? We can use dd.

The dd command allows you to specify a number of blocks to skip (using “skip=“), as well as a number of blocks to read (using “count=“). We’ll use “ibs=1” to specify that we want to work with a block size of 1 byte.

But we have a small problem, dd wants these values in decimal, but we have a range of hexadecimal numbers. No problem, we can convert with bash itself.

You might be lucky enough to have the strings program (part of binutils) available on your system. Sadly this isn’t always a given, but for this purpose we’ll assume you do. You have other options of course, but this is just easiest.

We can pipe the output of dd into strings and read all the human friendly bits.

In Bash our last command is stored in the environment variable “$_”

Knowing this we can pipe the output of dd into strings into grep and look for a line containing “$_“. Of course there is no reason why the heap might not contain many occurrences of “$_”, but as it turns out, we seem to get lucky here.

Now that we understand what is happening, lets see how the sausage is made.

Here’s a quick walk-through of the above script:

We start with some basic usage checking, then grab the “heap” line from our map.

Using some bash we grab just the starting value of the heap range, and then we convert that from hex to decimal.

Next we dd, not caring to bother specifying an upper limit, pipe the output to grep, where we pass the “-m1” switch which tells it to stop after the first occurrence (this helps us cut down on false positives, and makes it easier to script).

That’s it, short and sweet. With the possible exception of “strings” these commands are available on probably most of the Linux systems you work with.

How about that IMAP example?

Exactly the same, but with more parsing of the output. Here we’re using the IMAP server dovecot with a MySQL database containing the hashed user credentials and settings. We want to look at the memory of any dovecot auth-workers talking to MySQL. The areas of the heap that we are interested in involve Dovecot’s “PASSV” request. Instead of looking for “$_” we look for “PASSV” and we repeat this for all auth-workers, and we keep doing this until we get our answer.

Properly parsing the heap requires better understanding of the program and identifying the pointers that reference those areas of memory of interest to us. We can’t generically use offsets of the heap address range because these might change with the flow of the program execution, and even with the same inputs will vary from system to system depending on the machine architecture, compiler, and other environmental factors.

Forget proper; by clumsily grabbing for bits of human-friendly text we might manage to actually come up with a fairly reliable and reusable script.

In many cases this blind luck approach is good enough, and can save us the effort of chasing these values down through gdb or another debugger (at the cost of reliability in our results).

TL;DR when a quick strace fails, think about strings’ing the heap, you might be surprised at how helpful it can be.