A company I worked for had a big problem. They didn't use privilege separation anywhere. Network team had full root. Database team had full root. SRE had full root. System Administrators had full root. Managers had full root. Did I say "everywhere?" Yeah, everywhere, for every one of those teams and all of their staff.
Naturally, things would break - files would be deleted - systems would get misconfigured. This was a huge problem. The company needed to know and know now - Who let the dogs out?.
While enforcing proper sudo use (and limitations to who can access root and where/how) was not within my ability at the company, I could at least protect myself. The company supposedly had a keylogger on production systems, but - well, that's a story for another time and it didn't work.
Giving myself deniability
At this company I was tedious about making sure that the changes I made were documented. I mean this both in the actual company wiki/document repository regarding how things worked, but also within the issue-tracking system and procedural-logs to refer to and log individual changes also. When (inevitably) things would break, I just wanted the ability to comfortably say "It wasn't me."
"What about just keeping .bash_history files?" I'm glad you asked. Shell history files are fantasticly useful. Unfortunately, without some modifications, certain shells will not write out the history file unless the session is exited cleanly. Thus, having something like an SSH timeout sever your session can result in the history not being written out. Also, you'll see in the example below that seeing the command which was run does not always tell the whole story of what happened.
sudoreplay at a glance
Ever since it was first available, I've loved sudoreplay. In other sudo documentation I've published I love pointing out how useful sudoreplay is - especially with regarding editor sessions. sudoreplay ties in with sudo to allow you to replay (in time) sudo output logs.
Consider - without sudoreplay - you see this in a sudo log from an employee who has rights to edit this file on purpose:
firstuser$ sudoedit /etc/ourcustom.conf
10 minutes later, another user edits the file.
anotheruser$ sudoedit /etc/ourcustom.conf
Now imagine you're in such a bad place that there's no git/RCS/other management of this file. The configuration is now broken and the service won't start. Who did it?
With sudoreplay (and enabling log_input/log_output into your sudoers file) you could replace the editor session - and in realtime see the exact edits made by both users.
Keylogging myself with sudo
I wanted to have sessions of all work I had done, so anytime a question came up (even if only a question from myself, "Did I make a typo?", etc) I could answer it.
I wanted something semi-native (I did not want to spend all day searching online for something) and that I could consider secure. Since I was already familiar with sudo - and it was already installed on my workstation - it was definitely my golden hammer.
"But sudoreplay only works to replace sudo logs, certainly you aren't doing everything as root, are you?" Great assessment. That's right. sudoreplay is useful to replay sudo logs, and usually when I mention this unique use for it as protection/keylogging people familiar to some extents with sudo ask why I would do everything as root. The answer? I don't. sudo logs sudo sessions. sudo sessions are not strictly for elevating to root.
My setup
First, to avoid the "do everything as root" - consider:
Defaults log_input, log_output dayid maleah=(dayid) NOPASSWD: SETENV: ALL
OK, so I can do everything I want, as myself, without a password. What good is that? Well - as I said before - sudo does not have to be using the root user as the target.
Follow more:
16:04 dayid@maleah:~$ alias logme='/usr/local/bin/sudo -i -u dayid' 16:05 dayid@maleah:~$ logme 16:07 dayid@maleah:~$
Now I also have a simple bash alias to allow myself to... become myself? Yes! a 'myself' that is now within a sudo session and is thus being logged!
Without attaching video here this is truly a pinnacle "see for yourself" moment - once you're in this logged subshell/sudo session, open an editor and do whatever you'd like. Save the file (or not). List the session via sudoreplay -l, then play it back:
$ doas sudoreplay -l | tail -n 1
Feb 18 16:15:12 2021 : dayid : TTY=/dev/ttyp3 ; CWD=/home/dayid ; USER=root ; HOST=maleah.dayid.org ; TSID=00008O ; COMMAND=/usr/local/bin/vim -- /etc/ourcustom.conf
$ doas sudoreplay 0008O
sidenote: yes, I am mixing doas on my OpenBSD workstation with sudo. I keep sudo on this workstation to both allow me a place to test work-related sudo policies as well as for this sudoreplay function. If you are trying to do this on a Linux system then your similar command to list replay sessions would be just "sudo sudoreplay -l".
If seeing the editor session replayed at realtime speed (which you can fast-forward and slow as your desire) isn't useful/impressive to you - then you can probably stop now.
Less and less useful - thank goodness!
Fortunately these days I spend less and less time having to do rogue/terrible things directly on production machines - so using the above as a safety mechanism to cover myself and have logging/accountability even when the employer doesn't is not completely as useful. However, I have found that it is still very useful when first learning new things or working through documenting a new procedure. This allows me to dynamically explore and go through the whole procedure and - even allowing mistakes - replay it again to clean it up and document it later. This has been much more useful than trying to keep multiple windows open - one with an interactive shell and another with notes.txt - or try to rely on a shell history to document something later.
The bad
- Logs take disk-space. Yep.
- If you're handling protected data during your sessions, log_input and log_output can result in logging private data. You must use it carefully/appropriately.
- You must have the ability to edit the sudo file on the machine where you're running your self-logging from. For my use-case this is always a workstation so even if I am SSHing elsewhere I have that control.
- Spawning a subshell just to use logging does make more processes.
The good
- This makes taking notes for yourself much easier.
- If you ever question what you did - not just within a shell session but also across multiple hops of SSH or within something interactive - you can answer yourself and check.
- This gives you session logs that you can use as a proactive defense to confirm that breaks/errors on shared systems were not part of your session. Naturally providing your own log to prove your own innocence isn't truly failproof (since you could have modified it) - but still is useful and does no harm.
- You likely already have sudo installed on any workstation/system where you'd want to use this, if you don't, it's likely available and so you do not need additional custom setups for different machines.
