On tools

— 03 November 2011
by Paul
tags: programming

Table of contents

htop > top

If you ever worked on a UNIX thingy, I suspect you’ve had a process spinning, or filling up the whole memory, or something nasty like that. In the Windows world, you could probably rush to ctrl+alt+del to fire up the task manager, and kill the process.

But on our lovely UNIX, our beloved teacher told us all years ago: top, way to go, just type in in a terminal, see what’s going on, and eventually, lookup the line of the resources-greedy process in the table, while the screen is flickering and all because everything is messed up on the machine and kill it by pressing k, followed by the PID of the process, that of course, you have forgotten. Easy, no ? LOLWUT, says the Windows, user. And he’s right, LOLWUT indeed. Its damn hard to see the percentage of RAM used, also. Oh, and it’s ugly.

Here come htop, on its white horse, ready to free up some precious minutes and relieve your eyes. Please compare the damn thing:

A screenshot of htop and top

Better, isn't it ? Notice the fresh colors ! (top on the left, htop on the right)

And below the eye candyness, it has some very powerful feature:

  • Process selection, using the arrow ;
  • Display nice graphs on CPU usage for each cores, memory (including virtual, resident, etc.) ;
  • Kill (or in fact send any signal to) the selected process using F9 or k ;
  • Sort the processes by RAM usage, CPU consumption, uptime and several other goodies, using F6 ;
  • Ability to display the parent-child tree using F5 ;
  • Customisation (color scheme, layout, etc.);
  • Search (oh, I love this one) using F3 or /;
  • Run lsof on the highlighted process ;
  • Run strace on the selected process (very useful for debugging) ;
  • Tag process using the space bar, to track them more easily.

Oh, and the best feature: it’s command-line based, so I can run it on my server, via ssh on a remote machine, and I don’t have to use the mouse to do thing. And I can even run it from a tty when my machine is completely crashing and unresponsive (which happens way too often when you try to benchmark stuff).

vim > *

So, I don’t want to troll or anything, but let’s face it, vim is the best text editor available on any OS (including Windows, having worked three months on a Windows machine using the Windows port of GVim). It has all the features of the other text-editors (via the wonderful plug-in system), and has the powerful characteristic of being modal. While pointed as a disadvantages by many persons (who are not vim user, obviously), it is by far the best way to edit text (especially for a programmer, but since I write this blog in GVim, I tend to think it fits any kind of writing).

This won’t be a article on vim feature, since there are plenty on the internet (ask me if you want a good introduction), but rather a list of reason why it is superior.

  • It is available on any OS, command-line, graphical. By any, I mean virtually all Windows, UNIX (including MacOSX, all BSDs, all Linux distros). UNIX/Windows/other. vi is part of POSIX. If you learn vim, you can edit text on any operating system, without problem. It can allow you to edit text over ssh, and is so lightweight that you don’t even notice that you’re editing a distant file.
  • It has a text-based configuration : I keep my configuration in a git repository on Github, and if I have to use another computer, I do a simple git clone in my home, and it’s all set up according to my exact preference. A text-based configuration allow one to share his tricks. And I love sharing.
  • It has plugins for any languages that exist, and is extensible. The vimscript language has binding in all major script languages (Python, Ruby, etc.), and is quite simple to learn (in fact, the community is so active, that when I need a particular feature, in 99% of the cases, I just have to git clone a repo on Github).
  • It make you reach efficiency level that you can’t even imagine when you’re not using a modal editor. I personally hate it when I have to delete a string between double quote. I have to press ctrl+left multiple times to go before the first word in the string, then press ctrl+delete for each word in the string. In vim, I can issue di", meaning «  Delete Inside ” ».
  • It integrates with other command-line tools in a nice way : you can make from vim, and navigate between the errors, same for grep (or rather ack, see below), you can directly filter the text you are working one using the powerful UNIX machinery, insert text from the shell, etc. Again, it can seem a bit weird to insert text from the command line, but in fact, it is really useful (and calling sort directly on a file is definitely a cool feature).
  • When you are proficient with vim, you experience a kind of zen, which is explained in details by the author of this article on Stack Overflow. You speak another language, and any text manipulation you want to do can be expressed in the language, in a couple keystroke. In a week, you keep up with you old speed (using a regular text editor), in two weeks, you cannot understand how you managed to live this long without using a modal editor and you get a nice speedup.
  • You don’t have to touch the mouse to do a single thing in Vim. This result in less movements, an more focus, since you don’t lose your concentration while ranting about your text editor while doing modifications (which happened to me way too often in the past).
  • This one may not be a good reason to use Vim for everybody, but I think a cool thing with this software is that you can always learn something new, a guru trick, discover a new plugin, etc.
  • Last but not least, I gives you nerd-credibility. And you know how much this is important.

zsh > bash

I’ve heard some people are still using bash these days. Considering how much time zsh saved me since I started to use it, I think it’s worth a try.

Install it, and then type chsh login, where login is your login. It should prompt you for the shell you want, make sure to type the full path (i.e. /usr/bin/zsh or something). Then, you can either use it vanilla, or with one of the cool config available on the internet (here or here).

Among other features :

  • Very nice completion of files (for example, if you type evince, and then tab tab, it will auto complete to show only pdf files).
  • If you have a server on which you authenticate without having to type a password (public key, session reuse, etc.), zsh completes the remote files when you are doing an scp.
  • Very powerful globing : ls .vim/**/*.txt searches for text file recursively using ls under the .vim directory (i.e. search all plugin documentation), vim -p *(as-30) open the files accessed less than 30 seconds ago (zsh autocompletes at each step of the command, so you don’t have to remember everything : vim -p *(<TAB> yields the list of options).
  • Spelling correction (i.e. find typos in your commands).
  • History call substitution : cat !$ replaces !$ by the last argument of the last command, cat !* will replace !* by all the arguments of the last commands, (!:n expands to the nth argument.) etc.
  • Fully compatible with sh or bash scripts.
  • Auto pushd & popd thing : dirs -v show the stack, cd ~N change directory to the Nth.
  • Fuzzy directory substitution : /u/lo/li<TAB> yields to /usr/local/lib.
  • Deep directory naming : hash -d moz=/home/paul/workspace/mozilla-central, and then cd ~moz.
  • vared : edit an environment variable on the fly.

nc > telnet

I guess you’re used to use telnet when doing network programming/debugging. It is definitely a handy tool when you want to see how a server is responding. But to debug a client, it is quite useless. And these weird escape key sequence are always a pain to remember.

Here comes the mighty netcat, dubbed the swiss army knife of network programming. It can act both as a client and a server, both in UDP and TCP, and has a proper stderr output.

Some examples:

# the new wget
nc blog.paul.cx 80
> GET / HTTP/1.0 # type this, followed by two new line
>
>
# See what header a browser sends : open localhost:8080 in you browser
nc -l 8080
# Network file transfer
# on the fist computer
nc -l 1234 > file
# on the second computer, file transfer
nc 10.0.0.2 1234 < file

# Power of UNIX : compressed file transfer over network in one line
nc -l 1234 | gzip -df > file
cat file | gzip -f | nc 10.0.0.2 1234

# Send UPD packets to another machine
ns -u 10.0.0.2 1234

As far as I know, there is no case where telnet is superior to nc. However, socat, dubbed netcat++, is considered a superior alternative to both netcat and telnet, although not available in standard on a machine. It is capable of much more, at the expense of an harder to remember syntax (for me anyway).

ack > grep

Are you tired to type:

find . -name .git -prune -o -name '*.cpp' | xargs grep "symbol"

Anyway, I know I were, before going to the self-explanatory http://betterthangrep.com, and to issue a sudo aptitude install ack-grep. It is really a delight, as a programmer who doesn’t like IDEs:

  • Color output (yes, grep knows how to do that, but has no clue about the fact that it has matched in a .c file);
  • Language awareness
  • Ignore .git, .hg, .svn, binaries, core dumps and such;
  • Regexp as file argument;
  • Ability to have a default option set in .ackrc;

Here is a command with a similar behavior as the previous one:

ack --cpp symbol

Nice, isn’t it ? Of course, you can replace the --cpp switch by --js, --c, or even --java for the masochists.

Here is another one that kicks ass:

ack --nojs symbol

Hell yeah, it searches the location below . for things that are not in Javascript files.

Oh, and there is a killer feature, I let you try it out, because, obviously, you’ll use this tool:

ack --thpppt

On Debian and cousins, the packages as well as the name of the program is ack-grep (because ack is a kanji converter/checker), so I suggest to put this nice little line in your .zshrc (because of course, you don’t use bash anymore, do you ?).

alias ack='ack-grep'

gdb, valgrind, strace, ltrace, and ptrace > printf

Using Linux, MacOS or another flavour or UNIXish system, you have the advantage of having a whole range of debugging tools that have been polished after dozens and dozens of years of system programming by amazing people.

And yet it is common to debug using printf statements. Okay, I can acknowledge that gdb is a bit frightening at the beginning, but, being a command-line tools, numerous wrappers exist : ddd, QtCreator, Eclipse, cgdb you name it.

Anyway, I prefer to use it on the command line, as often. I won’t write a full gdb tutorial, because I’m in no way the most knowledgeable person about that program, but there is plenty of tutorial on the internet.

If you really want to debug using printf, use at least fprintf with stderr at first argument, because stderr is unbuffered, whereas stdout is line-buffered (that is, it waits for a '\n' or equivalent to output the line on your screen, meaning that a printf("plop"); is not printed immediately). Also, there is almost no excuses these days to have a handy header file containing a ready to use set of logging functions, which has the benefit of handling all this for you (plus thread safety, colors, date, thread d printing, etc.).

valgrind is probably the most powerful debugging program available to the UNIX programmer (from my modest experience). It allows you to track down memory leaks (using the --leak-check=yes|full), to profile the heap (with massif), the cache miss rate (with cachegrind), to profile using call graphs (with callgrind), to find deadlocks and other concurrent problems (using helgrind and drd for stack memory), to find pointer overrun (using ptrcheck).

Additionally, I like to use the kcachegrind program, which analyse graphically the output of numerous Valgrind tools (cachegrind, callgrind, etc.). Numerous converters between file formats exist, to be able, for example, to use the cool kcachegrind interface with gprof dumps.

It is often useful to know what is doing a given program, or where your own program seems to lock under certain circumstances. strace and ltrace are able to print, respectively, the system calls and the library calls a program is doing, as well as pretty printing in a convenient way the arguments of the functions. They are invaluable to have a quick idea of what is happening, before throwing in a heavier tool such as gdb.

vimpager > most > less > more

When it comes to reading manpages, I find it important to have a comfortable software : colors, key bindings, other features such as search can bring speed and efficiency to you manpages reading, when it comes to search for a particular command line option, or a errno value of a system call.

At first (when I was a baby programmer), I used more, because I did not question myself. I typed man ls, and read the whole thing, and eventually, I had no idea that more was the program used to display the text with all formatting. I used to think man was the whole program, rather shitty because it did not allow me to go up or other fancy feature, but hell yeah, it gave me access to knowledge so I wasn’t complaining.

Eventually, I knew about less, and thought that it would be cooler to use it as my pager for man pages. I discovered the -P switch for the man command, and it was pretty cool. I had search, I could move down and up, and had a plethora of other shortcuts that I forgot.

Once, a guy (I think it was the mighty Maxime Gaudin), showed me most, which, logically, was better than more or less. And for one obvious reason, it had all the features less had, but had color support. Yes, but not quite. One thing I find horrible is that the G (that is, capital g), is bound to « Go to line ».

Consider a typical use-case :

« Let’s go to the bottom, to see a quick example »

As a Vim user, I press G, and, with most, and I’m stuck, I don’t know what to do, so I type iiiiii, and eventually Enter and then it tells me that iiiiii is not recognized as a number, and then I remember that most uses t and b to go to the top and the bottom. Which seems logic, but I don’t like it. I couldn’t get used to it, constantly triggering the go to line feature, and losing several seconds each time. For a heavy terminal user, clearly, most is not an acceptable program.

Finally, I present the magnificent vimpager script, which has all the features vim has (let’s say it’s sufficient), including the perfect key mapping (hjkl to move, / to search, etc.) and the coloring of manpages. As a bulimic vim user, it is absolutely perfect for my use of the terminal. Additionally, several keys have been remapped, such as q to quit the pager, which seems quite logical and consistent. I would say that a user of less should use vimpager, as the key bindings are similar, and vimpager has color support.

Don’t wait, grab a copy of vimpager there : http://www.vim.org/scripts/script.php?script_id=1723 !

On a related note, when you want to consult a file but in read only, I was used to type less file, but in fact, during a late night read of the vim manpage, I found the view variant, which basically starts vim in read-only mode.

lsof

When you do development things, or sysadmin stuff, you always need to know what process has this file opened, or which ports are used by that software.

Well, I guess you could probably strace them using a nice cuisine of grep and friends, but guess what? There is a cooler way. And it’s always nice to be cooler, right ?

Let me introduce … lsof a.k.a. Mr. « I’ve got the longest list of options in the whole UNIX history » a.k.a. LiSt Open File

Let’s start gently. Which process has the file file opened :

lsof file

Then, the other way around : which files (including sockets, pipe, etc.) has a certain process opened :

lsof -p PID # PID is the PID of the process
lsof -c firefox

Another useful thing, listing the processes and file that are opened under /home/user, but not in /home/user/secret_things :

lsof +d /home/user -d /home/user/secret_things

And for network guys, list process interacting (i.e. connecting to, disconnecting to, connected to, etc.) list IPv4 addresses, using TCP, connected to paul.cx, and restrict to HTTP connections :

# full version
lsof -i 4TCP@paul.cx:http
        | |    |      |_ using HTTP
        | |    |
        | |    |_ interacting with paul.cx
        | |
        | |_ using TCP
        |
        4 ipv4
# short version :
lsof -i @paul.cx:http
# show all connections
lsof -i

So, here it is, no complaining ever about opened file somewhere or socket already bound to a port without SO_REUSEADDR, eventually preventing a server to run.

vim-orgmode > todo.txt

Honestly, I’m probably not the guy who uses vim-orgmode the most, and in the best bay, as I use it as an outliner/todo list manager, but anyway, it is pretty awesome, compared to my previous solution, the famous todo.txt everyone has.

Basically, it consists in one Vim plug-in which tries to be an orgmode-clone for Vim. So, yeah, it doesn’t have all the feature (since emacs’ orgmode can probably wash the dishes, using esc-meta-alt-ctrl-d), but it works well enough for me:

You can write things like this:

* This is a top level fold
** This is a sublevel fold
   This is text.

   - These are …
   - items of a list
   - You can insert:
     - date using <leader>sa
     - lists using <leader>gil
     - tags using <leader>t
** This is another sublevel fold
   This is text.
* This is a toplevel fold.

and it gets nicely formatted. The whole point is, for people not familiar with how an outliner works, that TAB folds and unfolds the folds, keeping the file structured, clean and nice. I tend to think that clearly organized notes are freeing the mind from having to think and remember about stuff.

If you’re crazy enough to have emacs installed (I’m not, at least on my server, that makes me half-crazy, I suppose), then you can do a <leader>eh or a <leader>ep to export as html or pdf, respectively. But I tend to use Markdown for the things I want to publish (like this article), or Latex if I want it to be pretty.

For reference, you can rush to /doc/orguide.txt. The installation process is quite usual, for a Vim plug-in: source the .vba, or clone it in the ~/.vim/bundle directory if you are a user of the powerful pathogen plug-in (it you’re not, you should, trust me).

If you tend to like graphical/web software, you can use TiddlyWiki instead, but you should definitely put some authentication in front of it (I use vim-orgmode in a tmux over ssh on my server, so I don’t need anything else). Oh, and consider to use more command line based tools, because, you know, it’s nicer and all.

Paul 03 November 2011
 
« Précédent
Suivant »