Tag Archives: linux


ssh for Programmers


Way back when, just after the dot-com-bomb, I found myself work­ing for a media com­pany named Metapa. We made mp3s and 30 sec­ond pre­view clips from boxes of CDs that tech-clueless record com­pa­nies snail-mailed us. Metapa was staffed with a ton of Unix-heads and I received a great school­ing in the ways of the Secure Shell, ssh. You could not get to a server with­out ssh­ing into it. Our “VPN” was a server run­ning ssh, con­fig­ured to only accept key-based login. There was no way to log in with just a pass­word. You would con­nect and use ssh’s built-in port for­ward­ing to route more fancy con­nec­tions, as required. I got to know ssh incred­i­bly well.

At more recent jobs, peo­ple have used ssh to man­u­ally con­nect to boxes and to man­u­ally issue com­mands, but they never did any­thing more advanced. I never under­stood exactly why. I’m sure some of this is that these have been mixed Windows/Linux envi­ron­ments with heav­ier reliance on Windows, so the “cul­ture” of using ssh for every­thing just isn’t present. Part of it may also be lack of expe­ri­ence with ssh.

This arti­cle attempts to touch on the slightly more advanced meth­ods of work­ing with ssh. I am not going to get into port for­ward­ing, since that requires a much more advanced men­tal model and a good grasp of how to con­fig­ure the local and remote soft­ware. I also will not get into rsync, which is a com­mon tool that pig­gy­backs atop ssh to syn­chro­nize fold­ers. What I will dis­cus is a few of the more com­mon advanced uses:

  • copy­ing files
  • log­ging in with­out keys
  • aliases for server names
  • dif­fer­ing user­names
  • log­ging in to the cloud (com­bi­na­tion aliases, user­names, and alter­nate keys)
  • a quick fix for changed host iden­ti­ties

Copying Files

ssh has a com­pan­ion app named scp, or Secure Copy. It works exactly like the reg­u­lar Unix cp com­mand except the source and des­ti­na­tion can take ssh file paths in the form of username@machine:path/to/file.txt.

For instance, if I wanted to push my “.vimrc” file to a remote sys­tem named frodo, I could run any one of the fol­low­ing com­mands:

# a technically correct, but wordy way to do it
scp .vimrc briane@frodo:.vimrc

# destination file is the same name, so no need to specify
scp .vimrc briane@frodo:

# assumes my username is the same on frodo, so no need to specify
scp .vimrc frodo:

Note that when scp con­nects, it is in your home direc­tory. These des­ti­na­tions use rel­a­tive paths (because the paths do not start with a slash) so they are rel­a­tive to your home folder. If you wanted to copy a log file, you might use an absolute path such as:

# copy the Apache error log to the current folder
scp briane@frodo:/var/log/apache/error.log .

Logging in Without Keys

By default ssh (as well as scp) uses your Unix pass­word to access the remote sys­tem. This is all well and good, but it also offers another way to iden­tify your­self that allows for a bit more con­ve­nience and a lot more secu­rity. This method is dual-key cryp­tog­ra­phy. Put sim­ply, this is a whole bunch of math shared between two files, a “pub­lic” key and a “pri­vate” key. The pri­vate key is, effec­tively, a ludi­crously large num­ber — like a pass­word — that you share with nobody. The pub­lic key is math­e­mat­i­cally derived from the pri­vate key and can be used to check the authen­tic­ity of the pri­vate key. You can share the pub­lic key far and wide. When using keys with ssh, you store a copy of your pub­lic key on the remote server. Every time you log in to it, your machine and the remote machine do a lit­tle dance involv­ing the pub­lic and pri­vate keys, and you’re instantly logged in — no pass­word required. It effec­tively shifts the authen­ti­ca­tion mech­a­nism from “what you know” (a pass­word) to “what you have” (a data file with that long num­ber).

In con­cept, this is all well and good, but it is some­times a lit­tle painful and tedious to set up by hand. You have to ensure the cor­rect file is in the cor­rect place. The per­mis­sions have to be per­fect or the ssh server will refuse your login. After all, if your pub­lic key is writable by other peo­ple, then some­one with access to the sys­tem can over­write it with their pub­lic key and log in as you, with your full per­mis­sions and cre­den­tials. Fortunately, if there is one thing that com­put­ers are good at automat­ing, it is sim­ple, annoy­ing, some­times painful tasks.

First, you will need to cre­ate a key (if you have not done so already). It’s a sim­ple com­mand:


Just keep hit­ting Return to answer all the ques­tions with their defaults, includ­ing an empty pass­word. Next, you can use a script like the fol­low­ing, named colonize.sh, that will send your key to a remote machine over a reg­u­lar ssh-with-password con­nec­tion and set up the per­mis­sions.

if [ -z "$1" ]; then
    echo "Function: copies ssh public keys to remote machines for passwordless login"
    echo "Usage: colonize.sh <username>@<servername or ip address>"
    exit 1
CONTENT=`cat ~/.ssh/$KEY`
echo "Attaching key to authorized_keys file"
ssh $1 "mkdir -p .ssh && chmod 700 .ssh && touch .ssh/$STORE \
    && chmod 644 .ssh/$STORE \
    && echo '$CONTENT' >> .ssh/$STORE"
echo "Complete!  You should be able to log in without a password now!"

Now if you want to col­o­nize a remote machine with your ssh keys, you’d just do some­thing like:

colonize.sh briane@bilbo

Enter your pass­word (for the last time ever!) and next time you ssh or scp with that machine, it will use your keys instead of your pass­word.

Server Aliases

There is a con­fig file in your dot-ssh folder named sim­ply .ssh/config. This holds a wealth of options for cus­tom per-server set­tings. Long ago, I had a vir­tual pri­vate server with Dreamhost (I’ve since moved to Linode for reli­a­bil­ity rea­sons; if you’d like to try it out, feel free to use my refer­ral URL). This server had the ugly and dif­fi­cult to remem­ber name of “ps24493.dreamhost.com”. I ssh and scp all the time, but had dif­fi­culty remem­ber­ing that exact server num­ber. Fortunately, there is a bet­ter way.

Your .ssh/config file lets you define a Host (what you type to con­nect) to a HostName (the actual server name). For exam­ple, my con­fig had the fol­low­ing two lines:

Host dreamhost
HostName ps24493.dreamhost.com

This means that “dreamhost” ends up as short­hand for the more annoy­ing “ps24493.dreamhost.com”. If I want to log in, I’d just “ssh dreamhost”. If I wanted to copy a file, I could “scp index.html dreamhost:/var/www/my_website”. It made it much eas­ier to access the server.

Differing Usernames

I actu­ally had one more line in that part of con­fig file. The account name I use on most of my local machines is sim­ply “brian”. On that vir­tual pri­vate server, it was the much more ver­bose but globally-unique “brianenigma”. So tech­ni­cally, I could not con­nect with a sim­ple “ssh dreamhost” but “ssh brianenigma@dreamhost”. But there is another line that can be added to the .ssh/config to allow the for­mer vari­ant with­out a user­name.

You can define a default user­name for a spe­cific sys­tem by putting a “User” entry in the “Host” sec­tion. For instance:

Host dreamhost
HostName ps24493.dreamhost.com
User brianenigma

This, then, allows me to type in “ssh dreamhost” instead of the much more wordy “ssh brianenigma@ps24493.dreamhost.com”.

Logging in to the Cloud

Amazon’s Web Services is an inter­est­ing beast. You get a machine up in the cloud that you can log into. It typ­i­cally has an ugly name (which can be nick­named — see two sec­tions back) and may have a dif­fer­ent user­name than the one you use on your local machine (which can also be rede­fined — see the pre­vi­ous sec­tion). It throws in yet another wrin­kle. Password-based login is for­bid­den and it assigns you a pub­lic and pri­vate key pair. Because it assigns you a key, it is obvi­ously going to be dif­fer­ent than any key you have pre­vi­ously gen­er­ated for all your other con­nec­tions.

There is an ugly way to spec­ify the key­store you down­loaded from Amazon on the com­mand line, but a far more sim­ply way is to define it in your .ssh/config, such as this entry that com­bines all of the above and adds the key­store:

Host cloud
HostName ec2-256-301-297-271.us-west-2.compute.amazonaws.com
User administrator
IdentityFile ~/.ssh/my_amazon_key.pem

Now you can log in to your AWS account by just typ­ing “ssh cloud”. ssh will expand that out to the very ugly equiv­a­lent:

ssh -i ~/.ssh/my_amazon_key.pem administrator@ec2-256-301-297-271.us-west-2.compute.amazonaws.com

Remote Host Identification has Changed

I work in a lab envi­ron­ment with lots of servers that are fre­quently refor­mat­ted. This means that the servers retain their name and IP address, but the ssh fin­ger­prints that uniquely iden­tify them (to help thwart third par­ties from pos­ing as the server and steal­ing your pass­words) get wiped out. In this case, if I attempt to log in, I get a big scary mes­sage like the fol­low­ing:

Someone could be eavesdropping on you right now (man-in-the-middle attack)!
It is also possible that the RSA host key has just been changed.
The fingerprint for the RSA key sent by the remote host is
Please contact your system administrator.
Add correct host key in /home/briane/.ssh/known_hosts to get rid of this message.
Offending key in /home/briane/.ssh/known_hosts: 6
Permission denied (publickey,password).

The fix here is to open up .ssh/known_hosts and delete line 6. Technically, delet­ing the entire file also works (and I have seen plenty of peo­ple do that as a “fix”), but that has the side-effect of forc­ing you to con­firm, all over again, every remote host you con­nect to as a new host. So: “delete line 6” is sim­ple enough to do, but also annoy­ing. We can auto­mate that. The fol­low­ing script, called fixssh.sh, will retain the past nine back­ups of your known_hosts file and delete the offend­ing line. In this case, you’d run it with “fixssh.sh 6”.


if [ -z "$1" ]; then
    echo "Put line number of your ~/.ssh/known_hosts file for deletion on the"
    echo "command line."
    exit 1

cd ~/.ssh/
rm -rf known_hosts.9
for NUM in 8 7 6 5 4 3 2 1 ; do 
    ((NUMPLUS=$NUM + 1))
    if [ -d known_hosts.$NUM ]; then 
        mv known_hosts.$NUM known_hosts.$NUMPLUS 
mv known_hosts known_hosts.1
cat known_hosts.1 | sed "$1 d" > known_hosts
chmod 644 known_hosts


I hope these tips help you become more pro­fi­cient at ssh usage. Once you have these down, you might search on your own for the more advanced top­ics like how to use rsync, to keep mas­sive amounts of files in sync, or using ssh port for­ward­ing.

Posted in: Code Software Work


Natural” scrolling across operating systems

Natural” Scrolling Overview

I am begin­ning to get used to the reversed scrolling behav­ior in the new OS X. For those that have not heard about it, this was one of the most con­tro­ver­sial changes in the new oper­at­ing sys­tem. They reversed the direc­tion of the action that is per­formed by spin­ning the mouse’s scroll wheel or using two fin­gers to scroll on a touch­pad. There have been a lot of heated debates on both sides of the fence.

If you think about it, the scroll wheel action is fairly arbi­trary. It was:

  • scroll the wheel down, the slider in the win­dow that marks your posi­tion moves down­ward (to match your fin­ger move­ment), the text of your doc­u­ment slides up, you reveal more text at the bot­tom edge of the win­dow.

It is now:

  • scroll the wheel down, the slider in the win­dow that marks your posi­tion moves upward, the text of your doc­u­ment slides down (to match your fin­ger move­ment), you reveal more text at the top edge of the win­dow.

In other words, the move­ment switched from manip­u­lat­ing the win­dow slider to manip­u­lat­ing the doc­u­ment itself. You might argue that it is a lit­tle more direct now — instead of manip­u­lat­ing the slider that manip­u­lates your posi­tion in the doc­u­ment, you are directly manip­u­lat­ing the doc­u­ment. You might argue that, at least for touch-based track­pad sys­tems (not mice, per se), the action is more iPhone-like. You might also argue that as a soci­ety we now have a cul­ture muscle-memory of how the scroll wheel works and chang­ing it would be akin to swap­ping the posi­tion of the gas and brake ped­als on a car or (less dan­ger­ous) build­ing ana­log clocks whose hands spun counter-clockwise.

I fig­ured I would at least give it a try, but had to jump in with both feet. That meant leav­ing the new default OS X reversed (“nat­ural”) scrolling, but that also meant hack­ing my Windows and Linux boxes to behave sim­i­larly — oth­er­wise, I found myself get­ting stuck in a limbo of not know­ing which direc­tion the scroller worked at any given time, hes­i­tat­ing with a lit­tle test nudge to ensure it was going to work the way I thought.


For me, the Windows hack was pretty easy. I already run AutoHotkey for a few key­board short­cuts. AutoHotkey is an app that inter­cepts key­presses and mouse clicks and lets you trans­form them to other actions. I have a few macros set up (sim­i­lar to TextExpander on the Mac) that lets me type an abbre­vi­a­tion, which gets expanded to a hard-to-enter uni­code char­ac­ter, pre-canned text, or snip­pet of code. I also have a key­stroke that auto­mat­i­cally opens the selected file in Notepad++. I just had to paste a few lines into my con­fig­u­ra­tion file, and it inter­cepts the up-scroll and passes it to the sys­tem as a down-scroll and vice-versa. Specifically, it looks a lit­tle some­thing like this:

; Reverse mouse wheel to be more like OS X
    Send {WheelDown}

    Send {WheelUp}


Under Linux, I had to edit my Xorg con­fig­u­ra­tion file. I opened /etc/X11/xorg.conf in a text edi­tor and looked for the ‘Section “InputDevice“‘ block that cor­re­sponded to the mouse. I then had to change (or add, if it was not already there) the “Option ZAxisMapping” to the fol­low­ing:

Option "ZAxisMapping" "5 4"

This works with a mouse locally plugged into the sys­tem. I still have a prob­lem when the key­board and mouse are remotely con­nected via Synergy in that the mouse scolling does not get trans­lated. I am not sure whether that is the server (on Windows) bypass­ing the AutoHotkey and/or the client (on Linux) side-stepping the Xorg con­fig­u­ra­tion. It may be both, since if I con­fig­ure only one or the other with a reversed mouse, there is no change to the behav­ior. I’d expect a change if one was revers­ing it and the other was un-reversing.

So aside from via Synergy, it’s all work­ing and I am doing a good job at adapt­ing, though I fear the day that I sit down at a coworker’s com­puter and get tripped up because their mouse wheel works “back­wards.”

Posted in: Software

On Google Chrome (Mac, Linux, and nightly builds)

On Google Chrome

As you may have heard, Google is work­ing on a web browser called Chrome. There have been beta ver­sions out for Windows and Linux for some time. The Mac ver­sion is still play­ing catch-up. Because Firefox, the browser I now use, has appeared to get more slow and clunky over time, I fig­ured I would give Chrome a try. My main gripes with Firefox (on the Mac) are the speed, the startup/shutdown time, and the way it keeps eat­ing up mem­ory such that I have to restart it once a day (and con­se­quently wait for it to shut down, then start up again).

I started look­ing at the Mac ver­sion, but (at the time) it was a joke. There were no add-ons and no way to man­age your book­marks. You could book­mark all you wanted, and those links would show up in the menus, but if you ever wanted to edit, move, or remove book­marks, there was no way to do so. Because I had sim­i­lar per­for­mance issues with my Linux com­puter at work, I tried forc­ing myself to use it full-time there, first, because the Linux ver­sion was much more feature-complete.

Honestly, I have not looked back. In Linux (and pre­sum­ably Windows), it’s solid enough and fast enough that I find it a bet­ter expe­ri­ence than Firefox. It took me a lit­tle while to get the right mix of exten­sions, but this is what I’ve added to my Chrome instal­la­tion:

  • AdSweep for ad block­ing. I’m not con­vinced that this is the best ad blocker out there, as I have not looked at any oth­ers yet. All I know is that this was one of the first ad block­ers for Chrome and that it per­forms well enough that I have not needed to look for alter­na­tives.
  • FlashBlock is a click-to-play Flash blocker sim­i­lar to the one I use under Firefox. This pre­vents, for instance, YouTube videos and embed­ded music from auto-starting until I click on them.
  • Google Mail Checker for show­ing how many email mes­sages I have wait­ing in my inbox. I only use this under Linux. On the Mac, I have Google Notifier, which bet­ter inte­grates with the oper­at­ing sys­tem (com­plete with Growl noti­fi­ca­tion).
  • Google Reader Notifier is the same as above, but for Google Reader instead of Gmail.
  • Xmarks Bookmark Sync ensures that all of my book­marks across all of my com­put­ers are syn­chro­nized: the Linux box at work, the Mac lap­top I shut­tle between home and work, the Mac server at home, and the iPhone.  The exten­sion itself is a lit­tle bit prone to crash­ing on Chrome, but does not lose or man­gle data, and oth­er­wise works fine.
  • Amazon2Powells adds a link to pages on Amazon to the cor­re­spond­ing prod­uct at Powell’s. It lets me browse Amazon, hop on the bus, then buy locally.
  • A mod­i­fi­ca­tion of Google Reader Minimalistic (based on the orig­i­nal at UserScripts) that tweaks the fonts and lay­out on Google Reader a lit­tle more to my lik­ing than the orig­i­nal ver­sion.

I have been happy with Chrome on Linux for about three weeks now. I have been using a nightly Chrome build on the Mac for about two weeks and have been happy with that. In the inter­ven­ing time, Google release a new devel­oper pre­view of the Mac (adding exten­sion sup­port), but I am stick­ing with the nightly build for the fol­low­ing rea­sons:

  • Although exten­sions are in both the devel­oper pre­view and nightly snap­shot, book­mark edit­ing is not. Only the nightly has the Bookmark Manager menu item enabled.
  • The tab-to-search auto­matic locat­ing and using of search forms is unavail­able in the devel­oper pre­view, but works in the nightly. In Firefox, I had a spe­cial book­mark set up (basi­cally the Amazon search URL with “%s” in place of the search string, with the key­word “ama”) that let me type in “ama cheese” to find cheese on Amazon. With Chrome, after I have searched Amazon once, it remem­bers. All I need to do is type in “ama[TAB]cheese” to do the same thing with no pre­vi­ous spe­cial setup.

Oddly enough, one thing I’m miss­ing on Mac Chrome (and I feel weird for even say­ing this) is Java. In Firefox, I dis­able Java. I greatly dis­like Java-in-the-browser. As I men­tioned before, I use a light­weight note-taking appli­ca­tion called TiddlyWiki. It is, effec­tively, a self-modifying HTML file. In Firefox, it uses some fancy JavaScript (after ask­ing the user for per­mis­sion, of course) to write the file back to disk after mod­i­fi­ca­tions. Under other browsers (Safari, Opera, and Chrome), it uses some Java Applet trick­ery to do the same thing, since those browsers do not have the option to bust out of the JavaScript “sand­box” and write to disk with­out a bit of Java-assist. The prac­ti­cal upshot is that I can­not cur­rently use TiddlyWiki on the Mac ver­sion of Chrome because it blocks JavaScript file writes and doesn’t sup­port Java Applet file writes. This isn’t a huge deal-breaker at present — I open Firefox to edit notes — but is an annoy­ance that I hope will be fixed before I’d con­sider Chrome a com­plete prod­uct. And it works under Linux, so I doubt it is an inten­tional over­sight.

Overall, I would score Chrome as such:

  • Chrome under Linux: A–
  • Chrome developer-preview on Mac: D+
  • Chrome nightly (, 35604) on Mac: B
Posted in: Code Software

The Apple keyboard under Linux, revisited

Since I made my orig­i­nal blog post about using the Apple key­board under Linux two years ago, did a bit more refine­ment to the process. I don’t think I blogged about it, but did post a few instruc­tions and con­fig­u­ra­tion files to my “notes to myself” wiki at StackOverflow (no, not THAT StackOverflow).


To this day, I have never got the “fn” key work­ing as an insert key. Looking back on the com­ments on that post, nei­ther had any­one else. This morn­ing, Damien Ciabrini left a com­ment point­ing to his instruc­tions to get the fn key work­ing. Glancing over them, every­thing looks safe and sane in con­cept. It’s a bit more low-level than I would like to get right now, see­ing as this would be on my pri­mary work machine and we are in a bit of a crunch right now, so any down­time is too much down­time, but I applaud him in get­ting deep enough into the sys­tem code to fig­ure it out! A win for Open Source!

Posted in: Code

Apple Keyboard Fn

Apple Keyboard

After play­ing with the new Apple key­board for 5–10 min­utes at the Apple Store the other week (the sales folks were curi­ous as to why I was writ­ing what looked like code into a TextEdit win­dow), I decided to get one. The key spac­ing isn’t a big deal at all–they just squared the beveled edges on regular-sized keys–unless you hap­pen to be some­one who reg­u­larly hits the edges of keys (and if you do, you’re prob­a­bly not typ­ing cor­rectly any­way.) The “play” on the keys is the same as my lap­top, which is also nice–muscle mem­ory for only one “depth” of keys/keyboard. So I took it to work and hooked it up to my Linux box.

I pro­ceeded to set up key map­pings (swap­ping option and com­mand, for instance), until I hit a snag. The new Apple key­board has a “fn” key where the “Insert” usu­ally is on PC key­boards. It’s a “Help” key on Apple key­boards, so it’s nice that they killed off that key–it was too easy to acci­den­tally hit and spawn the helper appli­ca­tion.

Unfortunately, it looks like this key is hard-wired to other key com­bi­na­tions (the F1..F12 keys and their cor­re­spond­ing bright­ness, speaker, iTunes, dash­board, etc. func­tions.) It doesn’t look like it gen­er­ates a key­board scan code that can be trapped. This nor­mally wouldn’t be a big deal, but I use the “shift-insert” key com­bi­na­tion for one par­tic­u­lar oper­a­tion that I use con­stantly: past­ing text into a ter­mi­nal win­dow. Middle-click doesn’t always work (it depends on whether the text is in the X Windows paste buffer or the KDE paste buffer, so gives incon­sis­tent results), but shift-insert always does what I want... yet this key­board has no insert key!

Instead, I mapped that insert func­tion­al­ity to shift-delete (the key just under Fn), which does not make a lot of log­i­cal sense, but is a similar-enough and usable key com­bi­na­tion that it works well for me. I have been using this for a week or two now, and I like it a lot bet­ter than the Dell key­board I was using pre­vi­ously.

Posted in: Dear Diary Work


powerpc-linux-gcc -I../linux- -I../linux- -D__KERNEL__ -m32 -Wall -Wundef -Wstrict-prototypes -Wno-trigraphs -fno-strict-aliasing -fno-common -O2 -msoft-float -pipe -ffixed-r2 -mmultiple -mno-altivec -mstring -Wa,-maltivec -fomit-frame-pointer -g -fno-stack-protector -Wdeclaration-after-statement -Wno-pointer-sign -c -o cheese_main.o cheese_main.c
In file included from ../linux-,
from ../linux-,
from cheese_main.c:2:
../linux- warning: "CONFIG_HZ" is not defined
../linux- warning: "CONFIG_HZ" is not defined
../linux- warning: "CONFIG_HZ" is not defined
../linux- warning: "CONFIG_HZ" is not defined
../linux- warning: "CONFIG_HZ" is not defined
../linux- warning: "CONFIG_HZ" is not defined
../linux- warning: "CONFIG_HZ" is not defined
../linux- error: #error You lose.

Posted in: Code Work

Wherein we want to strangle the Linux kernel developers

The Linux ker­nel devel­op­ers, in their mys­te­ri­ous and infi­nite wis­dom, decided that between ver­sion 2.6.17 and 2.6.19 they would change the nam­ing scheme for access­ing Flash chips.  I am sure the change is doc­u­mented in a foot­note some­where, but it is not ter­ri­bly obvi­ous.  It is obvi­ous that things that for­merly worked are now bro­ken, though.  It took me 2 to 3 hours of debug­ging to track down the fact that:


...should now be called:


Once a ker­nel para­me­ter was changed slightly, every­thing mag­i­cally started work­ing again.  Why the change?  Who knows!?  Would you have noticed the dif­fer­ence if you were not look­ing for it?  I didn’t see it for a very long time, per­son­ally.

Posted in: Code Work

The Reinstallation Blues

Please note that all blog posts before 8 April 2007 were auto­mat­i­cally imported from LiveJournal.  To see the com­ments and any LiveJournal-specific extras such as polls and user icons, please find the source post­ing at http://brianenigma.livejournal.com/2007/03/

Bad idea: upgrad­ing your sys­tem from Fedora Core 4 to Fedora Core 6 using noth­ing but the online updater (“yum”). In the­ory, this is sup­posed to work. In prac­tice, not so much.

Bad idea: upgrad­ing your bro­ken sys­tem from Fedora Core 4-ish to Fedora Core 6 using scratched CDs and/or a drive that thinks per­fectly fine CDs are scratched and halts half-way through

Bad idea: Burning a brand new Fedora Core 6 DVD to upgrade your half-Fedora-Core 4-ish-half-Fedora-Core-6 sys­tem to a full Fedora Core 6 sys­tem.

Best idea: decid­ing long ago that the /home direc­tory should be on a dif­fer­ent drive, refor­mat­ting the root filesys­tem, leav­ing the /home filesys­tem intact, and installing a fresh Fedora Core 6, a fresh set of ARM and PPC cross-compilers, and what­ever else that might have got­ten wiped away. With this sce­nario, I didn’t even have to rein­stall my Firefox add-ons since they’re safely stored in my home direc­tory.

Of course, because of all of this, lost about a half-day of time on Friday and a half-day today, while wait­ing for per­cent­age bars and cross­ing my fin­gers and such. I can do some things on my lap­top, but there are still a num­ber of things I can’t do there because I don’t have trusted OS X ver­sions of the tools.

Posted in: Dear Diary Work

Singularities in Software Design

Please note that all blog posts before 8 April 2007 were auto­mat­i­cally imported from LiveJournal.  To see the com­ments and any LiveJournal-specific extras such as polls and user icons, please find the source post­ing at http://brianenigma.livejournal.com/2006/11/

Even with all of the work I’ve done with the Linux ker­nel and ker­nel dri­vers, there still are a num­ber of large mys­ter­ies float­ing around out there. First, there is a large amount of code. It’s dif­fi­cult to keep it all in your head, which is okay if you are work­ing with a sin­gle mod­ule but sucks when you have to con­sider the inter­ac­tions between mod­ules. Second, there are things that are not defined in read­ily appar­ent ways.

Lately, I have taken to think­ing about the whole thing as a black hole with an event hori­zon, to use astro­nom­i­cal terms. Standing on the periph­ery, you can only see so much of the code. Because of it’s extreme size and (acci­den­tal or inten­tional) obfus­cated syn­tax, you can just can’t see past a cer­tain point. It’s all a mys­tery and you know that some­thing is going on in that sin­gu­lar­ity and some of the results can reach you, you just can’t see the inner work­ings of the mech­a­nism. In the case of black holes, grav­ity is so strong that light can’t escape and with­out the light, you can’t see the core. In the case of the Linux ker­nel, the code is so obscured (by indi­rec­tion and macros) in places that you can’t see what’s going on.

For instance, there is a par­tic­u­lar func­tion used through­out the PCI sub­sys­tem (pci_read_config_word(), if I remem­ber cor­rectly.) Because of an error with the wiring on our PCI data bus, this func­tion was lock­ing up (block­ing on a data read) for us. As best as any of us can tell, this func­tion just isn’t defined. It’s called, yes. It’s just not defined in any C code, assem­bly code, or com­piler macros. Grepping it only returns pro­gram loca­tions that use it, not a def­i­n­i­tion. This one still remains a mys­tery, except not an impor­tant one once we fixed our data lines. Grepping doesn’t always work, as in the next exam­ple.

Another exam­ple is with the Flash chip dri­vers. There is effec­tively a vari­able that is defined and used by sev­eral pieces of code but is never assigned. Well, it is. But it isn’t. How about a pseudocode exam­ple?

#define SOME_CHIP_ADDRESS 0x0C000000
struct SomeStruct {
    uint16 value1;
    uint16 value2;
struct SomeStruct *myStruct; // a common structure used everywhere
myStruct = malloc(sizeof(struct SomeStruct));

// ...lots of lines of code removed...

if (myStruct->value2)
    printk("Dilithium crystal buffer delay set to %d\n", myStruct->value2);

And that’s pretty much it. Keep in mind that the above code is sim­pli­fied. The real code spans thou­sands of lines over sev­eral dozen files. myStruct.value2 is not defined or used any­where else. It is always a good and expected con­stant value. Where does it get set? Nobody knows. Grep for it in the entire ker­nel tree, and it’s just not assigned any­where. Well, that’s not exactly true. It does get set, but indi­rectly from the con­tents of a chip at a hard­ware address:
memcpy((uint8 *) myStruct, (uint8 *) SOME_CHIP_ADDRESS, 4);
Unfortunately, you can’t just grep for all instances of myS­truct because very nearly every line of code uses it. You might as well man­u­ally look through all of the code.

In ret­ro­spect, that mem­cpy is effi­cient and prob­a­bly the best way to set the vari­ables, but when you are look­ing for an assign­ment, it becomes a layer of obfus­ca­tion and indi­rec­tion that works just like grav­ity in a black hole. Get enough of those and you have a sin­gu­lar­ity with an event hori­zon. You assume an assign­ment is hap­pen­ing some­where. All empir­i­cal obser­va­tions show that it is. None of the tools you have at your dis­posal can “see” it, because it’s hap­pen­ing just out­side of your vis­i­bil­ity. You’re used to the 99% of the code in the rest of the ker­nel that uses the equal sign to put val­ues into struc­tures.

Fortunately, with the ker­nel and devel­op­ment tools, we can build bet­ter tele­scopes. With expe­ri­ence, we can learn that “grep -r myStruct . | grep memcpy” is a new way of search­ing for some of the more obscure stuff, now that we real­ize that mem­cpy on the con­tain­ing struc­ture is used. Maybe we’ll find a bet­ter tele­scope for look­ing at black holes, too.

Posted in: Code


Please note that all blog posts before 8 April 2007 were auto­mat­i­cally imported from LiveJournal.  To see the com­ments and any LiveJournal-specific extras such as polls and user icons, please find the source post­ing at http://brianenigma.livejournal.com/2006/10/

Do you know what’s cool?
* Digging into the imple­men­ta­tion behind the pthread_create() sys­tem call, and in the process learn­ing about all kinds of nifty OS-level func­tion­al­ity (includ­ing clone and futex), while locat­ing (and fix­ing) an 8K-per-new-connection/thread mem­ory leak.
* Adjustment screws
* Gingerbread tea (it smells a lot more strong and sweet than it tastes, which is good because I’m not big on super-sweet drinks)

Do you know what’s not cool?
* LiveJournal for­get­ting my cus­tom style and revert­ing back to a default one.
* My “post by SMS” ver­i­fi­ca­tion mes­sage not going through, even after an hour.
* The TX Fault cir­cuitry and logic embed­ded inside of Finisar SFP mod­ules and their lack of sup­port for the optional TX Fault detected pin.

Posted in: Dear Diary