Tag Archives: linux

someone_is_doing_something_nasty

ssh for Programmers

Overview

Way back when, just after the dot-com-bomb, I found myself work­ing for a media com­pany named Metapa. We made mp3s and 30 sec­ond pre­view clips from boxes of CDs that tech-clueless record com­pa­nies snail-mailed us. Metapa was staffed with a ton of Unix-heads and I received a great school­ing in the ways of the Secure Shell, ssh. You could not get to a server with­out ssh­ing into it. Our “VPN” was a server run­ning ssh, con­fig­ured to only accept key-based login. There was no way to log in with just a pass­word. You would con­nect and use ssh’s built-in port for­ward­ing to route more fancy con­nec­tions, as required. I got to know ssh incred­i­bly well.

At more recent jobs, peo­ple have used ssh to man­u­ally con­nect to boxes and to man­u­ally issue com­mands, but they never did any­thing more advanced. I never under­stood exactly why. I’m sure some of this is that these have been mixed Windows/Linux envi­ron­ments with heav­ier reliance on Windows, so the “cul­ture” of using ssh for every­thing just isn’t present. Part of it may also be lack of expe­ri­ence with ssh.

This arti­cle attempts to touch on the slightly more advanced meth­ods of work­ing with ssh. I am not going to get into port for­ward­ing, since that requires a much more advanced men­tal model and a good grasp of how to con­fig­ure the local and remote soft­ware. I also will not get into rsync, which is a com­mon tool that pig­gy­backs atop ssh to syn­chro­nize fold­ers. What I will dis­cus is a few of the more com­mon advanced uses:

  • copy­ing files
  • log­ging in with­out keys
  • aliases for server names
  • dif­fer­ing user­names
  • log­ging in to the cloud (com­bi­na­tion aliases, user­names, and alter­nate keys)
  • a quick fix for changed host iden­ti­ties

Copying Files

ssh has a com­pan­ion app named scp, or Secure Copy. It works exactly like the reg­u­lar Unix cp com­mand except the source and des­ti­na­tion can take ssh file paths in the form of username@machine:path/to/file.txt.

For instance, if I wanted to push my “.vimrc” file to a remote sys­tem named frodo, I could run any one of the fol­low­ing com­mands:

# a technically correct, but wordy way to do it
scp .vimrc briane@frodo:.vimrc

# destination file is the same name, so no need to specify
scp .vimrc briane@frodo:

# assumes my username is the same on frodo, so no need to specify
scp .vimrc frodo:

Note that when scp con­nects, it is in your home direc­tory. These des­ti­na­tions use rel­a­tive paths (because the paths do not start with a slash) so they are rel­a­tive to your home folder. If you wanted to copy a log file, you might use an absolute path such as:

# copy the Apache error log to the current folder
scp briane@frodo:/var/log/apache/error.log .

Logging in Without Keys

By default ssh (as well as scp) uses your Unix pass­word to access the remote sys­tem. This is all well and good, but it also offers another way to iden­tify your­self that allows for a bit more con­ve­nience and a lot more secu­rity. This method is dual-key cryp­tog­ra­phy. Put sim­ply, this is a whole bunch of math shared between two files, a “pub­lic” key and a “pri­vate” key. The pri­vate key is, effec­tively, a ludi­crously large num­ber — like a pass­word — that you share with nobody. The pub­lic key is math­e­mat­i­cally derived from the pri­vate key and can be used to check the authen­tic­ity of the pri­vate key. You can share the pub­lic key far and wide. When using keys with ssh, you store a copy of your pub­lic key on the remote server. Every time you log in to it, your machine and the remote machine do a lit­tle dance involv­ing the pub­lic and pri­vate keys, and you’re instantly logged in — no pass­word required. It effec­tively shifts the authen­ti­ca­tion mech­a­nism from “what you know” (a pass­word) to “what you have” (a data file with that long num­ber).

In con­cept, this is all well and good, but it is some­times a lit­tle painful and tedious to set up by hand. You have to ensure the cor­rect file is in the cor­rect place. The per­mis­sions have to be per­fect or the ssh server will refuse your login. After all, if your pub­lic key is writable by other peo­ple, then some­one with access to the sys­tem can over­write it with their pub­lic key and log in as you, with your full per­mis­sions and cre­den­tials. Fortunately, if there is one thing that com­put­ers are good at automat­ing, it is sim­ple, annoy­ing, some­times painful tasks.

First, you will need to cre­ate a key (if you have not done so already). It’s a sim­ple com­mand:

ssh-keygen

Just keep hit­ting Return to answer all the ques­tions with their defaults, includ­ing an empty pass­word. Next, you can use a script like the fol­low­ing, named colonize.sh, that will send your key to a remote machine over a reg­u­lar ssh-with-password con­nec­tion and set up the per­mis­sions.

#!/bin/bash
if [ -z "$1" ]; then
    echo "Function: copies ssh public keys to remote machines for passwordless login"
    echo "Usage: colonize.sh <username>@<servername or ip address>"
    exit 1
fi
KEY=id_rsa.pub
STORE=authorized_keys
CONTENT=`cat ~/.ssh/$KEY`
echo "Attaching key to authorized_keys file"
ssh $1 "mkdir -p .ssh && chmod 700 .ssh && touch .ssh/$STORE \
    && chmod 644 .ssh/$STORE \
    && echo '$CONTENT' >> .ssh/$STORE"
echo "Complete!  You should be able to log in without a password now!"

Now if you want to col­o­nize a remote machine with your ssh keys, you’d just do some­thing like:

colonize.sh briane@bilbo

Enter your pass­word (for the last time ever!) and next time you ssh or scp with that machine, it will use your keys instead of your pass­word.

Server Aliases

There is a con­fig file in your dot-ssh folder named sim­ply .ssh/config. This holds a wealth of options for cus­tom per-server set­tings. Long ago, I had a vir­tual pri­vate server with Dreamhost (I’ve since moved to Linode for reli­a­bil­ity rea­sons; if you’d like to try it out, feel free to use my refer­ral URL). This server had the ugly and dif­fi­cult to remem­ber name of “ps24493.dreamhost.com”. I ssh and scp all the time, but had dif­fi­culty remem­ber­ing that exact server num­ber. Fortunately, there is a bet­ter way.

Your .ssh/config file lets you define a Host (what you type to con­nect) to a HostName (the actual server name). For exam­ple, my con­fig had the fol­low­ing two lines:

Host dreamhost
HostName ps24493.dreamhost.com

This means that “dreamhost” ends up as short­hand for the more annoy­ing “ps24493.dreamhost.com”. If I want to log in, I’d just “ssh dreamhost”. If I wanted to copy a file, I could “scp index.html dreamhost:/var/www/my_website”. It made it much eas­ier to access the server.

Differing Usernames

I actu­ally had one more line in that part of con­fig file. The account name I use on most of my local machines is sim­ply “brian”. On that vir­tual pri­vate server, it was the much more ver­bose but globally-unique “brianenigma”. So tech­ni­cally, I could not con­nect with a sim­ple “ssh dreamhost” but “ssh brianenigma@dreamhost”. But there is another line that can be added to the .ssh/config to allow the for­mer vari­ant with­out a user­name.

You can define a default user­name for a spe­cific sys­tem by putting a “User” entry in the “Host” sec­tion. For instance:

Host dreamhost
HostName ps24493.dreamhost.com
User brianenigma

This, then, allows me to type in “ssh dreamhost” instead of the much more wordy “ssh brianenigma@ps24493.dreamhost.com”.

Logging in to the Cloud

Amazon’s Web Services is an inter­est­ing beast. You get a machine up in the cloud that you can log into. It typ­i­cally has an ugly name (which can be nick­named — see two sec­tions back) and may have a dif­fer­ent user­name than the one you use on your local machine (which can also be rede­fined — see the pre­vi­ous sec­tion). It throws in yet another wrin­kle. Password-based login is for­bid­den and it assigns you a pub­lic and pri­vate key pair. Because it assigns you a key, it is obvi­ously going to be dif­fer­ent than any key you have pre­vi­ously gen­er­ated for all your other con­nec­tions.

There is an ugly way to spec­ify the key­store you down­loaded from Amazon on the com­mand line, but a far more sim­ply way is to define it in your .ssh/config, such as this entry that com­bines all of the above and adds the key­store:

Host cloud
HostName ec2-256-301-297-271.us-west-2.compute.amazonaws.com
User administrator
IdentityFile ~/.ssh/my_amazon_key.pem

Now you can log in to your AWS account by just typ­ing “ssh cloud”. ssh will expand that out to the very ugly equiv­a­lent:

ssh -i ~/.ssh/my_amazon_key.pem administrator@ec2-256-301-297-271.us-west-2.compute.amazonaws.com

Remote Host Identification has Changed

I work in a lab envi­ron­ment with lots of servers that are fre­quently refor­mat­ted. This means that the servers retain their name and IP address, but the ssh fin­ger­prints that uniquely iden­tify them (to help thwart third par­ties from pos­ing as the server and steal­ing your pass­words) get wiped out. In this case, if I attempt to log in, I get a big scary mes­sage like the fol­low­ing:

@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@
@    WARNING: REMOTE HOST IDENTIFICATION HAS CHANGED!     @
@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@
IT IS POSSIBLE THAT SOMEONE IS DOING SOMETHING NASTY!
Someone could be eavesdropping on you right now (man-in-the-middle attack)!
It is also possible that the RSA host key has just been changed.
The fingerprint for the RSA key sent by the remote host is
f0:d1:77:17:9a:55:de:ad:be:ef:00:00:08:67:53:09.
Please contact your system administrator.
Add correct host key in /home/briane/.ssh/known_hosts to get rid of this message.
Offending key in /home/briane/.ssh/known_hosts: 6
Permission denied (publickey,password).

The fix here is to open up .ssh/known_hosts and delete line 6. Technically, delet­ing the entire file also works (and I have seen plenty of peo­ple do that as a “fix”), but that has the side-effect of forc­ing you to con­firm, all over again, every remote host you con­nect to as a new host. So: “delete line 6” is sim­ple enough to do, but also annoy­ing. We can auto­mate that. The fol­low­ing script, called fixssh.sh, will retain the past nine back­ups of your known_hosts file and delete the offend­ing line. In this case, you’d run it with “fixssh.sh 6”.

#!/bin/bash

if [ -z "$1" ]; then
    echo "Put line number of your ~/.ssh/known_hosts file for deletion on the"
    echo "command line."
    exit 1
fi

cd ~/.ssh/
rm -rf known_hosts.9
for NUM in 8 7 6 5 4 3 2 1 ; do 
    ((NUMPLUS=$NUM + 1))
    if [ -d known_hosts.$NUM ]; then 
        mv known_hosts.$NUM known_hosts.$NUMPLUS 
    fi 
done
mv known_hosts known_hosts.1
cat known_hosts.1 | sed "$1 d" > known_hosts
chmod 644 known_hosts

Conclusion

I hope these tips help you become more pro­fi­cient at ssh usage. Once you have these down, you might search on your own for the more advanced top­ics like how to use rsync, to keep mas­sive amounts of files in sync, or using ssh port for­ward­ing.

Posted in: Code Software Work

os_x_lion_natural_gestures

Natural” scrolling across operating systems

Natural” Scrolling Overview

I am begin­ning to get used to the reversed scrolling behav­ior in the new OS X. For those that have not heard about it, this was one of the most con­tro­ver­sial changes in the new oper­at­ing sys­tem. They reversed the direc­tion of the action that is per­formed by spin­ning the mouse’s scroll wheel or using two fin­gers to scroll on a touch­pad. There have been a lot of heated debates on both sides of the fence.

If you think about it, the scroll wheel action is fairly arbi­trary. It was:

  • scroll the wheel down, the slider in the win­dow that marks your posi­tion moves down­ward (to match your fin­ger move­ment), the text of your doc­u­ment slides up, you reveal more text at the bot­tom edge of the win­dow.

It is now:

  • scroll the wheel down, the slider in the win­dow that marks your posi­tion moves upward, the text of your doc­u­ment slides down (to match your fin­ger move­ment), you reveal more text at the top edge of the win­dow.

In other words, the move­ment switched from manip­u­lat­ing the win­dow slider to manip­u­lat­ing the doc­u­ment itself. You might argue that it is a lit­tle more direct now — instead of manip­u­lat­ing the slider that manip­u­lates your posi­tion in the doc­u­ment, you are directly manip­u­lat­ing the doc­u­ment. You might argue that, at least for touch-based track­pad sys­tems (not mice, per se), the action is more iPhone-like. You might also argue that as a soci­ety we now have a cul­ture muscle-memory of how the scroll wheel works and chang­ing it would be akin to swap­ping the posi­tion of the gas and brake ped­als on a car or (less dan­ger­ous) build­ing ana­log clocks whose hands spun counter-clockwise.

I fig­ured I would at least give it a try, but had to jump in with both feet. That meant leav­ing the new default OS X reversed (“nat­ural”) scrolling, but that also meant hack­ing my Windows and Linux boxes to behave sim­i­larly — oth­er­wise, I found myself get­ting stuck in a limbo of not know­ing which direc­tion the scroller worked at any given time, hes­i­tat­ing with a lit­tle test nudge to ensure it was going to work the way I thought.

Windows

For me, the Windows hack was pretty easy. I already run AutoHotkey for a few key­board short­cuts. AutoHotkey is an app that inter­cepts key­presses and mouse clicks and lets you trans­form them to other actions. I have a few macros set up (sim­i­lar to TextExpander on the Mac) that lets me type an abbre­vi­a­tion, which gets expanded to a hard-to-enter uni­code char­ac­ter, pre-canned text, or snip­pet of code. I also have a key­stroke that auto­mat­i­cally opens the selected file in Notepad++. I just had to paste a few lines into my con­fig­u­ra­tion file, and it inter­cepts the up-scroll and passes it to the sys­tem as a down-scroll and vice-versa. Specifically, it looks a lit­tle some­thing like this:

; Reverse mouse wheel to be more like OS X
WheelUp::
    Send {WheelDown}
Return

WheelDown::
    Send {WheelUp}
Return

Linux

Under Linux, I had to edit my Xorg con­fig­u­ra­tion file. I opened /etc/X11/xorg.conf in a text edi­tor and looked for the ‘Section “InputDevice“‘ block that cor­re­sponded to the mouse. I then had to change (or add, if it was not already there) the “Option ZAxisMapping” to the fol­low­ing:

Option "ZAxisMapping" "5 4"

This works with a mouse locally plugged into the sys­tem. I still have a prob­lem when the key­board and mouse are remotely con­nected via Synergy in that the mouse scolling does not get trans­lated. I am not sure whether that is the server (on Windows) bypass­ing the AutoHotkey and/or the client (on Linux) side-stepping the Xorg con­fig­u­ra­tion. It may be both, since if I con­fig­ure only one or the other with a reversed mouse, there is no change to the behav­ior. I’d expect a change if one was revers­ing it and the other was un-reversing.

So aside from via Synergy, it’s all work­ing and I am doing a good job at adapt­ing, though I fear the day that I sit down at a coworker’s com­puter and get tripped up because their mouse wheel works “back­wards.”

Posted in: Software

On Google Chrome (Mac, Linux, and nightly builds)

On Google Chrome

As you may have heard, Google is work­ing on a web browser called Chrome. There have been beta ver­sions out for Windows and Linux for some time. The Mac ver­sion is still play­ing catch-up. Because Firefox, the browser I now use, has appeared to get more slow and clunky over time, I fig­ured I would give Chrome a try. My main gripes with Firefox (on the Mac) are the speed, the startup/shutdown time, and the way it keeps eat­ing up mem­ory such that I have to restart it once a day (and con­se­quently wait for it to shut down, then start up again).

I started look­ing at the Mac ver­sion, but (at the time) it was a joke. There were no add-ons and no way to man­age your book­marks. You could book­mark all you wanted, and those links would show up in the menus, but if you ever wanted to edit, move, or remove book­marks, there was no way to do so. Because I had sim­i­lar per­for­mance issues with my Linux com­puter at work, I tried forc­ing myself to use it full-time there, first, because the Linux ver­sion was much more feature-complete.

Honestly, I have not looked back. In Linux (and pre­sum­ably Windows), it’s solid enough and fast enough that I find it a bet­ter expe­ri­ence than Firefox. It took me a lit­tle while to get the right mix of exten­sions, but this is what I’ve added to my Chrome instal­la­tion:

  • AdSweep for ad block­ing. I’m not con­vinced that this is the best ad blocker out there, as I have not looked at any oth­ers yet. All I know is that this was one of the first ad block­ers for Chrome and that it per­forms well enough that I have not needed to look for alter­na­tives.
  • FlashBlock is a click-to-play Flash blocker sim­i­lar to the one I use under Firefox. This pre­vents, for instance, YouTube videos and embed­ded music from auto-starting until I click on them.
  • Google Mail Checker for show­ing how many email mes­sages I have wait­ing in my inbox. I only use this under Linux. On the Mac, I have Google Notifier, which bet­ter inte­grates with the oper­at­ing sys­tem (com­plete with Growl noti­fi­ca­tion).
  • Google Reader Notifier is the same as above, but for Google Reader instead of Gmail.
  • Xmarks Bookmark Sync ensures that all of my book­marks across all of my com­put­ers are syn­chro­nized: the Linux box at work, the Mac lap­top I shut­tle between home and work, the Mac server at home, and the iPhone.  The exten­sion itself is a lit­tle bit prone to crash­ing on Chrome, but does not lose or man­gle data, and oth­er­wise works fine.
  • Amazon2Powells adds a link to pages on Amazon to the cor­re­spond­ing prod­uct at Powell’s. It lets me browse Amazon, hop on the bus, then buy locally.
  • A mod­i­fi­ca­tion of Google Reader Minimalistic (based on the orig­i­nal at UserScripts) that tweaks the fonts and lay­out on Google Reader a lit­tle more to my lik­ing than the orig­i­nal ver­sion.

I have been happy with Chrome on Linux for about three weeks now. I have been using a nightly Chrome build on the Mac for about two weeks and have been happy with that. In the inter­ven­ing time, Google release a new devel­oper pre­view of the Mac (adding exten­sion sup­port), but I am stick­ing with the nightly build for the fol­low­ing rea­sons:

  • Although exten­sions are in both the devel­oper pre­view and nightly snap­shot, book­mark edit­ing is not. Only the nightly has the Bookmark Manager menu item enabled.
  • The tab-to-search auto­matic locat­ing and using of search forms is unavail­able in the devel­oper pre­view, but works in the nightly. In Firefox, I had a spe­cial book­mark set up (basi­cally the Amazon search URL with “%s” in place of the search string, with the key­word “ama”) that let me type in “ama cheese” to find cheese on Amazon. With Chrome, after I have searched Amazon once, it remem­bers. All I need to do is type in “ama[TAB]cheese” to do the same thing with no pre­vi­ous spe­cial setup.

Oddly enough, one thing I’m miss­ing on Mac Chrome (and I feel weird for even say­ing this) is Java. In Firefox, I dis­able Java. I greatly dis­like Java-in-the-browser. As I men­tioned before, I use a light­weight note-taking appli­ca­tion called TiddlyWiki. It is, effec­tively, a self-modifying HTML file. In Firefox, it uses some fancy JavaScript (after ask­ing the user for per­mis­sion, of course) to write the file back to disk after mod­i­fi­ca­tions. Under other browsers (Safari, Opera, and Chrome), it uses some Java Applet trick­ery to do the same thing, since those browsers do not have the option to bust out of the JavaScript “sand­box” and write to disk with­out a bit of Java-assist. The prac­ti­cal upshot is that I can­not cur­rently use TiddlyWiki on the Mac ver­sion of Chrome because it blocks JavaScript file writes and doesn’t sup­port Java Applet file writes. This isn’t a huge deal-breaker at present — I open Firefox to edit notes — but is an annoy­ance that I hope will be fixed before I’d con­sider Chrome a com­plete prod­uct. And it works under Linux, so I doubt it is an inten­tional over­sight.

Overall, I would score Chrome as such:

  • Chrome under Linux: A–
  • Chrome developer-preview on Mac: D+
  • Chrome nightly (4.0.291.0, 35604) on Mac: B
Posted in: Code Software

The Apple keyboard under Linux, revisited

Since I made my orig­i­nal blog post about using the Apple key­board under Linux two years ago, did a bit more refine­ment to the process. I don’t think I blogged about it, but did post a few instruc­tions and con­fig­u­ra­tion files to my “notes to myself” wiki at StackOverflow (no, not THAT StackOverflow).

Apple_Keyboard_Fn

To this day, I have never got the “fn” key work­ing as an insert key. Looking back on the com­ments on that post, nei­ther had any­one else. This morn­ing, Damien Ciabrini left a com­ment point­ing to his instruc­tions to get the fn key work­ing. Glancing over them, every­thing looks safe and sane in con­cept. It’s a bit more low-level than I would like to get right now, see­ing as this would be on my pri­mary work machine and we are in a bit of a crunch right now, so any down­time is too much down­time, but I applaud him in get­ting deep enough into the sys­tem code to fig­ure it out! A win for Open Source!

Posted in: Code

Apple Keyboard Fn

Apple Keyboard

After play­ing with the new Apple key­board for 5–10 min­utes at the Apple Store the other week (the sales folks were curi­ous as to why I was writ­ing what looked like code into a TextEdit win­dow), I decided to get one. The key spac­ing isn’t a big deal at all–they just squared the beveled edges on regular-sized keys–unless you hap­pen to be some­one who reg­u­larly hits the edges of keys (and if you do, you’re prob­a­bly not typ­ing cor­rectly any­way.) The “play” on the keys is the same as my lap­top, which is also nice–muscle mem­ory for only one “depth” of keys/keyboard. So I took it to work and hooked it up to my Linux box.

I pro­ceeded to set up key map­pings (swap­ping option and com­mand, for instance), until I hit a snag. The new Apple key­board has a “fn” key where the “Insert” usu­ally is on PC key­boards. It’s a “Help” key on Apple key­boards, so it’s nice that they killed off that key–it was too easy to acci­den­tally hit and spawn the helper appli­ca­tion.

Unfortunately, it looks like this key is hard-wired to other key com­bi­na­tions (the F1..F12 keys and their cor­re­spond­ing bright­ness, speaker, iTunes, dash­board, etc. func­tions.) It doesn’t look like it gen­er­ates a key­board scan code that can be trapped. This nor­mally wouldn’t be a big deal, but I use the “shift-insert” key com­bi­na­tion for one par­tic­u­lar oper­a­tion that I use con­stantly: past­ing text into a ter­mi­nal win­dow. Middle-click doesn’t always work (it depends on whether the text is in the X Windows paste buffer or the KDE paste buffer, so gives incon­sis­tent results), but shift-insert always does what I want... yet this key­board has no insert key!

Instead, I mapped that insert func­tion­al­ity to shift-delete (the key just under Fn), which does not make a lot of log­i­cal sense, but is a similar-enough and usable key com­bi­na­tion that it works well for me. I have been using this for a week or two now, and I like it a lot bet­ter than the Dell key­board I was using pre­vi­ously.

Posted in: Dear Diary Work

A LOSER IS YOU!

powerpc-linux-gcc -I../linux-2.6.19.1/arch/ppc/include -I../linux-2.6.19.1/include -D__KERNEL__ -m32 -Wall -Wundef -Wstrict-prototypes -Wno-trigraphs -fno-strict-aliasing -fno-common -O2 -msoft-float -pipe -ffixed-r2 -mmultiple -mno-altivec -mstring -Wa,-maltivec -fomit-frame-pointer -g -fno-stack-protector -Wdeclaration-after-statement -Wno-pointer-sign -c -o cheese_main.o cheese_main.c
In file included from ../linux-2.6.19.1/include/linux/sched.h:51,
from ../linux-2.6.19.1/include/linux/module.h:9,
from cheese_main.c:2:
../linux-2.6.19.1/include/linux/jiffies.h:18:5: warning: "CONFIG_HZ" is not defined
../linux-2.6.19.1/include/linux/jiffies.h:20:7: warning: "CONFIG_HZ" is not defined
../linux-2.6.19.1/include/linux/jiffies.h:22:7: warning: "CONFIG_HZ" is not defined
../linux-2.6.19.1/include/linux/jiffies.h:24:7: warning: "CONFIG_HZ" is not defined
../linux-2.6.19.1/include/linux/jiffies.h:26:7: warning: "CONFIG_HZ" is not defined
../linux-2.6.19.1/include/linux/jiffies.h:28:7: warning: "CONFIG_HZ" is not defined
../linux-2.6.19.1/include/linux/jiffies.h:30:7: warning: "CONFIG_HZ" is not defined
../linux-2.6.19.1/include/linux/jiffies.h:33:3: error: #error You lose.

Posted in: Code Work

Wherein we want to strangle the Linux kernel developers

The Linux ker­nel devel­op­ers, in their mys­te­ri­ous and infi­nite wis­dom, decided that between ver­sion 2.6.17 and 2.6.19 they would change the nam­ing scheme for access­ing Flash chips.  I am sure the change is doc­u­mented in a foot­note some­where, but it is not ter­ri­bly obvi­ous.  It is obvi­ous that things that for­merly worked are now bro­ken, though.  It took me 2 to 3 hours of debug­ging to track down the fact that:

physmap_flash

...should now be called:

physmap-flash

Once a ker­nel para­me­ter was changed slightly, every­thing mag­i­cally started work­ing again.  Why the change?  Who knows!?  Would you have noticed the dif­fer­ence if you were not look­ing for it?  I didn’t see it for a very long time, per­son­ally.

Posted in: Code Work

The Reinstallation Blues

Please note that all blog posts before 8 April 2007 were auto­mat­i­cally imported from LiveJournal.  To see the com­ments and any LiveJournal-specific extras such as polls and user icons, please find the source post­ing at http://brianenigma.livejournal.com/2007/03/

Bad idea: upgrad­ing your sys­tem from Fedora Core 4 to Fedora Core 6 using noth­ing but the online updater (“yum”). In the­ory, this is sup­posed to work. In prac­tice, not so much.

Bad idea: upgrad­ing your bro­ken sys­tem from Fedora Core 4-ish to Fedora Core 6 using scratched CDs and/or a drive that thinks per­fectly fine CDs are scratched and halts half-way through

Bad idea: Burning a brand new Fedora Core 6 DVD to upgrade your half-Fedora-Core 4-ish-half-Fedora-Core-6 sys­tem to a full Fedora Core 6 sys­tem.

Best idea: decid­ing long ago that the /home direc­tory should be on a dif­fer­ent drive, refor­mat­ting the root filesys­tem, leav­ing the /home filesys­tem intact, and installing a fresh Fedora Core 6, a fresh set of ARM and PPC cross-compilers, and what­ever else that might have got­ten wiped away. With this sce­nario, I didn’t even have to rein­stall my Firefox add-ons since they’re safely stored in my home direc­tory.

Of course, because of all of this, lost about a half-day of time on Friday and a half-day today, while wait­ing for per­cent­age bars and cross­ing my fin­gers and such. I can do some things on my lap­top, but there are still a num­ber of things I can’t do there because I don’t have trusted OS X ver­sions of the tools.

Posted in: Dear Diary Work

Singularities in Software Design

Please note that all blog posts before 8 April 2007 were auto­mat­i­cally imported from LiveJournal.  To see the com­ments and any LiveJournal-specific extras such as polls and user icons, please find the source post­ing at http://brianenigma.livejournal.com/2006/11/

Even with all of the work I’ve done with the Linux ker­nel and ker­nel dri­vers, there still are a num­ber of large mys­ter­ies float­ing around out there. First, there is a large amount of code. It’s dif­fi­cult to keep it all in your head, which is okay if you are work­ing with a sin­gle mod­ule but sucks when you have to con­sider the inter­ac­tions between mod­ules. Second, there are things that are not defined in read­ily appar­ent ways.

Lately, I have taken to think­ing about the whole thing as a black hole with an event hori­zon, to use astro­nom­i­cal terms. Standing on the periph­ery, you can only see so much of the code. Because of it’s extreme size and (acci­den­tal or inten­tional) obfus­cated syn­tax, you can just can’t see past a cer­tain point. It’s all a mys­tery and you know that some­thing is going on in that sin­gu­lar­ity and some of the results can reach you, you just can’t see the inner work­ings of the mech­a­nism. In the case of black holes, grav­ity is so strong that light can’t escape and with­out the light, you can’t see the core. In the case of the Linux ker­nel, the code is so obscured (by indi­rec­tion and macros) in places that you can’t see what’s going on.

For instance, there is a par­tic­u­lar func­tion used through­out the PCI sub­sys­tem (pci_read_config_word(), if I remem­ber cor­rectly.) Because of an error with the wiring on our PCI data bus, this func­tion was lock­ing up (block­ing on a data read) for us. As best as any of us can tell, this func­tion just isn’t defined. It’s called, yes. It’s just not defined in any C code, assem­bly code, or com­piler macros. Grepping it only returns pro­gram loca­tions that use it, not a def­i­n­i­tion. This one still remains a mys­tery, except not an impor­tant one once we fixed our data lines. Grepping doesn’t always work, as in the next exam­ple.

Another exam­ple is with the Flash chip dri­vers. There is effec­tively a vari­able that is defined and used by sev­eral pieces of code but is never assigned. Well, it is. But it isn’t. How about a pseudocode exam­ple?

#define SOME_CHIP_ADDRESS 0x0C000000
struct SomeStruct {
    uint16 value1;
    uint16 value2;
};
struct SomeStruct *myStruct; // a common structure used everywhere
myStruct = malloc(sizeof(struct SomeStruct));

// ...lots of lines of code removed...

if (myStruct->value2)
    printk("Dilithium crystal buffer delay set to %d\n", myStruct->value2);

And that’s pretty much it. Keep in mind that the above code is sim­pli­fied. The real code spans thou­sands of lines over sev­eral dozen files. myStruct.value2 is not defined or used any­where else. It is always a good and expected con­stant value. Where does it get set? Nobody knows. Grep for it in the entire ker­nel tree, and it’s just not assigned any­where. Well, that’s not exactly true. It does get set, but indi­rectly from the con­tents of a chip at a hard­ware address:
memcpy((uint8 *) myStruct, (uint8 *) SOME_CHIP_ADDRESS, 4);
Unfortunately, you can’t just grep for all instances of myS­truct because very nearly every line of code uses it. You might as well man­u­ally look through all of the code.

In ret­ro­spect, that mem­cpy is effi­cient and prob­a­bly the best way to set the vari­ables, but when you are look­ing for an assign­ment, it becomes a layer of obfus­ca­tion and indi­rec­tion that works just like grav­ity in a black hole. Get enough of those and you have a sin­gu­lar­ity with an event hori­zon. You assume an assign­ment is hap­pen­ing some­where. All empir­i­cal obser­va­tions show that it is. None of the tools you have at your dis­posal can “see” it, because it’s hap­pen­ing just out­side of your vis­i­bil­ity. You’re used to the 99% of the code in the rest of the ker­nel that uses the equal sign to put val­ues into struc­tures.

Fortunately, with the ker­nel and devel­op­ment tools, we can build bet­ter tele­scopes. With expe­ri­ence, we can learn that “grep -r myStruct . | grep memcpy” is a new way of search­ing for some of the more obscure stuff, now that we real­ize that mem­cpy on the con­tain­ing struc­ture is used. Maybe we’ll find a bet­ter tele­scope for look­ing at black holes, too.

Posted in: Code

Cool/Uncool

Please note that all blog posts before 8 April 2007 were auto­mat­i­cally imported from LiveJournal.  To see the com­ments and any LiveJournal-specific extras such as polls and user icons, please find the source post­ing at http://brianenigma.livejournal.com/2006/10/

Do you know what’s cool?
* Digging into the imple­men­ta­tion behind the pthread_create() sys­tem call, and in the process learn­ing about all kinds of nifty OS-level func­tion­al­ity (includ­ing clone and futex), while locat­ing (and fix­ing) an 8K-per-new-connection/thread mem­ory leak.
* Adjustment screws
* Gingerbread tea (it smells a lot more strong and sweet than it tastes, which is good because I’m not big on super-sweet drinks)

Do you know what’s not cool?
* LiveJournal for­get­ting my cus­tom style and revert­ing back to a default one.
* My “post by SMS” ver­i­fi­ca­tion mes­sage not going through, even after an hour.
* The TX Fault cir­cuitry and logic embed­ded inside of Finisar SFP mod­ules and their lack of sup­port for the optional TX Fault detected pin.

Posted in: Dear Diary