Migrating from Kindle to iPad: An Illustrated DRM Primer

by Brian Enigma on April 5, 2010 8:22pm

in Books,Gadgets

Over the week­end I made a tweet that sev­eral peo­ple asked about:

Basi­cally, they wanted to know how to con­vert DRM pro­tected Kin­dle books over to ePub books that will work on the iPad. 

I love the fact that Ama­zon was able to release a Kin­dle app for the iPad on the very first day, but in a side-by-side com­par­i­son, I found Apple’s iBooks to be a lit­tle more use­ful to me.  It’s a seri­ous set of trade-offs.  On the one hand, the Kin­dle app syn­chro­nizes with the cloud and remem­bers what page I last viewed.  If I pick up my actual Kin­dle or use the Kin­dle app on the iPhone, then I can resume exactly where I was.  There is no equiv­a­lent in iBooks.  On the other hand, the Kin­dle app has no dic­tio­nary, whereas the iBooks dic­tio­nary is stream­lined and non-obtrusive.  Weigh­ing the two fea­tures, the dic­tio­nary is more impor­tant to me.  I can man­u­ally fig­ure out where I am in a book if I switch devices, but I often run into words I do not know — for instance “blue­stock­ing” (a deroga­tory term to describe an intel­li­gent or lit­er­ary woman) — mean­ing I have to jump out of the Kin­dle app, load up a dic­tio­nary app, wait for it to fin­ish load­ing, type in the word, then switch back to the Kin­dle app.  It is just so much eas­ier to tap the word and get a def­i­n­i­tion in a pop-up bubble.

Given this, I started con­vert­ing my Kin­dle books to ePub for­mat.  This is a two step process.  For the folks that just want to do it and fig­ure out the details on their own, those steps are:

  • Use the “dedrm” Python script to strip the DRM from the Kin­dle *.azw file
  • Use Cal­i­bre to con­vert from azw (it’s really MOBI) to ePub

Strip the DRM from the Kin­dle file

This step is the most dif­fi­cult and takes a lit­tle Ter­mi­nal mojo.  The fol­low­ing direc­tions assume you are run­ning on a Mac (I am sure Python exists for Win­dows, but you’re on your own if you have a Microsoft box) and that you can do some basic nav­i­ga­tion around the Ter­mi­nal com­mand prompt.

The Kin­dle AZW files are really just MOBI (a stan­dard eBook for­mat) with some spe­cial DRM applied.  The DRM is a sym­met­ric cipher, mean­ing it needs a key­word to encrypt and the same key­word to decrypt.  That key­word is just a reshuf­fling of your Kindle’s ser­ial number.

First, you will need your Kin­dle, with the pur­chased book down­loaded onto it.  Next, you will need a set of Python scripts that go by the name dedrm (includ­ing kindlepid.py, mobidedrm.py, and monidedrm2.py).  I am not a lawyer, so I do not know if I can pro­vide a direct down­load link here.  I assume this stuff is cov­ered under Fair Use, but I am going to go the con­ser­v­a­tive route and just tell you to do a Google search.

Plug in your Kin­dle, and copy the AZW file out of its /Documents folder and into the same folder as your Python scripts.  To get the encryp­tion key that Ama­zon uses to encrypt your eBooks, cd to the direc­tory con­tain­ing the Python scripts and use this com­mand, sub­sti­tut­ing the B12345678 with your actual Kin­dle ser­ial number:

python ./kindlepid.py B12345678

This should come back with a result along the lines of:

Mobipocked PID for Kindle serial# B12345678 is 1Q4Y2VZ*RE

Save that mix of num­bers, let­ters, and aster­isk.  That’s the key that Ama­zon uses to encrypt all eBooks for your spe­cific Kindle.

Next, use that key to decrypt the book.  You’re actu­ally going to have to do this twice.  One will work and one will not (and it will be dif­fi­cult to know which one worked until you load it in a viewer in the next step).  You will need to run the dedrm v0.01 and dedrm 0.02 com­mands.  They dif­fer slightly in the com­pres­sion method used, but the wrong com­pres­sion adds some garbage and bad meta­data to your book.  Be sure to replace the 1Q4Y2VZ*RE with your key derived from the above com­mand.  Also, leave those single-quotes in there.

python ./mobidedrm.py mybook.azw mybook1.mobi '1Q4Y2VZ*RE'

python ./mobidedrm2.py mybook.azw mybook2.mobi '1Q4Y2VZ*RE'

Either mybook1.mobi or mybook2.mobi should be the cor­rect decrypted ver­sion of your eBook.  The next step is to fig­ure out which one is the good one and to con­vert it to ePub.

Con­vert AZW/MOBI to ePub

Import the first MOBI file into Cal­i­bre and use the “View” icon in the tool­bar to see it in the viewer.  Ver­ify that this is a non-broken document.

You first want to bring up the table of con­tents by using the icon in the tool­bar on the left.  Click around in the table of con­tents to ver­ify that chap­ter mark­ers really do line up with the chap­ter text.  Skim through a few dozen pages look­ing for weird HTML or XML frag­ments (angle brack­ets and equal signs that look out of place). 

If every­thing looks good, you imported the good doc­u­ment.  If things look funky, close the viewer, delete the doc­u­ment from Cal­i­bre, then import the other one.

Once you have the good doc­u­ment imported, select it and click on the “Con­vert E-Books” icon in the tool­bar.  Be sure to change the out­put for­mat to EPUB.  You can cus­tomize other options as you see fit.  Click “OK” to convert.

Once con­verted, you can find the file in your Cal­i­bre folder or you can use the export func­tion­al­ity to save it to disk.

Drag the result­ing *.epub file into the Books sec­tion of iTunes and you’re set.  The book you pur­chased on Ama­zon can now be read in Apple’s iBooks application.

Share and Enjoy:
  • Digg
  • Reddit
  • del.icio.us
  • StumbleUpon
  • Yahoo! Buzz
  • Facebook
  • Google Bookmarks
  • Technorati

If you liked this post, you may also enjoy:

  1. 1984 goes down the Mem­ory Hole
  2. One week with the iPad
  3. Why I prob­a­bly won’t be get­ting a Kin­dle 2
  4. Com­ments from all over (Vir­tual Kin­dle Style)
  5. The lit­tle details, like hyphens and em-dashes

{ 29 comments… read them below or add one }

1 Taylor April 19, 2010 at 10:55 pm

I love this post- thank you for taking the time to share it. I am, however, stuck. My kindle ebook was downloaded via the kindle app on the iPad, and as I have no Kindle device, I have no serial number that is easily accessible to me.

Is there any way to retrieve the necessary kindle serial number used on non-kindle devices?

Reply

2 Brian Enigma April 21, 2010 at 8:19 pm

Sorry, I’m not really familiar with how to get files out of the iPad, or even what they use as an encryption key once the file is available. Perhaps if/when a jailbreak is released for the iPad, it might be possible, but if it’s anything like the iPhone, that involves setting up a network and SSH server and lots of more hardcore unix-y things, whereas you can just copy the *.azw file out of the Kindle as if it’s an external drive.

Reply

3 Taylor April 21, 2010 at 8:24 pm

well I was able to get the .azw file out just fine by downloading the macintosh client of the kindle reader and whispersyncing it to my HD. But the dilemma of figuring out the key remains. Thanks for your response.

Reply

4 Castell Faber April 30, 2010 at 9:21 am

A concise explanation of how to get to your Iphone books and how to get a get a usable PID. Perhaps something analogous will help for iPad users.

http://www.mobileread.mobi/forums/showthread.php?t=65275

Reply

5 Bookworm May 8, 2010 at 2:19 pm

Your instructions are very helpful. I was able to convert a lot of my books into ePub from my Kindle to my iPad. But wondering, what about those titles that have spaces. I keep on getting a error msg that says : No such file or directory: “A” because my azw file has a title that begins with “A Fine…” I have tried putting “_”, “-”, ” “, ‘ ‘, “\” in place of the space but no dice… Will I be able to convert these books? Thanks for your help.

Reply

6 Brian Enigma May 8, 2010 at 2:33 pm

For filenames with spaces on the command line, you have two options. First, you can specify each space as “\ ” (that’s backslash space). Alternately, you can just wrap the whole thing in single-quotes like ‘A Fine…’ and skip using backslashes.

Or you could totally subvert it and rename the file to get rid of the spaces — make ‘em underscores or something. The filename is just a filename. The name of the book is a field contained inside of the file.

Reply

7 Bookworm May 8, 2010 at 8:24 pm

Thanks! I got rid of the spaces and the conversions were successful!

I also found that some of my Kindle files have an azw1 extension vs. just plain azw. I have not been able to convert them using python. Do I use a different converter?

Reply

8 Brian Enigma May 9, 2010 at 9:00 am

I can’t say that I have run into an azw1 file extension, so I am not sure I can offer any advice on that.

Reply

9 snorkledorf May 20, 2010 at 6:23 pm

az1 files are Amazon’s Topaz format (when downloaded via computer they come out with the .tpz extension, and .azw1 when grabbed with WhisperNet directly onto a Kindle). Topaz files are funkier than .azw files when converting. I’ve stripped the DRM from all of my Topaz files using the scripts in the “tools_v1.6b” archive (available here & there on the intertubes), but instead of just a de-DRM’ed book, it leaves me with an HTML file. Calibre can then convert that to epub, but the OCR isn’t so well corrected in Topaz files. You might want to edit it a bit before converting.

Reply

10 Brian Enigma May 20, 2010 at 8:56 pm

Oh, hey, thanks snorkledorf! I haven’t run into an azw1/tpz file yet, but I’ve snagged tools_v1.6b for if and when when I do.

And yes, I have even noticed weird and blatant OCR artifacts in regular azw files: http://netninja.com/2009/09/15/the-little-details-like-hyphens-and-em-dashes/

11 LobstahLovah May 9, 2010 at 5:26 am

The thang which concerns me is the app kindlepid.py is downloaded from Paraguay and probably comes loaded with spyware from some cloaked Chinese hacker – I’d rather wait for Apple to give us the app to import kindle books into iPad – in the meantime . . . just got my iPad last week and I love it – only have 6 books from kindle for iPhone so not too concerned about switching between apps to read. Inconvenient, yes, worth the risk of losing any personal info to a hacker with a keystroke logger imbedded into a script – NO – perhaps I am overly cautious but I am not taking the risk. Great info though, thank you for sharing

Reply

12 Brian Enigma May 9, 2010 at 9:17 am

I wouldn’t hold your breath on an official import tool. Converting yourself falls, as best as I can tell, in the fuzzy gray area of “Fair Use” under the DMCA. I’m not sure that a big company could provide the same thing — I have seen several companies that tried to provide DVD ripping software that then got sued out of existence.

As far as malware in the Python script: it is just a script file. It’s about 40 lines of text when you open it in a text editor. If you have any programmer-inclined friends, you can have them take a look at the content. I had similar concerns when I first downloaded from a site of similar questionable origins. Fortunately, Python is fairly straightforward and readable.

Reply

13 Sioned May 11, 2010 at 7:29 am

So how bout converting azw that are downloaded on my pc to epub format, is it possible?

Reply

14 Brian Enigma May 11, 2010 at 9:15 am

It is possible and the theory and steps are the same as above, but the key would be different. I don’t know how the PC version’s keys are generated, unfortunately. If you knew the encryption/decryption key for the PC version’s AZW file, you’d be able to do it.

Reply

15 Sioned May 12, 2010 at 8:56 pm

yep, this is the crux, if anyone has figured out how to get a Kindle for PC PID to use with python, PLEASE, let me know!

Reply

16 Taylor May 11, 2010 at 7:34 am

@sioned: that is precisely what this article covers.

Reply

17 Sioned May 11, 2010 at 8:20 am

no, this requires that you have a kindle….i do not :) I want to be able to do this without involving a kindle, only the pc.

Reply

18 Taylor May 11, 2010 at 9:24 am

Sorry sioned, your question only asked if azw files could be converted to epub, which is covered here (use calibre). However, the good news is that what you are trying to do can be accomplished too (I’ve done it).

Basically you will still use 90% of this guide. When you get to the bits hat require you to use a kindle, refer to the link that Castell posted (reply #4). The link will direct you to a guide that shows you how to get your ebook file off your device and onto your computer, and then how to get a useable serial number from iTunes. Once you’ve got that, you can return to Brian’s guide here and resume the process. Hope that helps.

Reply

19 Sioned May 11, 2010 at 11:05 am

Awesome guys, thanks Taylor and Brian. I will give it a try later today, frankly I’m tired of messing with this stuff for 2 days with minimal results, lol. ;)

Reply

20 Sioned May 12, 2010 at 8:55 pm

Ugh, I’ve been to all the links and I’m still struggling.. can u tell i’ve never done anything like this before? lol.
Anyway, here’s my problem, i don’t have an ipod or i-anything.

I have downloaded a book I want onto the Kindle for PC app, I have python, I need a Kindle PID to dedrm the .azw file so I can turn it into an .epub file and use it on a different ereader format.

All the links i’ve read tell you how to get the Kindle PID if you have an ipod, etc.., which I don’t, so is there anyway to get a PID to use with python to dedrm this book??

Thanks for all the help, i’m still hopeful i will get to read what i want the way i want…eventually. ;)

Reply

21 Taylor May 12, 2010 at 9:02 pm

Oh, my apologies Sioned, I have not done it the way you are asking. I think I heard of someone getting it to work, but I don’t remember in detail. I think there is some sort of dummy azw file you download, and when you run a script on it, it provides your PID. I would double check the link posted by castell above for this information, otherwise I don’t know how to help, sorry =(

Reply

22 john June 1, 2010 at 7:07 am

Brain Engima or others that might be able to assist,

Does anyone know how to convert .azw from Amazon subscription news papers into ipad readable format? I know the kindle for ipad app. and kindle for mac app works for amazon purchased books but it does not work for news paper subscriptions. Anyway, around this?

thanks,

Reply

23 Felipe June 25, 2010 at 11:12 am

I’ve used different versions of mobidedrm and all of them give me the following error:

Error: no key found. maybe the PID is incorrect

I’m sure my PID is correct and I’ve checked my Serial# a hundred times. I’ve tried both the PID with ‘ and without. The original book title is Zombie, so there’s no need for ‘ there.

I’ve tried this command line and several variations of it. I’ve also tried it with different ebooks both transfered from my Kindle and downloaded from Amazon.

python ./mobidedrm.py Zombie.azw mybook1.mobi ‘XXXXXXX*XX’

I’m running out of ideas. Anyone has any suggestions?

Reply

24 Jane Evans July 28, 2010 at 8:37 am

No matter what I do, I get the message “Error: no key found. maybe the PID is incorrect” I have obtained the 40 digit number from both iPad AND iPhone. I used the converter found at Kindle Tools PID Generator as well as the “python kindlepid.py XXXXXXXXXXXXXXXX” There is both an older version which generates numbers with the *XX in them and a newer one which has no * and produces the same number as the PID Generator. I have the Kindle Whispersync application on my Mac and have the downloaded files in a folder “My Kindle Content” There, each azw file has a companion mbp file. The web says “This file is used to store metadata like bookmarks, annotations, last read position and other data. The MBP file is also used on the Kindle to store information about a book being read on the device.” Any connection to the problem of conversion? I am stuck – and it seemed so easy.

Reply

25 Felipe July 28, 2010 at 9:29 am

The same thing is happening to me and I’m also on a Mac.

Reply

26 Brian Enigma July 28, 2010 at 10:30 am

Do keep in mind that every device or application that has a copy of the ebook has a different key. The file downloaded over WiFi or 3G into your iPad has a different key than the one downloaded over WiFi or 3G on your iPhone, which has a different key than the one downloaded by the Kindle app on the Mac. If you’re trying to use the iPhone or iPad key on the desktop software’s file, it won’t work. You’ll need the key of the desktop software. I’m not familiar enough with the software to know what the key might be or how to find out what it is (I mainly download onto my hardware Kindle and use its serial number, so haven’t had the need to investigate further).

Reply

27 Snorkledorf July 29, 2010 at 4:54 pm

One thing to keep in mind is that with the Kindle 2.5 update a month or two ago, they changed their DRM. Apparently now it uses a separate key per-book instead of a single DRM key on a given device (per-PID). I’m not sure how that effects files downloaded onto the iPad/iPhone, but files downloaded on my Mac for reading on my Kindle don’t de-DRM easily anymore using the scripts.

What I’m doing to work around this is use VMWare to run Windows on my Mac (yeah I know, I know), and run the Windows version of the Kindle app there. It still uses the old DRM scheme so the PID and such work as before. I’ve also turned off auto-update on it, so it hopefully won’t update itself to the new DRM scheme.

Reply

28 Jane Evans July 29, 2010 at 5:18 pm

@Snorkledorf: How did you find a PID for the Mac itself, as the iPhone or iPad ones will not work with the files downloaded on my Mac? Finding that would be helpful.

Reply

29 Snorkledorf July 29, 2010 at 10:28 pm

Well that’s the thing, I’ve never actually figured out how to strip the DRM for books that are downloaded via/for the Mac version of Kindle reader. The PC version DRM is a known quantity however, so for me it’s worth it to fire up Windows in a virtual machine on the Mac, then do all the de-DRM’ing inside Windows, using Kindle Reader for PC.

Since any Mac can run Windows (I use VMware Fusion, but Parallels or Boot Camp will work just as well), that was just the easier route for me. Even if I have to grit my teeth for the duration. :)

Reply

Leave a Comment

{ 2 trackbacks }

Previous post:

Next post: