Transferred everything from Apple Mail to hosted Google Apps

My ISP decided that their email service sucks because they offer limited space (100MB! oh, please) and spam is a huge problem. Earlier this week, the ISP announced they are moving to Google Apps. My email address is changing (though slightly). This is a mini shock to me. I’ve been with the same ISP since 2001. I resisted getting one of those free webmail accounts so I can avoid changing email addresses if things go south. I started with dial up service and resisted signing up for cable to wait for DSL to be installed around where I lived. Even when my phone company offers a cheaper combo package of DSL with MSN, I still prefer having my local ISP to handle my internet service. Local businesses are getting sucked up by corporate America everyday so I try to support local businesses as much as I can.

I started with importing contacts. I had been using Outlook Express and then Entourage until less than a year ago. Entourage was starting up really slow and I am guessing the huge number of messages I have been keeping might be the problem. I switched to Apple Mail and the program startup speed improved. But Apple Mail utilizes Address Book, another application of Mac OS X to maintain contacts. I had to do a little work moving contacts from Entourage to Address Book and that only required exporting contacts to a text file for import. In the case of going from Address Book to the Gmail interface, there is one extra step. Address Book keeps everything in vCards. I located a nifty utility through Hawk Wings and exported everything in Address Book to a CSV file which is the standard for importing contacts in Gmail. One problem with this method is all the groups I set up in Address Book ended up in the Notes field of each contact in Gmail. I had to create new groups and reorganize. I didn’t have that many groups (or contacts in that matter) so it’s no big deal.

The next step to complete migration is transferring email messages. I wasn’t too worried about it although I have more than several thousands of messages. I thought if I can find a utility to transfer contacts, there has to be something out there for messages. GMail Loader came up in my search early but it only works for the true Gmail account (ends with @gmail.com), not my Google Apps account (ends with @yourdomain.com). I figured it all out anyway and it took a lot of extra steps. It requires using Python megawidgets, GMail Loader, and a true Gmail account. If you have Tiger on your Mac, you can follow these steps:

  1. Download and unzip Python megawidgets to the Desktop.
  2. From the Finder, select Go -> Go To Folder, and type in /Library/Python/2.3/site-packages/
  3. Move the Pmw subfolder within the src folder from your desktop into the site-packages folder.
  4. Download the Linux/OSX/source only version of GMail Loader.
  5. Select messages in Apple Mail and select File -> Save As. Select Format: Raw Message Source. Type in the filename of your choice and add the .mbox file extension. Click Save. (I recommend starting with 1 message as a test because GMail Loader takes some trial and error to use successfully.)
  6. Open Terminal (Applications/Utilities folder). Type in the command to change directories to the location of the gmlw.py script (in my case, it’s cd /Users/username/Desktop/gmlw). Type in pythonw gmlw.py, and hit Return.
  7. The GMail Loader window opens. Check the box for “Requires Authentication”. Type in your username and password for your Gmail account (the one ends with @gmail.com, not the one @yourdomain.com).
  8. Click the “Find” button and locate the .mbox file you created in Step 5.
  9. Under the File Type option, I selected the Less Strict option for mBox. I left the Message Type as Goes to Inbox. Type in your full Gmail address in the box at the bottom.
  10. Click “Send to Gmail”. If you starts with 1 message in Step 5, a success or error message will come up quickly.
  11. If you get an error message, you need to try a different SMTP server address. Open this list (Update: the list of alternate addresses is no longer available for free, check out the picture below taken from the June 2007 Wayback Machine‘s archive) and copy any one of the server addresses from the “Answer” column to the SMTP server box in the GMail Loader window. In my case, alt2.gmail-smtp-in.l.google.com works for me. Repeat trial and error until you get a successful transfer.
    gmailsmtp-jun07.jpg
  12. Sign in to your Gmail account. The message you exported in Step 5 should be in the Inbox. Enable POP Download. Sign out.
  13. Sign in and set up your Google Apps account to fetch messages from your Gmail account (see Google Apps Help Center instructions).
  14. Drink some tea and wait a little while for the fetching to happen.
  15. When the message appears in your Google Apps account Inbox, raise your arms and scream hurray (I didn’t do it so you should, it’s already Step 15 for your sake). Seriously, label the message you just imported and archive it.
  16. Repeat Steps 5, 8, 10, and 15 for a group of messages from one Apple Mail folder at a time.
  17. Disable fetching of messages between accounts if you wish.

Step 16 assumes you organize your messages into folders in Apple Mail. I have more than 30 folders. If I transfer all the messages to the Inbox in Gmail in one setting, I will lose all the organization. During the time I write this post, I am still stuck at Step 10 with GMail Loader transferring over 900 messages from one of my folders. Another tip about labeling and archiving: keep your Inbox in the Google Apps account empty so when a batch of messages arrive, you can select them all to label and archive without getting mixed up with some existing messages. Also, if there are messages (whether archived or not) in your Gmail account, they all get fetch to the Google Apps account. You might have heard of the free tool, gXFER which can handle transfers (contacts, email and calendar) between Gmail and Google Apps accounts. The drawback for this tool is all the manual labels (the ones not created by filtering) will get lost in the process.

I could have set up Apple Mail to start picking up messages from my new email address and forget about all the trouble I still have to finish. The advantage of this change is I will be able to access all my past and present emails from anywhere I can access the web. It will be a pleasant change during my occasional travels throughout the year.

Credits:

Update (Sept 8, 07): It took me almost 2 weeks to transfer a small amount of messages each night to get most of them uploaded to my Google Apps account. During the process, I found some messages to be unimportant so I skipped about 50-100 of them. All in all, I think I uploaded over 3,000 messages based on Google Apps telling me I have over 2,900 “conversations” in the account. The process was not completely smooth. During the transfer of one big batch of messages, GMail Loader got stuck. I tried to reload the same batch again several times but it kept getting stuck. The solution was to try a different SMTP server address by visiting this list (see update in Step 11) again. It looks to me the server addresses keep getting updated over time. From that point on, I rotated several server addresses for every couple hundreds of messages and I never got stuck again. Another thing to watch out for is Gmail’s spam filter. Some messages I uploaded landed in the spam folder in my Gmail account so they wouldn’t get forwarded to the Google Apps account. I had to keep my eye on both accounts to make sure all the messages went through.

The last thing to do is clean up. Although I selected to have the messages deleted from the Gmail account after forwarding, they are not deleted completely. They ended up in the Trash. That was actually a good thing because I had messages in the Gmail account at the beginning that I didn’t intend to have forwarded to the Google Apps account. I brought them out of Trash and everything went back to how it was before. And of course, I disabled “POP download” in Gmail and “Get Mail” in Google Apps to discontinue this massive email messages pass-through.

Advertisements

7 thoughts on “Transferred everything from Apple Mail to hosted Google Apps

  1. Interesting approach, Mike. I am not a programmer either so I didn’t think about using Apple Script. May be there is a way to do it in Automator, too, who knows?

    Does your method retain all the history of the original messages? What I mean is that do the messages in your Gmail/Google Apps account retain the same Sender, Received Date, attachments, CC, etc? When I first asked my ISP for help, they told me to forward all my old messages and that would just change all the Senders to me and the Received Date get updated to whenever the forward happens. My method keeps all the info intact. Messages that came years ago still look exactly the same.

    Anyway, good to hear about alternative methods.

  2. Thank you for posting this resource. I wish there was an easier way to do this, but I’m determined to get it working. I followed the first few steps with relative ease, but got stuck on step 6. This is what Terminal returned to me when I tried to change the directory, hence not opening the Gmail loader window:

    Traceback (most recent call last):
    File “gmlw.py”, line 42, in
    import os, sys, mailbox, smtplib, time, Pmw, webbrowser, tkFileDialog, re
    ImportError: No module named Pmw

    Any ideas?

  3. I don’t think I can be much help because I rarely use Terminal but I can follow directions. Here are some thoughts:

    My main question is what version of OS X is on your Mac. The instructions I put together are written for Tiger. If your OS is 10.2 or earlier, refer to paskal’s “gmail loader instructions for mac os x” about downloading and installing Tcl/Tk Aqua and MacPython. If your OS is Leopard, I have no experience with that.

    Otherwise, it looks to me your machine was not able to locate Pmw. Revisit steps 2 and 3 above, double check the folder /Library/Python/2.3/site-packages/ to make sure the Pmw folder and its contents are there.

    Also, make sure you downloaded the Linux/OSX/source only version of GMail Loader, not the Windows version.

  4. Thanks for your reply.

    I have Leopard, so the process might be a little different. I did try using the windows version on a PC and wasn’t able to upload an e-mail. I wonder if Google is blocking this procedure now. It may be a SMTP server problem, and the list you reference requires an account with DNS stuff. I’ll contact Mark Lyon’s and see if he has any suggestions.

    Thanks again.

  5. I found this information very helpful, but a bit cumbersome when your account has many different folders. I ended up taking a different approach: Setting up my new Google Apps email as an IMAP account within Apple Mail. I then just dragged and dropped messages from my old email account to the new account and it worked pretty flawlessly.

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s