Eventually these clippings will get deleted or filed in a folder and forgotten. In the past I’ve written Spotlight importers to try and gather all these clippings and then export them in a format I could use in a bookmark manager application, such as WebNoteHappy. But most bookmark manager apps tend to be quite limited and I’ve found online services such as delicious, instapaper and now pinboard.in to be far superior to any desktop application.

But getting from an internet clipping on my desktop to an entry in an online bookmark manager usually involves a lot of manual labour.

I’ve found a great solution using Hazel and a custom Python script. I have a Hazel rule that finds files with a “.webloc” file extension on my desktop. The rule then runs a single python script to add the URL to my pinboard.in account and then moves the file into the trash. Usually Hazel notices the clipping, adds it to pinboard.in and trashes the file within a couple of seconds.

The python script is where all the magic happens:

#!/usr/bin/python

import sys
import urllib
import urllib2
import re
import subprocess
import Foundation
from Carbon import File, Files, Res

def infoForWebloc(inPath):
    theDisplayName = Foundation.NSFileManager.defaultManager().displayNameAtPath_(inPath)
    resNum = Res.FSOpenResourceFile(inPath, File.FSGetResourceForkName(), Files.fsRdPerm)
    Res.UseResFile(resNum)
    theResource = Res.Get1Resource('url ', 256)
    theURLFromResource = theResource.data
    Res.CloseResFile(resNum)
    theData = Foundation.NSData.dataWithContentsOfFile_(inPath)
    thePropertyList = Foundation.NSPropertyListSerialization.propertyListWithData_options_format_error_(theData, 0, None, None)
    theURLFromData = thePropertyList['URL'] 
    return theDisplayName, theURLFromResource

def getAccount(inServer):
    theArguments = ['security', 'find-internet-password', '-s', inServer, '-g']
    thePipe = subprocess.Popen(theArguments, stdout=subprocess.PIPE, stderr=subprocess.PIPE)
    theOutput = thePipe.stdout.read()
    theMatch = re.search(r'"acct"<blob>="(.+)"', theOutput)
    theUsername = theMatch.groups()[0]
    theOutput = thePipe.stderr.read()
    theMatch = re.match(r'^password: "(.+)"$', theOutput)
    thePassword = theMatch.groups()[0]
    return theUsername, thePassword

def upload(inURL, inDescription):
    password_mgr = urllib2.HTTPPasswordMgrWithDefaultRealm()
    top_level_url = "https://api.pinboard.in/v1"
    theUsername, thePassword = getAccount('api.pinboard.in')
    password_mgr.add_password(None, top_level_url, theUsername, thePassword)
    handler = urllib2.HTTPBasicAuthHandler(password_mgr)
    opener = urllib2.build_opener(handler)
    inURL = urllib.quote(inURL)
    inDescription = urllib.quote(inDescription)
    theURL = 'https://api.pinboard.in/v1/posts/add?url=%s&shared=no&replace=yes&description=%s' % (inURL, inDescription)
    theResult = opener.open(theURL)

def main(args):
    for thePath in args:
        theDescription, theURL = infoForWebloc(thePath)
        upload(theURL, theDescription)

if __name__ == '__main__':
    main(sys.argv[1:])

This script reads the .webloc file to extract the URL (.webloc files store the URL in both the Carbon resource fork and in the data fork in a property list format, this script extracts the URL from both locations but only uses the URL from the resource fork.

Then script uploads the URL to pinboard.in via pinboard.in’s delicious style API. The script gets your pinboard.in username and password from your keychain. To add the username and password to your keychain you’ll want to run this little shell script.

#!/bin/sh

HOST=api.pinboard.in
read -p "Username: " USERNAME
read -p "Password: " -s PASSWORD
security add-internet-password -U -r http -s "$HOST" -a "$USERNAME" -w "$PASSWORD"

You can download the Hazel rule here, but do remember to set up your pinboard.in keychain item before installing it.

Instead of using the entire desktop as a sort of a dropbox for URLs you could easily adapt the script and put it into a automator action or perhaps a system service.

One feature that added recently that hg lacked was the ability to store repository passwords in the Mac OS X keychain. In fact mercurial lacks the ability to store a repository password anywhere other than in the url used to access the repository (i.e. http://user:password@example.com/path). I think this is a bit of security flaw. The password is stored as plain text within the repository config file (‘.hg/hgrc’) file.

Fortunately Mercurial provides a simple interface for extension modules. After a little bit of hacking I was able to write an hg extension that stores and retrieves the password from the keychain. I’ve put the code online for anyone to use. hgkeychain:

• Download: hgkeychain.tar.bz2
• Source Code
• README

And yes the source code is currently hosted in a Subversion repository :-) I did mention I’m in the middle of transitioning to Mercurial didn’t I?

Update

I’ve update the project for Mercurial 1.1 – it should now work on 1.1 and also 1.0 (and lower). I’ve also decided to move the code to bitbucket instead: https://www.bitbucket.org/schwa/hgkeychain/

I’ve written an Automator action that allows you to write python scripts directly inside your Automator workflow.

“Run Python Script” Automator action (catchy title) is written using Python and PyObjC (now built-in to Mac OS X 10.5). Apple already provides “Run AppleScript” and “Run Shell Script” actions with Automator which give Automator a high degree of flexibility. However Python is my preferred scripting language and by writing a custom action purely for Python I was able to take advantage of some PyObjC features that in my opinion make my action superior to the provided Apple scripting action. I’m releasing all the source code to the action under the BSD Open Source license.

One of the main advantages of the action is that Python (with the help of PyObjC) understands the AppleEvent descriptors that Automator uses to pass data between workflow actions. This allows the user to create a custom action that accepts and provides data of any type. In contrast Apple’s “Run Shell Script” action can only accept and provide text (usually limited to file paths). Some work does need to be done to make working with any data (which is represented by NSAppleEventDescriptor objects) more convenient.

The default Python script for a new action in a workflow follows:

import sys

def main(input, *args, **kwargs):
    '''Your script goes here.'''
    print 'Hello world'
    return sys.stdout

This is pretty simple. The action will optionally convert the input into Python types (currently limited to converting typeAlias AppleEvent descriptors into path strings) to allow simple processing. Output is (optionally) converted from Python types to native Automator types. See the Examples directory for more.

The following screenshot shows the action used in a real workflow. This workflow asks the user for a keyword and then the Python action downloads photos from the morguefile public stock photo archive that are related to the keyword. Once download it performs some processing on the files (scales them to 640 by 480) and produces a PDF “Contact Book” from the images.

The source code uses a 10.5 specific technique to create Python based bundles. It also has a simple Python syntax colouring NSTextView (using Python itself to colourise the source).

I’m calling the Action a 0.1 release and have some plans/ideas for further releases. See the TODO list.

Update: I’ve added a sample script showing how to upload files to Amazon S3 from within a workflow.

The code in subversion is a lot newer than the binary I’m linking to. If possible do a checkout and build the plugin yourself.

Update: This Automator action has issues on Snow Leopard and some Leopard versions. Use at your own risk.

Update: Automator’s “Run Terminal Script” looks like it supports and understands Python native. That means, although my .action was better (syntax highlighting, better control of input and output, better logging, etc), I probably won’t bother to update it to Snow Leopard.

[schwa@umbriel] Dumps$ admin create test
[schwa@umbriel] Dumps$admin load test < dumpfile 'vnadmin: Invalid control character '0x0d' in path 'trunk/Path/To/Some/Directory/Icon<

My attempt to recover from a dumpfile failed because my repository contained a folder with a custom icon. Crickey. I find it odd that subversion lets me check these bad files into subversion in the first place. But whatever, I had to recover this dump file. Fortunately there is a tool called ‘dumpfilter’ that can be used to filter arbitrary paths in dump files. But after removing the first bad Icon file I found this particular repository had more than one. Instead of removing them by hand I started to write a shell script to automate the filtering process. Of course my shellfu is too low so I switched to Python:

#!/usr/bin/python
import commands
import re
import sys
theInputFile = sys.argv[1]
theOutputFile = None
if len(sys.argv) > 2:
    theOutputFile = sys.argv[2]
    thePattern = re.compile(r'^Node-path: (.+/Icon\r)$')
    thePaths = [thePattern.match(theLine).group(1) for theLine in open(theInputFile) if thePattern.match(theLine)]
    if theOutputFile == None:
        print 'Found the following paths:'
        for thePath in thePaths:
            print '\t\'%s\'' % thePath.replace('\r', '\\r')
        else:
            theCommand = 'dumpfilter exclude %s < %s > %s' % (' '.join(['\'%s\" % thePath for thePath in thePaths]), theInputFile, theOutputFile)
            print commands.getstatusoutput(theCommand)[1]

This script, when passed a path to an existing subversion dump file, will list all paths within the dump that contain custom icons. When passed a path to an existing dump file and the name of a new dump file it will remove all paths containing custom icons. You can then use this new dump file to populate your subversion repository without those pesky files. Update: From the comments, a link to Ryan Wilcox’s blog page concerning the problem (and a more detailed description of the solution): http://www.wilcoxd.com/blog/howto-get-out-the-icon-files-in-a-subversion-repo.html

WordPressTool downloads all postings into plain text files (one per posting). You can then treat your postings as regular text files: grep through them, perform global search and replaces, and so on. You can then run the script again (don’t forget to tweak it to change the import operation to an output) and it will upload all the updated posts to the server.

I’ve just used it to fix all the links to my subversion repository and it works great (BBEdit’s multi-file search and replace worked a treat.)

Fortunately I had already written some code to export the content from ExpressionEngine. All I needed to do was write some more code to import the data into WordPress.

I am posting this code online and briefly describing it so that others in similar situations might be able to transfer their content.

“ExpressionEngineExport.py” uses Sqlobject to talk directly to the EE database (in my case via a SSH tunnel because this code was running on my local machine, talking to a firewalled MySQL server). I model as much of EE as I needed with Sqlobject classes then I “simply” walk through the sql tables sucking out data, formatting it and fixing it where possible and then writing it to an XML file.

“WordPressImporter.py” takes the XML file and via Sqlobject imports it directly into WordPress’s database.

The code works pretty reliably for me (I had a test server set up and tested the code quite thoroughly before importing the data into my “production” server), however I make absolutely no guarantees that it will work for you. I am posting the code “as-is” in the hopes that it will be useful but I do not plan on supporting this code in any way.

The code can be found within my subversion directory.