Thursday, December 18, 2008

Bookmarklet to find cheapest service to call a mobile from a fixed line for the current time and day

Callchecker (UK) has a great service to lookup the best 0845/0870 service to call a UK mobile from your fixed line telephone. This varies by weekday, evening or weekend rate. The following bookmarklet looks up the right page based on the time on your computer.
javascript:(function(){
var t='weekday',d=new Date();
if(d.getDay()==0||d.getDay()==6){t='weekend'}else{
if(d.getHours()>17||d.getHours<8){t='evening'}};
window.location='http://callchecker.moneysavingexpert.com/\
ukcallchecker/mobile-numbers/'+t
})()

Here it is as a bookmark that you can test out or drag and drop into your browser Bookmarks.

Thursday, December 11, 2008

Bookmarklet to add recycle comment in GreenMetropolis

I'm selling a fair number of my old paperbacks on GreenMetropolis.com. When listing a book I invariably want to note that I will send in recycled packaging in the comments field. I was using iMacro to speed this up but I've finally got around to creating a simple bookmarklet to do the same thing. Here it is:

javascript:(
function(){
var cmts=document.getElementsByTagName('TEXTAREA');
for(var ic=0;ic<cmts.length;ic++){
if(cmts[ic].name=='comment'){
cmts[ic].value+='I will send with reused or recycled packaging.'
}
}
}
)()

And here it is as a link you can drag and drop into your own bookmarks or try it out by clicking on it here: Add note to TextArea



Reset text area

This would be fairly easy to customize for your own text or to adapt to a different form that you use regularly and get tired of typing in the same old text or if you are familiar with RegEx tweaking the text you already have in the text box. For example changing the text in the text box to Title Case or Shout.

Tuesday, December 9, 2008

Creating an intelligent shortcut (bookmarklet) in Firefox

On of the frequent address shortcuts I use in Firefox is to redirect "w sometext" in the address bar to lookup a term (sometext) in Wikipedia. As I frequently go to my "Watchlist" in Wikipedia I though it would be neat to default to that page if I had not specified a term. A little research and I'd worked out how to get this to work as a bookmarklet. In the address bar any text added to the shortcut is referenced as "%s".

Using this basic structure it would be pretty easy to add a more complex set of conditions to make the same shortcut do many more things, such as looking up a book and working out if the text you put in was an ISBN or a title.

Here's my example formatted so you can read it:


javascript:(
function(){
  if("%s".length==0){
     // Default
    window.location="http://en.wikipedia.org/wiki/Special:Watchlist"
  }else{
     // Wikipedia lookup
    window.location="http://en.wikipedia.org/wiki/%s"
  }
}
)()


And here it is formatted as a link you can drop into your bookmarks, don't forget to edit it so that it has "w" as a keyword/shortcut:
Wikipedia lookup

Monday, December 1, 2008

Moving my mobile number from Virgin to the evil ASDA / Wal*mart empire

Sadly Virgin aren't doing all that much to compete with the johnny-cum-lately ASDA. ASDA are currently charging a flat rate for PAYG of 8p/min for all UK landline and all mobile calls with 4p per text message. Virgin claimed to charge 15p/min and 3p for texts on PAYG but in practice (as I don't know anyone using Virgin and my usage is so low it was never worth buying a "bundle") their charges were actually a whopping 40p/min and 12p/text.

It's a great pity but as my average bill even at these rates is always less than £5/month I can't afford to ignore the ASDA deal.

Transfer happened this weekend. I emailed Virgin via their website on Thursday, phoned ASDA with the transfer code on Friday and my mobile number was tranferred today. Surprisingly painless compared to the last time I had to do this (it took 3 weeks).

Sunday, November 16, 2008

Use Greasemonkey to link UK postcodes to Google Maps

See my last blog post for general details about using Greasemonkey. This titbit of source code uses a large standard regex to match any UK postcode in a web page text (you could be matching paragraph elements) and converts it into a hyperlink to a google map of that area. It assumes that this titbit is inside a script that searches relevant elements of the page and tests them as variable "thisElement".

// Convert postcodes to google map links (z=15 puts names on tube stations)
thisElement.innerHTML=thisElement.innerHTML.replace(/([A-PR-UWYZ0-9][A-HK-Y0-9][AEHMNPRTVXY0-9]?[ABEHMNPRVWXY0-9]? {1,2}[0-9][ABD-HJLN-UW-Z]{2}|GIR 0AA)/,"<a href='http://maps.google.com/maps?f=q&hl=en&geocode=&q=$1&ie=UTF8&z=15&iwloc=addr' target=_blank>$1</a>");

Friday, November 14, 2008

Write your own Greasemonkey script to highlight and hide items on a website

Here are some handy code segments that have taken me a little while to work out. Greasemonkey is a useful Firefox plugin with a massive shared library of scripts to do interesting things like adding collapsible folders to your online gmail web page. You can also write your own javascript code for Greasemonkey to apply to web sites you name.

The following code I use for a mystery shopping website I visit regularly but their table of possible assignments available in London can be several pages long. This script helps filter and colour in the table with my preferences (for example out of 80 London assignments I currently have 7 highlighted as suitable).

// ==UserScript==
// @name Website filter
// @namespace Filters
// @include http://mywebsite.com

/* Here I'm searching out items in a table that have been
highlighted in bold already. I'm then searching the bold
text and if it contains "William", I then hide the row
completely. I then go on to pick out items with "London"
in the text in dark green */


var allElements, thisElement;
allElements = document.evaluate(
'//tr/td/b',
document,
null,
XPathResult.UNORDERED_NODE_SNAPSHOT_TYPE,
null);
for (var i = 0; i < allElements.snapshotLength; i++) {
thisElement = allElements.snapshotItem(i);
// Hide these rows
if(thisElement.innerHTML.search(/William/i)>=0) {
thisElement.parentNode.parentNode.style.display="none";
}
// Highlight these elements
if(thisElement.innerHTML.search(/London/i)>=0) {
thisElement.style.color="darkgreen";
}
}

/* In this section I'm searching for any table cell and
checking the text for certain postcodes and addresses.
If there is a match, the cell is highlighted in bold
blue text. */


allElements = document.evaluate(
'//td',
document,
null,
XPathResult.UNORDERED_NODE_SNAPSHOT_TYPE,
null);
for (var i = 0; i < allElements.snapshotLength; i++) {
thisElement = allElements.snapshotItem(i);
// Highlight these addresses
if(thisElement.innerHTML.search(/SE(22|15|1\s)|London Bridge/i)>0) {
with(thisElement.style){
color="blue";
fontWeight="bold";
}
}
}

// ==/UserScript==

Explanations

Regular expressions (Regex) can be a bit cryptic but as you'll need to tailor these in the above code I though I'd explain what these bits do:
search(/William/i) ==> Search anywhere in the string for "William" and return the starting position within the string, the 'i' means that it will not be case sensitive.
search(/SE(22|15|1\s)|London Bridge/i) ==> Search anywhere in the string for "SE22", "SE15", "SE1 " or "London Bridge", ignore case and return the position.

You will also note that the first search that hides matching rows uses parentNode twice on the assumption that the bold text will be in a cell which will be in a row, so the second parent node is the "TR" tag containing the row. By changing the style.display attribute this dynamically hides the row from view (to reset and show the row again you would set style.display to null). Phew.

Useful links

Greasemonkey: https://addons.mozilla.org
XPath syntax: http://www.w3schools.com/XPath

Monday, November 10, 2008

iMacro script to swap between Flickr albums

The following handy iMacro script logs you from one Flickr account to another. I currently have six of these, one for each separate free Flickr album/account I have on the go (there's a maximum of 200 photos you can share for free in each account). All you have to do is install the iMacro plugin into your browser and save the following script not forgetting to add your login and password in the "User set variables" section.

Obviously if Flickr change their main site layout then this may break the script and it would have to be tweaked.

Script Starts

' Last Edit: 13 Nov 2008

' User set variables
' !VAR1 = your login ID, !VAR2 = your password

SET !VAR1 yourlogin@yahoo.com
SET !VAR2 password

' This script is intended to toggle you out of your
' current Flickr account and into one set by the
' variables above so you can easily swap between
' several albums

' Assumptions
' - A Sign Out link is on every Flickr page
' - You are currently logged into Flickr

' Main script

TAB T=1
FILTER TYPE=IMAGES STATUS=ON
SET !ERRORIGNORE YES
TAG POS=1 TYPE=A ATTR=TXT:Sign<SP>Out
TAG POS=1 TYPE=A ATTR=TXT:Sign<SP>in*
WAIT SECONDS=2
TAG POS=1 TYPE=A ATTR=TXT:Sign<SP>in<SP>as<SP>a*
SET !ERRORIGNORE NO
WAIT SECONDS=2
TAG POS=1 TYPE=INPUT:TEXT FORM=NAME:login_form ATTR=NAME:login CONTENT={{!VAR1}}
SET !ENCRYPTION NO
TAG POS=1 TYPE=INPUT:PASSWORD ATTR=ID:passwd CONTENT={{!VAR2}}
TAG POS=1 TYPE=INPUT:SUBMIT ATTR=VALUE:Sign<SP>In
FILTER TYPE=IMAGES STATUS=OFF
URL GOTO=http://www.flickr.com/activity
TAG POS=1 TYPE=SELECT FORM=* ATTR=ID:act-since CONTENT=%lastlogin


Script Ends

Latest changes:
  1. I added the 2 second wait times as Sxipper (a password plugin) seemed to be causing 'focus' problems.
  2. ErrorIgnore is used so that the script still works (though slower) if you have logged off but are still on the Flickr site.

Wednesday, October 22, 2008

Leopard problems with OpenOffice

Since installing an update from Apple to my X11 application, OpenOffice has refused to run. Damned annoying that Apple's development labs can't test updates properly.

Luckily I found a solution on a technical forum of running NeoOffice instead. NeoOffice uses exactly the same open source as Open Office, so saved files are completely interchangeable with OpenOffice, with the odd additional feature thrown in. NeoOffice runs natively on Leopard and so doesn't rely on X11. So far it seems to run rather better than Open Office and I have had great fun mapping out my kitchen using the drawing application (very similar features to Visio) which includes the facility to draw to scale, so my kitchen units can be represented quite accurately for my workmen to follow.

Next time I'm running updates from Apple I'll research the forums before rashly saying yes to new bugs.

Tuesday, September 9, 2008

Stumbling across some MacMini tips/tricks

Tip 1 - Booting Puppy from CDScreen capture of 9 desktops on MacMini with Puppy Linux running in middle desktop
Having a good play around with Puppy Linux led me to getting this to work within Parallels running on Leopard with the MacMini (intel core duo). It succeeded quite well, neatly running Puppy directly from the ISO file, though it's a bit freaky being able to flick between 9 desktops on the Mac and a further 4 running on Puppy (note that my machine has been upgraded to 2GB of RAM so reserving 500MB for Puppy Linux was no problem). The only drawback was that Compiz Fusion would not run within Parallels (I really wanted to see the 3D cube working). I decided to try running from CD direct.

Booting from CD was not an obvious process as unlike a PC the Mac will not boot automatically from CD just because it's in the drive. The trick is to keep the Option key pressed when your machine powers up. It then checks bootable devices and gives you a neat graphic display of the options. Puppy then ran perfectly well including all the lovely 3D Compiz goodness. Though it's not a replacement for me working on Leopard it could be a really handy backup if I run into hard disk problems.
Footnotes:
  • I'm unsure if there's a way to get the boot up to work straight from a USB drive, this doesn't seem to be part of the standard boot up routine.
  • Parallels doesn't seem to let me share the Mac hard disk or USB drives with Puppy though I haven't spent any time researching a workaround.
  • The benefit of taking virtual machine snapshots is a key bonus, I can install options and change settings on puppy with the confidence that I can roll back to a happy working state.

Tip 2 - Ejecting a CD
While creating Puppy bootable CDs I had a problem with one not being read properly on the Mac (probably a scratched disk). Annoyingly it would not eject using software control. This included trying to get it to eject using direct commands from within Terminal. My final solution was to keep F12 pressed on the keyboard when powering up, magically the disk ejected. It seems an amazingly user unfriendly feature to have no manual eject process, not even a pin hole to work a paper-clip into, particularly on a machine which is so notoriously difficult to force open the case. From now on, I'm being extra careful to ensure disks look spotless before inserting them.

Saturday, September 6, 2008

How to breathe life back into a 10 year old Compaq laptop using Xubuntu

Many years ago I lent an old slightly damaged laptop to a friend (the laptop locking mechanism was broken and the screen slightly damaged). Last year he bought a cheap but highly functional laptop after my loaner, now nearly ten years old, stopped working and eventually returned it to me for possible repair and use.

It is a Compaq E500 which has a rather stylish black and grey layout, good quality keyboard and a clear colour screen (in 1999 it was retailing for $1,970 and in 2008 they sell on eBay for under $100). The hard disk had fatally crashed and windows now failed to boot up only ever giving a blue screen of death. I'm unsure if the hard disk has a physical problem but after it laying around on my study floor for six months I thought it might be possible to replace Windows with a suitable free small version of Linux so the machine could become a handy wireless web laptop and guest machine. [E500 Spec: 14.1 in 1024x786 TFT display (1600x1200 external), 12GB hard disk, CDROM 24x, Video Out, Pentium III 650MHz, 192MB SDRAM.]

After failing to get the standard Ubuntu to install from a CD I had to hand (I gave up after letting the disk whizz around for over a half hour) I did a little research and opted for Xubuntu which is a cut down version for machines with more limited resources. I downloaded and created a CD from the ISO file on my trusty MacMini. The E500 install took over an hour but the main screen gave a countdown bar so you can see something is happening. I re-booted when it told me and it worked first time. It boots up more quickly than it used to under Windows and had even recognized my 2GB USB flash drive!

After a little play to ensure that it seemed responsive enough I dug out an old Belkin wireless card (802.11b) and was quite astonished when I plugged it in and the OS used it to find wireless connections without any further configuration. I had to tweak the wifi settings to connect to my secure network but had it working as a wireless laptop fairly painlessly.1

It's now a smart "netbook" (with a much better screen than the Acer Aspire One) for checking email while watching TV. The remaining issues are hardware specific2 and I'm not sure I'll get around to fixing them, e.g. recognizing the function key buttons for changing brightness and volume & the second touch pad key doesn't do anything. All in all I'm pretty delighted to turn a useless brick into a nicely functional spare laptop.
Footnotes

[1] The old Belkin wireless card is so out of date that it can't support an encrypted connection so security on this laptop is reliant on screening by MAC address. In the process of testing it out I found wicd is a neat free Linux graphical tool for sniffing out and managing your wireless connection. I have just bought a cheap 802.11g card off eBay to solve the security problem.

[2] The hardware side remains a puzzle. The E500's BIOS is dated 1999 and there's a new BIOS that helps to solve the problem of buttons being available to non-Windows OS's (see discussion). Having reformatted the disk I can't install the new BIOS to flash (using a .exe file that HP have available) or (for some reason I can't fathom) run DOS from a floppy even with it set as the first boot device.

Postscript

I have tried booting from CD using the latest Puppy Linux (Dingo). It runs well, possibly faster than Xubuntu but I haven't managed to map the keyboard touchpad correctly yet. If I get this working I'll give it good trial as Xubuntu tends to take a couple of seconds to change screen tabs when running Opera, while I haven't noticed any real performance problems with Puppy Linux so far. When you think about it, it's pretty impressive to be able to run the latest version of the Opera browser on such an old machine.

Saturday, August 9, 2008

How to add 8GB of memory to Sony Ericsson K750i

My previous memory stick in my mobile phone was a 4GB MS Pro Duo card. This is the maximum the phone is supposed to be able to cope with. I took a slight risk and paid £23 for an 8GB MS2 card (with Pro Duo adapter) with the intent of extending my mobile's life as a thrifty guy's iPod.

Initially there seemed to be a problem reliably writing files but after re-formatting and using a card reader (rather than the phone cable) to update files, it seems to do the job. I formatted using DOS with the command FORMAT E: /FS:FAT32 /A:4096, check this discussion as to why I chose these options. The phone status only reports a 4GB card but it actually does cope with 8GB.

I'm amazed to fit such a large card on a three year old mobile phone. This is enough space to take a large MP3 music collection (about 50 CD's worth of music, including 15 hours of Wagner's Ring of the Nibelung), 40 podcast subscriptions, a couple of free audio books, various maps supporting my GPS bluetooth gadget, document backups and 2.5GB spare for future growth.

For document backups I use TrueCrypt to keep an encrypted 400MB disk image which can be mounted on my Mac or a PC that includes all my scanned documents over the last two years (I use a Fujitsu ScanSnap scanner to keep PDF versions of bank statements, tax bills etc.) as well as other key reference documents. This may seem an odd thing to keep on the phone but if I was on holiday and my laptop got stolen I could still print out identity documents if someone lent me a PC. By using encryption if the phone was stolen at least I wouldn't worry about my ID being compromised.

My old 4GB memory card? This is now in my pocket camera (Sony T100) giving enough space for 3 hours of 640x480 video or 1,600 photos at 2592x1944 (5MB) resolution.

Thursday, August 7, 2008

How to write a spreadsheet formula for ISBN-10 to ISBN-13 and ASIN

I wanted to convert 10 digit ISBNs to the new 13 digit format and vice versa for my "checking second-hand book prices given an ISBN" spreadsheet. It turns out that (for the moment) any ISBN-13 starting with 978 can be converted to an ISBN-10 and this will be the same as the ASIN used by Amazon, so this can be used to look up their data. The problem is that the last digit of the ISBN may change due to it being calculated using a different ISBN standard, so to convert from one to the other your spreadsheet needs to do the checksum calculation.

The following formulae are based on a 13 or 10 digit ISBN in cell A3 and cell A6 reserved for a check digit calculation. These should be in one line in a spreadsheet cell, the layout here is to help readability.

Formula to create ISBN-13 from ISBN-10
=if(
len(A3)<11,
concatenate(
"978",
mid(A3,1,9),
mod(
mid(A3,1,1)*3+
mid(A3,2,1)+
mid(A3,3,1)*3+
mid(A3,4,1)+
mid(A3,5,1)*3+
mid(A3,6,1)+
mid(A3,7,1)*3+
mid(A3,8,1)+
mid(A3,9,1)*3,
10
)
),
A3
)

Formula to create ISBN-10 from ISBN-13 (starting with 978)
=if(
len(A3)>10,
concatenate(
mid(A3,4,9),
if(A6=10,"X",A6)
),
A3
)

To calculate the check digit (A6)
=if(
len(A3)>10,
mod(
mid(A3,4,1)+
mid(A3,5,1)*2+
mid(A3,6,1)*3+
mid(A3,7,1)*4+
mid(A3,8,1)*5+
mid(A3,9,1)*6+
mid(A3,10,1)*7+
mid(A3,11,1)*8+
mid(A3,12,1)*9,
11
),
""
)

Tuesday, May 27, 2008

Track Flickr group membership history using iMacro


User Guide
This script keeps a running log (every time you execute it), by updating a discussion message and/or saving a snapshot of group memberships. For the running log you have to create a message in each Flickr group you are an administrator for (or specifically declare), called "Member count". This log keeps a historical record of "date, membership total, total number of items". Before adding a new log entry it checks if the last entry (and the one before that) was less than a fortnight ago and updates it semi-intelligently.
Why?
I created this script as there is no easy alternative for keeping a history of membership for restricted groups in Flickr.
NotesI'm an administrator for 9 Flickr groups and maintain membership logs in 4 other groups. Watching the macro chug along updating the logs is quite satisfying. It takes 10 seconds per group (loading 3 to 4 web pages each time and added waiting time, to avoid possible page freezes, composes most of that time) but this can increase up to three-fold depending on Flickr traffic. For a macro that I only need to run once every one or two weeks that doesn't seem too shabby.

[Source code now maintained at Sourceforge.net]

iMacro setupiMacro installs itself with some example JavaScript files (.js) which are visible in the folders that iMacro shows you in Firefox.

If you find the location of these folders on your hard disk (on the Mac these are under /<your_username>/iMacros/Macros/) and then save the script in this location with a ".js" extension you should be able to see the file when you refresh the iMacro folder list in Firefox.

You should then be able to play the script from within iMacro (you have to already be logged in to your Flickr account and have created "Member count" discussion topics for groups you want to log). In the first instance I suggest you try the test test version which will run fully and show you the intended changes in the detailed change log but will not actually commit them.

See wiki.imacros.net for further help.

Monday, May 26, 2008

Flickr iMacro script to invite all your Flickr friends to a group

There was no automatic way to invite all your friends to join a group so I knocked up this quick script to take you from any group main page to the invite page, extract your list of friends (and no other contacts) and then send off the invite. The invite only goes to friends not yet in the group and not previously invited so you will not be bugging them with unwanted spam.

Note that more than 500 friends may result in time-out errors waiting for Flickr to respond. In this situation you may be better off inviting a few hundred at a time by recording an iMacro selecting the first checkbox on the invite page, then generalize it by removing the ATTR=NAME: and copy that line 200 times but change POS=1 to POS=R1 so that the next checkbox relative to the cursor position is ticked each time. When you've trimmed to a more manageable number then this script can apply again.

[JavaScript source code START]
/*
Description:
JavaScript for iMacro to create a block invite for all your
friends (but not family or other contacts) to join a Flickr group.
For more information see http://useroffline.blogspot.com

Assumptions:
- You have iMacro installed.
- You have your group's main page currently displayed.

Why:
There is no block invite facility built into Flickr.
*/

// Get friends list
var friendArr=new Array();
iimPlay("CODE:\
TAG POS=1 TYPE=A ATTR=TXT:Invite\n\
TAG POS=1 TYPE=A ATTR=TXT:Choose<SP>from<SP>your<SP>contacts?\n\
TAG POS=5 TYPE=TR EXTRACT=HTM\
");
// Tidy up extracted data
friendArr=iimGetLastExtract(1).split("name=\"");
friendArr.splice(0,1);
for(i in friendArr){
friendArr[i]=friendArr[i].split("\"")[0];
friendArr[i]="TAG POS=1 TYPE=INPUT:CHECKBOX ATTR=NAME:"+friendArr[i]+" CONTENT=YES";
}
iimPlay("CODE:"+
"' Total no. of friends = "+friendArr.length+"\n"+
friendArr.join("\n"));
// Send the message
iimPlay("CODE:\
TAG POS=1 TYPE=INPUT:SUBMIT ATTR=NAME:Submit&&VALUE:PREVIEW*\n\
TAG POS=1 TYPE=INPUT:SUBMIT ATTR=NAME:done_send&&VALUE:SEND*\
");

[JavaScript source code END]

Thursday, May 22, 2008

Re-list books on Amazon using iMacro

User GuideThis script imports my Google spreadsheet of books and then pastes them to sell on Amazon books. It takes about 20 seconds to list each book but you only process those in your spreadsheet marked to be re-listed.
Example Google Spreadsheet.
Why?Unless you pay £28.75 per month to Amazon Books to become a professional seller, you don't get the facility to import from a spreadsheet. This iMacro code does exactly that, for free, though it is of course intended for smallish quantities (in practice I'm normally adding less than 6 books at a time and have less than 20 books up for sale).
Having recently realized that the handful of books I was trying to sell had not only all expired but had also permanently gone from my closed listings so I had to put them in from scratch, I had the incentive to spend a few hours playing with JavaScript and iMacro to find a decent work around.
Known Bugs
When I have more than 20 books the last ones on the list are ignored as already listed, probably due to export limitations with iMacro. My work around is to list new books at the top of the spreadsheet and it will work fine. You could also cut and paste previously listed books into a second page of the spreadsheet and keep the front page for new books to be listed.

Sunday, May 18, 2008

How to add a book title from an ISBN or ASIN in a Google spreadsheet

Having recently created an iMacro to import books from Google spreadsheet to Amazon, I realized that the formula I was re-using to automatically put in the book title was quite a neat feature of Google docs that most people are probably unaware of.

I use the importXML function. This finds data based on an HTML tag in any webpage and you can create that source webpage name as a formula. This gives a really neat way of cribbing data about a book straight from it's Amazon webpage to be used on your own spreadsheet (also known as website scraping).

Here's my example formula (assuming the ISBN or ASIN is at cell A1), read this as one line:
=importXML( concatenate( "http://www.amazon.co.uk/exec/obidos/ASIN/", A1), "//span[@id='btAsinTitle']")

By looking at the webpage source code for any Amazon book you can find HTML tags around all of the product data. In this case the HTML tag <span> with the ID attribute set to "btAsinTitle" surrounds the book title. If you wanted to grab data on the number of pages, publisher or even lowest second hand price, this is all possible in a similar manner.

I have also created neat formulas to grab best match full film title and release date from IMDB.com as a search sheet to support my DVD index so I can quickly cut & paste details based on typing in key words from the film title.

The only limitation is there is a maximum of 50 such formulas in any spreadsheet. However if you are building a catalogue or sales list, it is a fairly simple matter to cut & paste the values over the formulas for books or DVDs already indexed so only new entries use the formula.

Sunday, April 27, 2008

How to find the cheapest train tickets by website scraping

One of my niggles with train timetable websites is that it can be very tricky to find the cheapest train ticket if you want to ask questions like "What is the cheapest ticket if I could travel on Sunday, Monday or Tuesday next month?" or even worse "What would be the cheapest 1st class ticket if can travel any weekend in June?". If you phone up a train hotline they would probably give up (and you are normally paying 10p/min while they try) or they would guess and might miss the best deal.

My solution was to use iMacro. This is a macro recording plugin for Firefox that can automate interactions with a website. I can now set up an initial script and set the browser to look for the cheapest possible ticket.

For example, below are the results for London to Nottingham travelling in the mornings (9am to 1pm) from 3rd to 5th May searching for the cheapest direct train ticket in any 2 hour slot. It took 20 seconds to configure the script (after taking a day to develop it) and my computer took 5 minutes to run the search as it has to enter data on the timetable website 12 times (3 days x 2 time slots x two classes of travel) and wait each time for the website to refresh with new data. I chose the Virgin Trains website as it seemed to have quite a nice interface which made it easy to pick the first radio button to find the cheapest ticket and seemed to return the most comprehensive results (I kept finding discrepancies on other sites, which surprised me as I assumed they shared the same underlying data).
Results for London to Nottingham 
[click to show longer example]

Searched on Sun, 27 Apr 2008 12:44:50 GMT

Saturday 3 May 2008
Slot | Time | Std | Time | 1st
09:00 | 08:55 | £ 15.00 | 08:55 | £ 18.00
11:00 | 12:55 | £ 15.00 | 11:55 | £ 18.00

Sunday 4 May 2008
Slot | Time | Std | Time | 1st
09:00 | 09:00 | £ 15.00 | 09:00 | £ 18.00
11:00 | 11:00 | £ 15.00 | 11:00 | £ 18.00

Monday 5 May 2008
Slot | Time | Std | Time | 1st
09:00 | 08:55 | £ 11.00 | 08:55 | £ 18.00
11:00 | 10:55 | £ 11.00 | 10:55 | £ 18.00
I have tested with very long queries taking more than an hour to run and the only problem may be time-outs from the website. In these cases iMacro appears to lock but by pressing the Pause/Resume button will continue without losing data. Note it is best not to use the browser when running an iMacro script but I have successfully used a different browser (Safari) at the same time without any problems.

The macro is run from a javascript file in iMacro which calls iMacro commands (previously I passed variables to an iim file but it seems easier to put it all in one file) and I tweak the "Set up query" section of the javascript to query a particular train route, this could easily be set from user prompts. For the time being it is always cheapest to get two singles rather than a return so I've only bothered writing the macro for a single. Note that the slot hours (set in array myhour) are based on the maximum number of hours that Virgin trains will display in the particular train route. For London/Nottingham this is 2 hours and for London/Redruth this is 3 hours. Nice bonus functionality I've included are returning the date in long text form from the Virgin site and keeping the iMacro code display updated telling you the estimated time left.

The latest version opens a results window to show the data in an html table. You can use iimDisplay() but it is limited to a tiny window and was (at the time of writing) not resizeable by scripting.

Here's the source code. Note that the source code is not word wrapped and long lines may appear truncated, but if you cut & paste to your editor you should see all the text.

JavaScript source code Virgin-Trains.js (click to show)

Monday, April 14, 2008

How to use free GPS on a K750i mobile phone

Last year I bought a tiny bluetooth GPS device off ebay for £20. I set up my laptop with it and we managed to drive around Germany using a copy of Autoroute showing our planned route, current GPS coordinates and it even shouted helpful instructions such as "turn left in 400 meters"! We planned our route in the hotel the night before and as we are often looking for poorly road-signed ancient sites it is terribly handy to have the confidence of the computer pointing you down an empty track rather than guesswork.

I did get the GPS recognized by my Sony Ericsson K750i (due to a previous hack it runs using the W800i firmware) and used some trial software to show coordinates but little else. I spent some time yesterday further researching and managed to get it working quite beautifully yesterday.

The simple free java application is called TrekBuddy and it has an associated wiki site. After connecting to the GPS via bluetooth, the application shows where you are on a scrolling map that you define and store locally on the memory stick. I have it working with an almost A to Z street level map of London (spanning West Hammersmith through to Catford). You can load other maps as you go along so a later improvement would be to install a higher level road map of the UK. In the case of my map of London it was only 2mb in size (I use a 4gb stick!) so it would be no issue to carry around several maps.

I had to spend some time tweaking the configuration, in particular the initial location of the maps had to be entered by hand to point to the memory stick ("file:///E:/other/mapdata/london") and it took a while to work out that I needed to tick the option to enable a large atlas in order to display anything. The only other hard part was preparing the map, luckily someone has worked all this out and I imported maps from Open Street Map (setting default datum as WGS 84) using the tool listed on the wiki. I also managed to import using the rather neat Google Maps to TrekBuddy tool on the same page. The latter being slightly more limited as to the overall size of map but with the benefits of a better quality street map (just right for a pedestrian map in Central London).

Thursday, February 14, 2008

Talktalk (Carphone) compromised my password!

I had been getting mysterious emails to someone else that I thought was spam nonsense. This morning, after getting a few about membership of the same club, I emailed the club secretary back suggesting they had the wrong email (assuming some sort of typo).

This afternoon Talktalk Technical Support phoned me to apologise for giving out my password! Over Christmas my email had stopped working and I called to check why and they "reset" my account somehow and I agreed a new default password. I tried logging in afterwards to the online account manager but had lost the password they send you in the post to set up your account and after hanging on the 0870 number for a couple of minutes I gave up and left the password as their default random looking one. It turns out that a new customer in January had been given my email account details and the password as their account. Consequently that customer has been logging into my email account over the last few weeks, probably thinking that email to me was spam and sending email from my address!

I can't believe that a company can be so cavalier about customer privacy. Needless to say, I've gone online and changed the password again from the new one agreed on the phone.

Lessons learned:
  1. Never send anything confidential by email unencrypted (at least put a password on a document)
  2. Always change passwords from defaults or if given temporarily to someone else
  3. Stick to more secure mixed numbers and letters passwords of a decent length (8 characters or more), a tool like Sxipper can be helpful if you have to keep switching between accounts
  4. If it's important then digitally sign it
  5. Expect companies (like Carphone) to give away your details by accident and know who to contact if you think something has been compromised

Tuesday, February 12, 2008

Mac mini - six things they did not tell you about switching from PC to Macintosh

Last November, after my PC died for about the 4th time I knew there was something seriously wrong, probably something odd happening with the motherboard. I thought I'd treat myself to something different so I splashed out on a Mac mini (Core Duo).
Tip: If you are buying a Mac from Apple, consider the cost of memory upgrades. I would have opted for a 2GB Mac mini rather than the standard 1GB version but they wanted £80 for the privilege. The actual memory cost from Crucial.com is £32 and when I sold my old memory on eBay for £12 I realized I had saved myself £60!

Mostly it's worked as promised but the following painful annoyances rather caught me out:
  1. Parallels did not come as standard, something I really needed for running some of my old windows applications. My only other option would have been to reserve a major chunk of my hard disk and use dual booting, something that I've had real issues with in the past. I took a trial copy of Parallels and it works fairly well apart from being too slow. In fact so slow that in January I upgraded the memory (to 2GB), though as I'm techie enough to do this myself it was under 30 quid.

  2. Webcam did not work. This has been a headache. Not only did my Philips USB webcam not work, there are no valid drivers to make it work (I wasted hours trying to get a generic driver to work). Even worse, there are no guarantees that any other cheap camera I can find will actually work. I still don't have a replacement webcam.

  3. Scanner did not work. I have a rather expensive Fujitsu ScanSnap scanner. Unfortunately there are no drivers that will make it work natively in Leopard. I eventually got it running under Windows 2000 in Parallels, hardly ideal.

  4. EVE Online does not work. I tried out a free account with EVE Online (a well known and popular multi-player game for the Mac), sadly it doesn't run reliably on the Mac mini and freezes up randomly.

  5. Can't network with a PC. This is particularly annoying. I've been trying for several weeks to get this working. I have worked through inbuilt help and googled for solutions to no avail. I can get my Mac to recognize my home networked PC and even read/write files to its hard disk but not the reverse. I wanted this to work as I'm using a 500GB USB drive as my Time Machine and I've kept space on it for PC backups but as the Mac remains invisible to the PC my archive/backup process is frustratingly manual and I can't share a central file library.

  6. Can't use my microphone/headphone. This was not made clear in the summary specification and is a really stupid design flaw. The audio-in socket is actually only for digital microphones. This means that my headphone microphones will not work (though confusingly they fit the socket and it took me quite a while to find out this was not some sort of driver problem). Consequently since buying the Mac I have not been able to use Skype, a real problem as I use this as my home office number. I have been searching for a second-hand iMic device (they're a bit pricey new), a USB adaptor for microphones, I should be able to find one for under £15. The alternative would have been to get a bluetooth microphone, but I can't bear having yet another device to keep charged up.