How to find the cheapest train tickets by website scraping

One of my niggles with train timetable websites is that it can be very tricky to find the cheapest train ticket if you want to ask questions like "What is the cheapest ticket if I could travel on Sunday, Monday or Tuesday next month?" or even worse "What would be the cheapest 1st class ticket if can travel any weekend in June?". If you phone up a train hotline they would probably give up (and you are normally paying 10p/min while they try) or they would guess and might miss the best deal.

My solution was to use iMacro. This is a macro recording plugin for Firefox that can automate interactions with a website. I can now set up an initial script and set the browser to look for the cheapest possible ticket.

For example, below are the results for London to Nottingham travelling in the mornings (9am to 1pm) from 3rd to 5th May searching for the cheapest direct train ticket in any 2 hour slot. It took 20 seconds to configure the script (after taking a day to develop it) and my computer took 5 minutes to run the search as it has to enter data on the timetable website 12 times (3 days x 2 time slots x two classes of travel) and wait each time for the website to refresh with new data. I chose the Virgin Trains website as it seemed to have quite a nice interface which made it easy to pick the first radio button to find the cheapest ticket and seemed to return the most comprehensive results (I kept finding discrepancies on other sites, which surprised me as I assumed they shared the same underlying data).

Results for London to Nottingham 
[click to show longer example]
Searched on Sun, 27 Apr 2008 12:44:50 GMT

Saturday  3 May 2008
Slot  | Time  |   Std   | Time  |   1st
09:00 | 08:55 | £ 15.00 | 08:55 | £ 18.00
11:00 | 12:55 | £ 15.00 | 11:55 | £ 18.00

Sunday  4 May 2008
Slot  | Time  |   Std   | Time  |   1st
09:00 | 09:00 | £ 15.00 | 09:00 | £ 18.00
11:00 | 11:00 | £ 15.00 | 11:00 | £ 18.00

Monday  5 May 2008
Slot  | Time  |   Std   | Time  |   1st
09:00 | 08:55 | £ 11.00 | 08:55 | £ 18.00
11:00 | 10:55 | £ 11.00 | 10:55 | £ 18.00

London to Nottingham 
[click to show shorter example]
London to Nottingham
Virgin Trains Wed, 30 Apr 2008 23:26:26 GMT
Thursday 1 May 2008
Slot
Time
Std
Time
1st
08
09:55
£ 30.00
09:55
£ 30.00
10
09:55
£ 30.00
09:55
£ 30.00
12
11:55
£ 49.00
11:55
£ 86.00
14
13:55
£ 49.00
13:55
£ 86.00
16
15:55
£ 49.00
15:55
£ 86.00
18
19:30
£ 49.00
17:55
£ 86.00
Friday 2 May 2008
Slot
Time
Std
Time
1st
08
09:55
£ 22.00
09:55
£ 29.00
10
10:55
£ 18.00
10:55
£ 18.00
12
11:55
£ 18.00
11:55
£ 22.00
14
13:55
£ 22.00
13:55
£ 22.00
16
17:00
£ 45.00
17:00
£ 45.00
18
17:55
£ 45.00
17:55
£ 45.00
Saturday 3 May 2008
Slot
Time
Std
Time
1st
08
07:55
£ 18.00
07:55
£ 18.00
10
09:55
£ 22.00
09:55
£ 22.00
12
12:55
£ 15.00
12:55
£ 18.00
14
14:55
£ 15.00
13:55
£ 18.00
16
15:55
£ 15.00
15:55
£ 18.00
18
17:55
£ 18.00
17:55
£ 22.00
Sunday 4 May 2008
Slot
Time
Std
Time
1st
08
09:00
£ 18.00
09:00
£ 18.00
10
10:00
£ 18.00
10:00
£ 18.00
12
12:00
£ 18.00
12:00
£ 22.00
14
14:00
£ 22.00
14:00
£ 22.00
16
16:00
£ 22.00
16:00
£ 22.00
18
21:00
£ 18.00
20:00
£ 22.00
Monday 5 May 2008
Slot
Time
Std
Time
1st
08
07:55
£ 11.00
07:55
£ 18.00
10
09:55
£ 15.00
09:55
£ 22.00
12
12:55
£ 18.00
12:55
£ 22.00
14
13:55
£ 22.00
13:55
£ 29.00
16
15:55
£ 22.00
17:45
£ 22.00
18
17:55
£ 22.00
19:30
£ 22.00
Tuesday 6 May 2008
Slot
Time
Std
Time
1st
08
09:55
£ 15.00
09:55
£ 22.00
10
09:55
£ 15.00
09:55
£ 22.00
12
11:55
£ 15.00
11:55
£ 22.00
14
13:55
£ 18.00
£    
16
15:55
£ 49.00
15:55
£ 86.00
18
19:55
£ 22.00
19:55
£ 29.00
Wednesday 7 May 2008
Slot
Time
Std
Time
1st
08
09:55
£ 15.00
09:55
£ 22.00
10
09:55
£ 15.00
09:55
£ 22.00
12
11:55
£ 15.00
11:55
£ 22.00
14
13:55
£ 18.00
13:55
£ 29.00
16
15:55
£ 49.00
15:55
£ 86.00
18
19:55
£ 18.00
19:55
£ 29.00
Thursday 8 May 2008
Slot
Time
Std
Time
1st
08
09:55
£ 15.00
09:55
£ 22.00
10
09:55
£ 15.00
09:55
£ 22.00
12
11:55
£ 15.00
11:55
£ 22.00
14
13:55
£ 18.00
13:55
£ 29.00
16
15:55
£ 49.00
15:55
£ 86.00
18
19:55
£ 22.00
19:55
£ 29.00
Friday 9 May 2008
Slot
Time
Std
Time
1st
08
09:55
£ 18.00
09:55
£ 22.00
10
11:55
£ 11.00
11:55
£ 18.00
12
11:55
£ 11.00
11:55
£ 18.00
14
13:55
£ 18.00
13:55
£ 22.00
16
17:00
£ 45.00
17:00
£ 45.00
18
19:55
£ 40.00
17:55
£ 45.00
Saturday 10 May 2008
Slot
Time
Std
Time
1st
08
07:55
£  9.00
07:55
£ 14.00
10
11:55
£ 14.00
11:55
£ 14.00
12
12:55
£ 11.00
11:55
£ 14.00
14
13:55
£ 11.00
13:55
£ 14.00
16
15:55
£ 15.00
15:55
£ 18.00
18
17:55
£ 18.00
17:55
£ 18.00
Sunday 11 May 2008
Slot
Time
Std
Time
1st
08
09:00
£ 11.00
09:00
£ 14.00
10
10:00
£ 11.00
10:00
£ 14.00
12
12:00
£ 18.00
12:00
£ 22.00
14
14:00
£ 18.00
14:00
£ 22.00
16
16:00
£ 22.00
16:00
£ 22.00
18
21:00
£ 15.00
21:00
£ 18.00
Monday 12 May 2008
Slot
Time
Std
Time
1st
08
09:55
£  9.00
09:55
£ 18.00
10
09:55
£  9.00
09:55
£ 18.00
12
11:55
£ 49.00
11:55
£ 18.00
14
13:55
£ 15.00
13:55
£ 22.00
16
15:55
£ 40.00
15:55
£ 64.00
18
19:55
£ 18.00
19:55
£ 29.00
Tuesday 13 May 2008
Slot
Time
Std
Time
1st
08
09:55
£  9.00
09:55
£ 18.00
10
09:55
£  9.00
09:55
£ 18.00
12
11:55
£  9.00
11:55
£ 18.00
14
13:55
£ 15.00
13:55
£ 22.00
16
15:55
£ 40.00
15:55
£ 64.00
18
19:55
£ 18.00
19:55
£ 29.00
Wednesday 14 May 2008
Slot
Time
Std
Time
1st
08
09:55
£  9.00
09:55
£ 18.00
10
09:55
£  9.00
09:55
£ 18.00
12
11:55
£  9.00
11:55
£ 18.00
14
13:55
£ 15.00
13:55
£ 22.00
16
15:55
£ 40.00
15:55
£ 64.00
18
19:55
£ 18.00
19:55
£ 22.00
Search time 142 minutes 27 seconds

I have tested with very long queries taking more than an hour to run and the only problem may be time-outs from the website. In these cases iMacro appears to lock but by pressing the Pause/Resume button will continue without losing data. Note it is best not to use the browser when running an iMacro script but I have successfully used a different browser (Safari) at the same time without any problems.

The macro is run from a javascript file in iMacro which calls iMacro commands (previously I passed variables to an iim file but it seems easier to put it all in one file) and I tweak the "Set up query" section of the javascript to query a particular train route, this could easily be set from user prompts. For the time being it is always cheapest to get two singles rather than a return so I've only bothered writing the macro for a single. Note that the slot hours (set in array myhour) are based on the maximum number of hours that Virgin trains will display in the particular train route. For London/Nottingham this is 2 hours and for London/Redruth this is 3 hours. Nice bonus functionality I've included are returning the date in long text form from the Virgin site and keeping the iMacro code display updated telling you the estimated time left.

The latest version opens a results window to show the data in an html table. You can use iimDisplay() but it is limited to a tiny window and was (at the time of writing) not resizeable by scripting.

Here's the source code. Note that the source code is not word wrapped and long lines may appear truncated, but if you cut & paste to your editor you should see all the text.

JavaScript source code Virgin-Trains.js (click to show)Virgin-Trains.js (click to hide)

// Set up query

var myfirstday=10;
var mynodays=3;
var mymonth="05";
var myleaving="London";
var mygoing="Nottingham";
var myhour = new Array("09","11");
  /* Every 2 hours for Nottingham trains as they are every 30 mins */

/** No need to edit below this line if you only want to configure the query **
/************************************************************************************
  Author: http://useroffline.blogspot.com
  Created: 28 April 2008
  Last Edit: 1 May 2008
  Title: iMacro Virgin Trains timetable search

  Description:
  Fills out query form on Virgin Trains timetable website (covers all UK timetables)
  and extracts the cheapest standard and first class ticket prices and times in any
  timeslot (normally two or three hour slots depending on what the timetables look
  like from Virgin for the particular route).
  Results are displayed in a new browser window in an html table which is written to
  as the search completes each calendar day.

  Assumptions:
  1. Virgin trains site has no technical difficulties
  2. iMacro has 'view Javascript while running' option set to off (for better speed)
  3. Infrequent usage to avoid IP blocking by Virgin
  4. Browser will not be used for anything else while running the macro
  5. Route does not use the new timetable layout (Virgin is upgrading layout)

  Suggestions for improvement:
  o Add link to each train in results table to live timetable page (probably not
    Virgin)
************************************************************************************/

// Open results window

var cr = "\r\n";
top.results=window.open( "", "myWindow", "menubar=1,resizable=1,status=1,width=500,height=600,scrollbars=1");
top.results.document.write("<html><title>Train Timetable search results</title><head>"+cr+
  "<style>td {border-width:thin;border-color:black;-moz-border-radius: 8px 8px 8px 8px;}"+cr+
  "</style></head>"+cr+
  "<body bgcolor=lightgreen style='font-family:Tahoma, Arial;'>");
top.results.window.status="Initializing, time to make a cup of tea.";

// Variables

var c=0, j, price=" na ", count=0, mytime=" na ", longdate=" na ", pArr=new Array();
var classArr=new Array("Std","1st");
var startDate=new Date();
var ret=cr+"<h3>"+myleaving+" to "+mygoing+"</h3>"+cr+"<br>Virgin Trains "+startDate.toGMTString()+"<p>"+cr;
var retTemp, runTimeM, runTimeS, runTimeMS, timeLeftS, timeLeftM, loopTime;
var mydate = new Array;

// Pad dates ("1" becomes "01")

var i;
for (i=0;i < mynodays;i++) {
  mydate[i]=(myfirstday+i).toString();
  if (mydate[i].length < 2) { mydate[i]='0'+mydate[i] }
  }
var loopTotal=mydate.length*myhour.length*2;

// Initialize to avoid timeout errors

iimPlay("CODE:URL GOTO=http://www.virgintrains.co.uk");

// Main loop

ret+="<table class='results' bgcolor=lightyellow cellpadding=4>";
top.results.document.write(ret);
ret="";

/* Static parts of the code */
pArr[1]="URL GOTO=http://www.virgintrains.co.uk/default.aspx";
pArr[7]="TAG POS=1 TYPE=SELECT FORM=NAME:qtt ATTR=NAME:outMinuteField CONTENT=$00"+cr+
    "TAG POS=1 TYPE=INPUT:IMAGE FORM=NAME:qtt ATTR=ID:_ctl15_findTimes";
pArr[9]="TAG POS=1 TYPE=INPUT:RADIO FORM=NAME:Form ATTR=ID:*ServiceTypeRadioButtons_1"+cr+
    /* ServiceType 1 = direct trains only, 0 = Any train */
    "TAG POS=1 TYPE=INPUT:IMAGE FORM=NAME:Form ATTR=ALT:Check<SP>availability"+cr+
    "TAG POS=1 TYPE=INPUT:RADIO EXTRACT=TXT"+cr+
    "TAG POS=1 TYPE=TD ATTR=CLASS:tableCell EXTRACT=TXT"+cr+
    "TAG POS=2 TYPE=TD ATTR=CLASS:tableCell EXTRACT=TXT"+cr+
    "TAG POS=3 TYPE=TD ATTR=CLASS:tableCell EXTRACT=TXT"+cr+
    "TAG POS=4 TYPE=TD ATTR=CLASS:tableCell EXTRACT=TXT"+cr+
    "TAG POS=5 TYPE=TD ATTR=CLASS:tableCell EXTRACT=TXT"+cr+
    "TAG POS=1 TYPE=TD ATTR=CLASS:tableTitle EXTRACT=HTM"+cr+
    "WAIT SECONDS=4";  // Could help to avoid lock-up?

for (j in mydate) {
  retTemp='<tr align=center ><td bgcolor=lightgreen >Slot</td><td bgcolor=lightgreen >Time</td><td bgcolor=lightgreen >Std</td><td bgcolor=lightgreen >Time</td><td bgcolor=lightgreen >1st</td></tr>';
  for (i in myhour) {
  retTemp+='<tr align=center ><td bgcolor=lightgreen >'+myhour[i]+'</td>';
  for (c=0;c < 2;c++) {
    count++;
    /* Estimate time left */
    var now = new Date();
    runTimeMS=now.valueOf()-startDate.valueOf();
    runTimeS=parseInt(runTimeMS/1000);
    runTimeM=parseInt(runTimeS/60);
    runTimeS %=60;
    loopTime=runTimeMS/count;
    timeLeftS=parseInt((loopTotal+1-count)*loopTime/1000);
    timeLeftM=parseInt(timeLeftS/60);
    timeLeftS %=60;
    /* Results window status bar feedback */
    top.results.window.status = "Loop "+count+" of "+loopTotal+
    " [Last result "+longdate+" @ "+mytime+", "+classArr[1-c]+" = £"+price.replace(/ /g,"")+"]";
    /* Run iMacro */
    pArr[0]="CODE:"+"' "+runTimeM+" minutes "+runTimeS+" seconds current run time"+cr+
        "' "+timeLeftM+" minutes "+timeLeftS+" seconds estimate to complete";
    pArr[2]="TAG POS=1 TYPE=INPUT:TEXT FORM=NAME:qtt ATTR=ID:idLeaving CONTENT="+myleaving;
    pArr[3]="TAG POS=1 TYPE=INPUT:TEXT FORM=NAME:qtt ATTR=ID:idGoing CONTENT="+mygoing;
    pArr[4]="TAG POS=1 TYPE=SELECT FORM=NAME:qtt ATTR=NAME:outDay CONTENT=$"+mydate[j];
    pArr[5]="TAG POS=1 TYPE=SELECT FORM=NAME:qtt ATTR=NAME:outMth CONTENT=$"+mymonth;
    pArr[6]="TAG POS=1 TYPE=SELECT FORM=NAME:qtt ATTR=NAME:outHourField CONTENT=$"+myhour[i];
    pArr[8]="TAG POS=1 TYPE=INPUT:RADIO FORM=NAME:Form ATTR=ID:*TicketClassRadioButtons_"+c.toString();
    iimPlay(pArr.join(cr));
    /* Get long date */
    try {longdate=iimGetLastExtract(7).split('>')[2].replace(/  /g," ")}
    catch (err) {}
    /* Parse train time and price */
    price=iimGetLastExtract(1).substr(5+(iimGetLastExtract(1).substr(4)).search("_"));
    price='    '.substr(1,6-price.length)+price;      // Pad out to 6 characters
    mytime=parseInt(iimGetLastExtract(1).substr(0,1));  // Find the table column number of train time
    mytime=iimGetLastExtract(mytime+1);          // Get time in column header
    retTemp+='<td>'+mytime+'</td><td>£'+price+'</td>';
  }  /* end class */
  retTemp+='</tr>';
  }  /* end hour */
  ret+="<tr><td colspan=5 bgcolor=lightgreen >"+longdate+'</td></tr>'+retTemp;
  top.results.document.write(ret);
  ret='';
}  /* end date */

// Calculate total running time

var endDate=new Date();
var runTimeS=parseInt((endDate.valueOf()-startDate.valueOf())/1000);
var runTimeM=parseInt(runTimeS/60);
runTimeS %=60;
ret+='</table>'+cr+'<p>Search time '+runTimeM+' minutes '+runTimeS+' seconds';
ret+=cr+"<br>Created by <a href='http://useroffline.blogspot.com' title='Author details'>UserOffline</a> 2008";

// Display Results

top.results.window.status="Finished";
top.results.document.write(ret+cr+"</body></html>");
top.results.document.close();

How to use free GPS on a K750i mobile phone

Google high level map

Last year I bought a tiny bluetooth GPS device off ebay for £20. I set up my laptop with it and we managed to drive around Germany using a copy of Autoroute showing our planned route, current GPS coordinates and it even shouted helpful instructions such as "turn left in 400 meters"! We planned our route in the hotel the night before and as we are often looking for poorly road-signed ancient sites it is terribly handy to have the confidence of the computer pointing you down an empty track rather than guesswork.

Open Street Map

I did get the GPS recognized by my Sony Ericsson K750i (due to a previous hack it runs using the W800i firmware) and used some trial software to show coordinates but little else. I spent some time yesterday further researching and managed to get it working quite beautifully yesterday.

The simple free java application is called TrekBuddy and it has an associated wiki site. After connecting to the GPS via bluetooth, the application shows where you are on a scrolling map that you define and store locally on the memory stick. I have it working with an almost A to Z street level map of London (spanning West Hammersmith through to Catford). You can load other maps as you go along so a later improvement would be to install a higher level road map of the UK. In the case of my map of London it was only 2mb in size (I use a 4gb stick!) so it would be no issue to carry around several maps.

Google low level map

I had to spend some time tweaking the configuration, in particular the initial location of the maps had to be entered by hand to point to the memory stick ("file:///E:/other/mapdata/london") and it took a while to work out that I needed to tick the option to enable a large atlas in order to display anything. The only other hard part was preparing the map, luckily someone has worked all this out and I imported maps from Open Street Map (setting default datum as WGS 84) using the tool listed on the wiki. I also managed to import using the rather neat Google Maps to TrekBuddy tool on the same page. The latter being slightly more limited as to the overall size of map but with the benefits of a better quality street map (just right for a pedestrian map in Central London).

Talktalk (Carphone) compromised my password!

I had been getting mysterious emails to someone else that I thought was spam nonsense. This morning, after getting a few about membership of the same club, I emailed the club secretary back suggesting they had the wrong email (assuming some sort of typo).

This afternoon Talktalk Technical Support phoned me to apologise for giving out my password! Over Christmas my email had stopped working and I called to check why and they "reset" my account somehow and I agreed a new default password. I tried logging in afterwards to the online account manager but had lost the password they send you in the post to set up your account and after hanging on the 0870 number for a couple of minutes I gave up and left the password as their default random looking one. It turns out that a new customer in January had been given my email account details and the password as their account. Consequently that customer has been logging into my email account over the last few weeks, probably thinking that email to me was spam and sending email from my address!

I can't believe that a company can be so cavalier about customer privacy. Needless to say, I've gone online and changed the password again from the new one agreed on the phone.

Lessons learned:

1. Never send anything confidential by email unencrypted (at least put a password on a document)
2. Always change passwords from defaults or if given temporarily to someone else
3. Stick to more secure mixed numbers and letters passwords of a decent length (8 characters or more), a tool like Sxipper can be helpful if you have to keep switching between accounts
   
4. If it's important then digitally sign it
5. Expect companies (like Carphone) to give away your details by accident and know who to contact if you think something has been compromised
   

Mac mini - six things they did not tell you about switching from PC to Macintosh

Last November, after my PC died for about the 4th time I knew there was something seriously wrong, probably something odd happening with the motherboard. I thought I'd treat myself to something different so I splashed out on a Mac mini (Core Duo).

Tip: If you are buying a Mac from Apple, consider the cost of memory upgrades. I would have opted for a 2GB Mac mini rather than the standard 1GB version but they wanted £80 for the privilege. The actual memory cost from Crucial.com is £32 and when I sold my old memory on eBay for £12 I realized I had saved myself £60!

Mostly it's worked as promised but the following painful annoyances rather caught me out:

1. Parallels did not come as standard, something I really needed for running some of my old windows applications. My only other option would have been to reserve a major chunk of my hard disk and use dual booting, something that I've had real issues with in the past. I took a trial copy of Parallels and it works fairly well apart from being too slow. In fact so slow that in January I upgraded the memory (to 2GB), though as I'm techie enough to do this myself it was under 30 quid.


2. Webcam did not work. This has been a headache. Not only did my Philips USB webcam not work, there are no valid drivers to make it work (I wasted hours trying to get a generic driver to work). Even worse, there are no guarantees that any other cheap camera I can find will actually work. I still don't have a replacement webcam.


3. Scanner did not work. I have a rather expensive Fujitsu ScanSnap scanner. Unfortunately there are no drivers that will make it work natively in Leopard. I eventually got it running under Windows 2000 in Parallels, hardly ideal.


4. EVE Online does not work. I tried out a free account with EVE Online (a well known and popular multi-player game for the Mac), sadly it doesn't run reliably on the Mac mini and freezes up randomly.


5. Can't network with a PC. This is particularly annoying. I've been trying for several weeks to get this working. I have worked through inbuilt help and googled for solutions to no avail. I can get my Mac to recognize my home networked PC and even read/write files to its hard disk but not the reverse. I wanted this to work as I'm using a 500GB USB drive as my Time Machine and I've kept space on it for PC backups but as the Mac remains invisible to the PC my archive/backup process is frustratingly manual and I can't share a central file library.


6. Can't use my microphone/headphone. This was not made clear in the summary specification and is a really stupid design flaw. The audio-in socket is actually only for digital microphones. This means that my headphone microphones will not work (though confusingly they fit the socket and it took me quite a while to find out this was not some sort of driver problem). Consequently since buying the Mac I have not been able to use Skype, a real problem as I use this as my home office number. I have been searching for a second-hand iMic device (they're a bit pricey new), a USB adaptor for microphones, I should be able to find one for under £15. The alternative would have been to get a bluetooth microphone, but I can't bear having yet another device to keep charged up.