How to find the cheapest train tickets by website scraping

One of my niggles with train timetable websites is that it can be very tricky to find the cheapest train ticket if you want to ask questions like "What is the cheapest ticket if I could travel on Sunday, Monday or Tuesday next month?" or even worse "What would be the cheapest 1st class ticket if can travel any weekend in June?". If you phone up a train hotline they would probably give up (and you are normally paying 10p/min while they try) or they would guess and might miss the best deal.

My solution was to use iMacro. This is a macro recording plugin for Firefox that can automate interactions with a website. I can now set up an initial script and set the browser to look for the cheapest possible ticket.

For example, below are the results for London to Nottingham travelling in the mornings (9am to 1pm) from 3rd to 5th May searching for the cheapest direct train ticket in any 2 hour slot. It took 20 seconds to configure the script (after taking a day to develop it) and my computer took 5 minutes to run the search as it has to enter data on the timetable website 12 times (3 days x 2 time slots x two classes of travel) and wait each time for the website to refresh with new data. I chose the Virgin Trains website as it seemed to have quite a nice interface which made it easy to pick the first radio button to find the cheapest ticket and seemed to return the most comprehensive results (I kept finding discrepancies on other sites, which surprised me as I assumed they shared the same underlying data).

Results for London to Nottingham 
[click to show longer example]
Searched on Sun, 27 Apr 2008 12:44:50 GMT

Saturday  3 May 2008
Slot  | Time  |   Std   | Time  |   1st
09:00 | 08:55 | £ 15.00 | 08:55 | £ 18.00
11:00 | 12:55 | £ 15.00 | 11:55 | £ 18.00

Sunday  4 May 2008
Slot  | Time  |   Std   | Time  |   1st
09:00 | 09:00 | £ 15.00 | 09:00 | £ 18.00
11:00 | 11:00 | £ 15.00 | 11:00 | £ 18.00

Monday  5 May 2008
Slot  | Time  |   Std   | Time  |   1st
09:00 | 08:55 | £ 11.00 | 08:55 | £ 18.00
11:00 | 10:55 | £ 11.00 | 10:55 | £ 18.00

London to Nottingham 
[click to show shorter example]
London to Nottingham
Virgin Trains Wed, 30 Apr 2008 23:26:26 GMT
Thursday 1 May 2008
Slot
Time
Std
Time
1st
08
09:55
£ 30.00
09:55
£ 30.00
10
09:55
£ 30.00
09:55
£ 30.00
12
11:55
£ 49.00
11:55
£ 86.00
14
13:55
£ 49.00
13:55
£ 86.00
16
15:55
£ 49.00
15:55
£ 86.00
18
19:30
£ 49.00
17:55
£ 86.00
Friday 2 May 2008
Slot
Time
Std
Time
1st
08
09:55
£ 22.00
09:55
£ 29.00
10
10:55
£ 18.00
10:55
£ 18.00
12
11:55
£ 18.00
11:55
£ 22.00
14
13:55
£ 22.00
13:55
£ 22.00
16
17:00
£ 45.00
17:00
£ 45.00
18
17:55
£ 45.00
17:55
£ 45.00
Saturday 3 May 2008
Slot
Time
Std
Time
1st
08
07:55
£ 18.00
07:55
£ 18.00
10
09:55
£ 22.00
09:55
£ 22.00
12
12:55
£ 15.00
12:55
£ 18.00
14
14:55
£ 15.00
13:55
£ 18.00
16
15:55
£ 15.00
15:55
£ 18.00
18
17:55
£ 18.00
17:55
£ 22.00
Sunday 4 May 2008
Slot
Time
Std
Time
1st
08
09:00
£ 18.00
09:00
£ 18.00
10
10:00
£ 18.00
10:00
£ 18.00
12
12:00
£ 18.00
12:00
£ 22.00
14
14:00
£ 22.00
14:00
£ 22.00
16
16:00
£ 22.00
16:00
£ 22.00
18
21:00
£ 18.00
20:00
£ 22.00
Monday 5 May 2008
Slot
Time
Std
Time
1st
08
07:55
£ 11.00
07:55
£ 18.00
10
09:55
£ 15.00
09:55
£ 22.00
12
12:55
£ 18.00
12:55
£ 22.00
14
13:55
£ 22.00
13:55
£ 29.00
16
15:55
£ 22.00
17:45
£ 22.00
18
17:55
£ 22.00
19:30
£ 22.00
Tuesday 6 May 2008
Slot
Time
Std
Time
1st
08
09:55
£ 15.00
09:55
£ 22.00
10
09:55
£ 15.00
09:55
£ 22.00
12
11:55
£ 15.00
11:55
£ 22.00
14
13:55
£ 18.00
£    
16
15:55
£ 49.00
15:55
£ 86.00
18
19:55
£ 22.00
19:55
£ 29.00
Wednesday 7 May 2008
Slot
Time
Std
Time
1st
08
09:55
£ 15.00
09:55
£ 22.00
10
09:55
£ 15.00
09:55
£ 22.00
12
11:55
£ 15.00
11:55
£ 22.00
14
13:55
£ 18.00
13:55
£ 29.00
16
15:55
£ 49.00
15:55
£ 86.00
18
19:55
£ 18.00
19:55
£ 29.00
Thursday 8 May 2008
Slot
Time
Std
Time
1st
08
09:55
£ 15.00
09:55
£ 22.00
10
09:55
£ 15.00
09:55
£ 22.00
12
11:55
£ 15.00
11:55
£ 22.00
14
13:55
£ 18.00
13:55
£ 29.00
16
15:55
£ 49.00
15:55
£ 86.00
18
19:55
£ 22.00
19:55
£ 29.00
Friday 9 May 2008
Slot
Time
Std
Time
1st
08
09:55
£ 18.00
09:55
£ 22.00
10
11:55
£ 11.00
11:55
£ 18.00
12
11:55
£ 11.00
11:55
£ 18.00
14
13:55
£ 18.00
13:55
£ 22.00
16
17:00
£ 45.00
17:00
£ 45.00
18
19:55
£ 40.00
17:55
£ 45.00
Saturday 10 May 2008
Slot
Time
Std
Time
1st
08
07:55
£  9.00
07:55
£ 14.00
10
11:55
£ 14.00
11:55
£ 14.00
12
12:55
£ 11.00
11:55
£ 14.00
14
13:55
£ 11.00
13:55
£ 14.00
16
15:55
£ 15.00
15:55
£ 18.00
18
17:55
£ 18.00
17:55
£ 18.00
Sunday 11 May 2008
Slot
Time
Std
Time
1st
08
09:00
£ 11.00
09:00
£ 14.00
10
10:00
£ 11.00
10:00
£ 14.00
12
12:00
£ 18.00
12:00
£ 22.00
14
14:00
£ 18.00
14:00
£ 22.00
16
16:00
£ 22.00
16:00
£ 22.00
18
21:00
£ 15.00
21:00
£ 18.00
Monday 12 May 2008
Slot
Time
Std
Time
1st
08
09:55
£  9.00
09:55
£ 18.00
10
09:55
£  9.00
09:55
£ 18.00
12
11:55
£ 49.00
11:55
£ 18.00
14
13:55
£ 15.00
13:55
£ 22.00
16
15:55
£ 40.00
15:55
£ 64.00
18
19:55
£ 18.00
19:55
£ 29.00
Tuesday 13 May 2008
Slot
Time
Std
Time
1st
08
09:55
£  9.00
09:55
£ 18.00
10
09:55
£  9.00
09:55
£ 18.00
12
11:55
£  9.00
11:55
£ 18.00
14
13:55
£ 15.00
13:55
£ 22.00
16
15:55
£ 40.00
15:55
£ 64.00
18
19:55
£ 18.00
19:55
£ 29.00
Wednesday 14 May 2008
Slot
Time
Std
Time
1st
08
09:55
£  9.00
09:55
£ 18.00
10
09:55
£  9.00
09:55
£ 18.00
12
11:55
£  9.00
11:55
£ 18.00
14
13:55
£ 15.00
13:55
£ 22.00
16
15:55
£ 40.00
15:55
£ 64.00
18
19:55
£ 18.00
19:55
£ 22.00
Search time 142 minutes 27 seconds

I have tested with very long queries taking more than an hour to run and the only problem may be time-outs from the website. In these cases iMacro appears to lock but by pressing the Pause/Resume button will continue without losing data. Note it is best not to use the browser when running an iMacro script but I have successfully used a different browser (Safari) at the same time without any problems.

The macro is run from a javascript file in iMacro which calls iMacro commands (previously I passed variables to an iim file but it seems easier to put it all in one file) and I tweak the "Set up query" section of the javascript to query a particular train route, this could easily be set from user prompts. For the time being it is always cheapest to get two singles rather than a return so I've only bothered writing the macro for a single. Note that the slot hours (set in array myhour) are based on the maximum number of hours that Virgin trains will display in the particular train route. For London/Nottingham this is 2 hours and for London/Redruth this is 3 hours. Nice bonus functionality I've included are returning the date in long text form from the Virgin site and keeping the iMacro code display updated telling you the estimated time left.

The latest version opens a results window to show the data in an html table. You can use iimDisplay() but it is limited to a tiny window and was (at the time of writing) not resizeable by scripting.

Here's the source code. Note that the source code is not word wrapped and long lines may appear truncated, but if you cut & paste to your editor you should see all the text.

JavaScript source code Virgin-Trains.js (click to show)Virgin-Trains.js (click to hide)

// Set up query

var myfirstday=10;
var mynodays=3;
var mymonth="05";
var myleaving="London";
var mygoing="Nottingham";
var myhour = new Array("09","11");
  /* Every 2 hours for Nottingham trains as they are every 30 mins */

/** No need to edit below this line if you only want to configure the query **
/************************************************************************************
  Author: http://useroffline.blogspot.com
  Created: 28 April 2008
  Last Edit: 1 May 2008
  Title: iMacro Virgin Trains timetable search

  Description:
  Fills out query form on Virgin Trains timetable website (covers all UK timetables)
  and extracts the cheapest standard and first class ticket prices and times in any
  timeslot (normally two or three hour slots depending on what the timetables look
  like from Virgin for the particular route).
  Results are displayed in a new browser window in an html table which is written to
  as the search completes each calendar day.

  Assumptions:
  1. Virgin trains site has no technical difficulties
  2. iMacro has 'view Javascript while running' option set to off (for better speed)
  3. Infrequent usage to avoid IP blocking by Virgin
  4. Browser will not be used for anything else while running the macro
  5. Route does not use the new timetable layout (Virgin is upgrading layout)

  Suggestions for improvement:
  o Add link to each train in results table to live timetable page (probably not
    Virgin)
************************************************************************************/

// Open results window

var cr = "\r\n";
top.results=window.open( "", "myWindow", "menubar=1,resizable=1,status=1,width=500,height=600,scrollbars=1");
top.results.document.write("<html><title>Train Timetable search results</title><head>"+cr+
  "<style>td {border-width:thin;border-color:black;-moz-border-radius: 8px 8px 8px 8px;}"+cr+
  "</style></head>"+cr+
  "<body bgcolor=lightgreen style='font-family:Tahoma, Arial;'>");
top.results.window.status="Initializing, time to make a cup of tea.";

// Variables

var c=0, j, price=" na ", count=0, mytime=" na ", longdate=" na ", pArr=new Array();
var classArr=new Array("Std","1st");
var startDate=new Date();
var ret=cr+"<h3>"+myleaving+" to "+mygoing+"</h3>"+cr+"<br>Virgin Trains "+startDate.toGMTString()+"<p>"+cr;
var retTemp, runTimeM, runTimeS, runTimeMS, timeLeftS, timeLeftM, loopTime;
var mydate = new Array;

// Pad dates ("1" becomes "01")

var i;
for (i=0;i < mynodays;i++) {
  mydate[i]=(myfirstday+i).toString();
  if (mydate[i].length < 2) { mydate[i]='0'+mydate[i] }
  }
var loopTotal=mydate.length*myhour.length*2;

// Initialize to avoid timeout errors

iimPlay("CODE:URL GOTO=http://www.virgintrains.co.uk");

// Main loop

ret+="<table class='results' bgcolor=lightyellow cellpadding=4>";
top.results.document.write(ret);
ret="";

/* Static parts of the code */
pArr[1]="URL GOTO=http://www.virgintrains.co.uk/default.aspx";
pArr[7]="TAG POS=1 TYPE=SELECT FORM=NAME:qtt ATTR=NAME:outMinuteField CONTENT=$00"+cr+
    "TAG POS=1 TYPE=INPUT:IMAGE FORM=NAME:qtt ATTR=ID:_ctl15_findTimes";
pArr[9]="TAG POS=1 TYPE=INPUT:RADIO FORM=NAME:Form ATTR=ID:*ServiceTypeRadioButtons_1"+cr+
    /* ServiceType 1 = direct trains only, 0 = Any train */
    "TAG POS=1 TYPE=INPUT:IMAGE FORM=NAME:Form ATTR=ALT:Check<SP>availability"+cr+
    "TAG POS=1 TYPE=INPUT:RADIO EXTRACT=TXT"+cr+
    "TAG POS=1 TYPE=TD ATTR=CLASS:tableCell EXTRACT=TXT"+cr+
    "TAG POS=2 TYPE=TD ATTR=CLASS:tableCell EXTRACT=TXT"+cr+
    "TAG POS=3 TYPE=TD ATTR=CLASS:tableCell EXTRACT=TXT"+cr+
    "TAG POS=4 TYPE=TD ATTR=CLASS:tableCell EXTRACT=TXT"+cr+
    "TAG POS=5 TYPE=TD ATTR=CLASS:tableCell EXTRACT=TXT"+cr+
    "TAG POS=1 TYPE=TD ATTR=CLASS:tableTitle EXTRACT=HTM"+cr+
    "WAIT SECONDS=4";  // Could help to avoid lock-up?

for (j in mydate) {
  retTemp='<tr align=center ><td bgcolor=lightgreen >Slot</td><td bgcolor=lightgreen >Time</td><td bgcolor=lightgreen >Std</td><td bgcolor=lightgreen >Time</td><td bgcolor=lightgreen >1st</td></tr>';
  for (i in myhour) {
  retTemp+='<tr align=center ><td bgcolor=lightgreen >'+myhour[i]+'</td>';
  for (c=0;c < 2;c++) {
    count++;
    /* Estimate time left */
    var now = new Date();
    runTimeMS=now.valueOf()-startDate.valueOf();
    runTimeS=parseInt(runTimeMS/1000);
    runTimeM=parseInt(runTimeS/60);
    runTimeS %=60;
    loopTime=runTimeMS/count;
    timeLeftS=parseInt((loopTotal+1-count)*loopTime/1000);
    timeLeftM=parseInt(timeLeftS/60);
    timeLeftS %=60;
    /* Results window status bar feedback */
    top.results.window.status = "Loop "+count+" of "+loopTotal+
    " [Last result "+longdate+" @ "+mytime+", "+classArr[1-c]+" = £"+price.replace(/ /g,"")+"]";
    /* Run iMacro */
    pArr[0]="CODE:"+"' "+runTimeM+" minutes "+runTimeS+" seconds current run time"+cr+
        "' "+timeLeftM+" minutes "+timeLeftS+" seconds estimate to complete";
    pArr[2]="TAG POS=1 TYPE=INPUT:TEXT FORM=NAME:qtt ATTR=ID:idLeaving CONTENT="+myleaving;
    pArr[3]="TAG POS=1 TYPE=INPUT:TEXT FORM=NAME:qtt ATTR=ID:idGoing CONTENT="+mygoing;
    pArr[4]="TAG POS=1 TYPE=SELECT FORM=NAME:qtt ATTR=NAME:outDay CONTENT=$"+mydate[j];
    pArr[5]="TAG POS=1 TYPE=SELECT FORM=NAME:qtt ATTR=NAME:outMth CONTENT=$"+mymonth;
    pArr[6]="TAG POS=1 TYPE=SELECT FORM=NAME:qtt ATTR=NAME:outHourField CONTENT=$"+myhour[i];
    pArr[8]="TAG POS=1 TYPE=INPUT:RADIO FORM=NAME:Form ATTR=ID:*TicketClassRadioButtons_"+c.toString();
    iimPlay(pArr.join(cr));
    /* Get long date */
    try {longdate=iimGetLastExtract(7).split('>')[2].replace(/  /g," ")}
    catch (err) {}
    /* Parse train time and price */
    price=iimGetLastExtract(1).substr(5+(iimGetLastExtract(1).substr(4)).search("_"));
    price='    '.substr(1,6-price.length)+price;      // Pad out to 6 characters
    mytime=parseInt(iimGetLastExtract(1).substr(0,1));  // Find the table column number of train time
    mytime=iimGetLastExtract(mytime+1);          // Get time in column header
    retTemp+='<td>'+mytime+'</td><td>£'+price+'</td>';
  }  /* end class */
  retTemp+='</tr>';
  }  /* end hour */
  ret+="<tr><td colspan=5 bgcolor=lightgreen >"+longdate+'</td></tr>'+retTemp;
  top.results.document.write(ret);
  ret='';
}  /* end date */

// Calculate total running time

var endDate=new Date();
var runTimeS=parseInt((endDate.valueOf()-startDate.valueOf())/1000);
var runTimeM=parseInt(runTimeS/60);
runTimeS %=60;
ret+='</table>'+cr+'<p>Search time '+runTimeM+' minutes '+runTimeS+' seconds';
ret+=cr+"<br>Created by <a href='http://useroffline.blogspot.com' title='Author details'>UserOffline</a> 2008";

// Display Results

top.results.window.status="Finished";
top.results.document.write(ret+cr+"</body></html>");
top.results.document.close();