Set Up a Private Airfare Watchdog by Your Own

There are some public airfare watchdogs online. But we still have tons of reasons to set up a private one. For example, we do not want to share our email address, considering privacy and/or spam. Additionally, private watchdog has quicker response. And flexible customization is also a strong motivation. In this article, I will provide a solution to set up a simple but effective private airfare watchdog.

1. Install Firefox. Yes, it is possible to write a program to crawl the airfare information. But the solution with Firefox is more portable and robust.  More importantly, it needs less labor hours.

2. Install an extension for Firefox: iMacros. This is the core component to crawl the airfare information from different agents, such as flychina.com and onetravel.com. We need little programming background to write the script. Actually, the iMacros could record your activities on the webpage, and generate the script automatically.  As an example, we consider the open jaw trip from Los Angeles (LAX) to San Francisco (SFO) on 2013/11/11 and from Las Vegas (LAS) to Los Angeles (LAX) on 2013/12/12. The following script gets and stores the airfare quote page of flychina.com and onetravel.com to our specific directory E:/flight/temp/flychina and E:/flight/temp/onetravel respectively.

   1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30

   VERSION BUILD=8530828 RECORDER=FX TAB T=1 SET !REPLAYSPEED SLOW SET !EXTRACT_TEST_POPUP NO SET !ERRORIGNORE YES URL GOTO=http://www.onetravel.com/default.aspx?tabid=1917 TAG POS=1 TYPE=INPUT:TEXT FORM=ID:Form ATTR=ID:ctl09_ctl04_ctl00_repCityList_ctl00_Seg_From CONTENT=LAX TAG POS=1 TYPE=INPUT:TEXT FORM=ID:Form ATTR=ID:ctl09_ctl04_ctl00_repCityList_ctl00_Seg_Date CONTENT=11/11/2013 TAG POS=1 TYPE=INPUT:TEXT FORM=ID:Form ATTR=ID:ctl09_ctl04_ctl00_repCityList_ctl00_Seg_To CONTENT=SFO TAG POS=1 TYPE=INPUT:TEXT FORM=ID:Form ATTR=ID:ctl09_ctl04_ctl00_repCityList_ctl01_Seg_From CONTENT=LAS TAG POS=1 TYPE=INPUT:TEXT FORM=ID:Form ATTR=ID:ctl09_ctl04_ctl00_repCityList_ctl01_Seg_Date CONTENT=12/12/2014 TAG POS=1 TYPE=INPUT:TEXT FORM=ID:Form ATTR=ID:ctl09_ctl04_ctl00_repCityList_ctl01_Seg_To CONTENT=LAX TAG POS=1 TYPE=SELECT FORM=ID:Form ATTR=ID:ctl09_ctl04_ctl00_repCityList_ctl00_Seg_Date_PreferredTime CONTENT=%1100 TAG POS=1 TYPE=SELECT FORM=ID:Form ATTR=ID:ctl09_ctl04_ctl00_repCityList_ctl01_Seg_Date_PreferredTime CONTENT=%1100 TAG POS=1 TYPE=INPUT:IMAGE FORM=ID:Form ATTR=ID:ctl09_ctl04_ctl00_Search WAIT SECONDS=180 TAG POS=1 TYPE=HTML ATTR=* EXTRACT=HTM SAVEAS TYPE=EXTRACT FOLDER=E:flighttemponetravel FILE=+{{!NOW:yyyymmddhhnnss}} URL GOTO=http://www.flychina.com/lfw/desk.aspx?OneWay=2&lan= TAG POS=1 TYPE=INPUT:TEXT FORM=ID:Desk ATTR=ID:tbFrom CONTENT=LAX TAG POS=1 TYPE=INPUT:TEXT FORM=ID:Desk ATTR=ID:tbTo CONTENT=SFO TAG POS=1 TYPE=INPUT:TEXT FORM=ID:Desk ATTR=ID:DepDate CONTENT=11/11/2013 TAG POS=1 TYPE=INPUT:TEXT FORM=ID:Desk ATTR=ID:tbFrom1 CONTENT=LAS TAG POS=1 TYPE=INPUT:TEXT FORM=ID:Desk ATTR=ID:tbTo1 CONTENT=LAX TAG POS=1 TYPE=INPUT:TEXT FORM=ID:Desk ATTR=ID:DepDate1 CONTENT=12/12/2014 TAG POS=1 TYPE=INPUT:SUBMIT FORM=ID:Desk ATTR=ID:ibDeskSearch WAIT SECONDS=180 TAG POS=1 TYPE=HTML ATTR=* EXTRACT=HTM SAVEAS TYPE=EXTRACT FOLDER=E:/flight/temp/flychina FILE=+{{!NOW:yyyymmddhhnnss}} WAIT SECONDS=3600

   Then we could run this script in loop. But beware of the quota of these sites. Lots of them have a daily maximum allowed search limitation. Once you reached it, the inquiry return nothing but warming message. As far as I know, flychina.com has such limitation. And if you search on onetravel.com every three or four minutes, you could do it for 24hour/day and 7days/week.

   If you have any programming background, you will find the script is easy to modify. You could improve it according your specific requirements.

3. Install Python 2.7.X on your computer.

4. Install pygooglevoice (http://code.google.com/p/pygooglevoice/). It is a Python package to use Google Voice. We use it to send SMS notification. But it is a little out of date, because Google changed the API interface. So we need fix the change before being able to use it. The fix solution could be found here (https://code.google.com/p/pygooglevoice/issues/detail?id=76&q=galx). If you do not need the SMS notification, you could just ignore this step. For ignoring the SMS notification, you need to comment 16th line in the source code with “#”.
5. Copy the following code onto your computer, and give it the execution right if on *nix platform.
   
   Python

   1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80 81 82 83 84 85 86 87 88 89 90 91 92 93 94 95 96 97 98 99 100 101 102 103 104 105 106 107 108 109 110 111 112 113 114 115 116 117 118 119 120 121 122 123 124 125 126 127 128 129 130 131 132 133 134 135 136 137 138 139 140 141 142 143 144 145 146 147 148 149 150 151 152 153 154 155 156 157 158 159 160 161 162 163 164 165 166 167 168 169 170 171 172 173 174 175 176 177 178 179 180 181 182 183 184 185 186 187 188 189 190 191 192 193 194 195 196 197 198 199 200 201 202 203 204 205 206 207 208 209 210 211 212 213 214 215 216 217 218 219 220 221 222 223 224 225 226 227 228 229 230 231 232 233 234 235 236 237 238 239 240 241 242 243 244 245 246 247 248 249 250 251 252 253 254 255 256 257 258 259 260 261 262 263 264 265 266 267 268 269 270 271 272 273 274 275 276 277 278 279 280 281 282 283 284 285 286 287 288 289 290 291 292 293 294 295 296 297 298 299 300 301 302 303 304 305 306 307 308 309 310 311 312 313 314 315 316 317 318 319 320 321 322 323 324 325 326 327 328 329 330

   #============================================================================= # Author: Sheng Yu - https://34.145.67.234 # Email : yusheng123 at gmail dot com # Last modified: 2013-11-18 10:01 # Filename: flight_monitor_gv.py # Description: a component to monitor the flight's price # Changelog: #       2013-11-12: fix a bug in __get_iti_iflychina and __get_iti_onetravel #                   When the site returns warming information after search #                   limitation, the old functions will throw exceptions. # #                   fix a bug in __del__ function of Gmail class and SMS class #                   In some rare cases, the __del__ will throw an exception as #                   AttributeError: instance has no attribute 'mailServer' or #                   'voice'. # #       2013-11-18: fix bug in logging in the Gmail and/or Google Voice. When #                   no action is taken for a long time, the connection to #                   server might be lost. Solution: catch such exception, and #                   reconnect if needed. #============================================================================= import sys, os, time, re, smtplib, getpass, math from email.mime.text import MIMEText from email.mime.multipart import MIMEMultipart from email.mime.base import MIMEBase from email import encoders # If you do NOT want to use SMS notification, comment the following ONE line from googlevoice import Voice # Configuration Section # If found flights below this price, send email THRESHOLD_EMAIL = 1400 # If found flights below this price, send SMS THRESHOLD_SMS = 1300 # Every INTERVAL seconds, scan the fold for new price quote INTERVAL = 300 # Gmail account information # If you left them empty here, the program will ask for them everytime mail_usr = "" mail_pwd = "" # Google Voice account information # If you left them empty here, the program will ask for them everytime sms_usr = "" sms_pwd = "" # END Configuration Section class Gmail:     ''' A class to send email via Gmail servers     '''     def __init__(self, usr, pwd):         self.usr = usr         self.pwd = pwd         self.login()         return     def login(self):         # Logging into the server         self.mailServer = smtplib.SMTP("smtp.gmail.com", 587)         self.mailServer.ehlo()         self.mailServer.starttls()         self.mailServer.ehlo()         self.mailServer.login(self.usr, self.pwd)         return     def __del__(self):         if hasattr(self, 'mailServer'):             try:                 self.mailServer.quit()                 self.mailServer.close()             except:                 del self.mailServer         return     def send(self, to, subject, text, attach=None):         ''' send an email to address "to" with text being the body             If attach is not None, it will be sent as attachment         '''         msg = MIMEMultipart()         msg['From'] = self.usr         msg['To'] = to         msg['Subject'] = subject         msg.attach(MIMEText(text))         if attach != None:             part = MIMEBase('application', 'octet-stream')             part.set_payload(open(attach, 'rb').read())             encoders.encode_base64(part)             part.add_header('Content-Disposition',                 'attachment; filename="%s"' % os.path.basename(attach))             msg.attach(part)         try:             self.mailServer.sendmail(self.usr, to, msg.as_string())         except:             self.mailServer.close()             self.mailServer.login()             #Try again             self.mailServer.sendmail(self.usr, to, msg.as_string()) class SMS:     ''' A class to send SMS via Gmail Voice     '''     def __init__(self, usr, pwd):         ''' Initialize the object with a Google Voice account         '''         self.usr = usr         self.pwd = pwd         self.voice = Voice()         self.voice.login(usr, pwd)         return     def __del__(self):         if hasattr(self, 'voice'):             try:                 self.voice.logout()             except:                 del self.voice         return     def send(self, phoneNumber, text):         ''' send a SMS to phoneNumber with text         '''         try:             self.voice.send_sms(phoneNumber, text)         except:             self.voice = Voice()             self.voice.login(self.usr, self.pwd)             self.voice.send_sms(phoneNumber, text)         return class Quote:     ''' A parser to get information from a specific file, containing         flight quote     '''     def __init__(self, quote_file):         self.filename = quote_file         self.price_finder = {"flychina": self.__get_price_iflychina,                              "onetravel": self.__get_price_onetravel}         self.itinerary_finder = {"flychina": self.__get_iti_iflychina,                                  "onetravel": self.__get_iti_onetravel}         if not self.site_name() in self.price_finder:             # No such price parser is available             raise Exception("No suitable price parser is installed: %s." % self.site_name())         if not self.site_name() in self.itinerary_finder:             # No such itinerary parser is available             raise Exception("No suitable itinerary parser is installed: %s." % self.site_name())         return     def site_name(self):         ''' Get the site name of current flight quote         '''         return os.path.basename(os.path.dirname(self.filename))     def check_time(self):         ''' Get the quote time according to the file name         '''         # basename(filename) is in form of extract20130326213855.csv         when = os.path.basename(self.filename)[7:-4]         when = when[:4]+"-"+when[4:6]+"-"+when[6:8]+" "+when[8:10]+                 ":"+when[10:12]+":"+when[12:]         return when     def get_price(self):         ''' Get the quote price             The actual work is done by different specific parser         '''         price = self.price_finder[self.site_name()]()         if price == None:             return float('nan')         else:             return price     def get_itinerary(self):         ''' Get the itinerary in the quote             The actual work is done by different specific parser         '''         itinerary = self.itinerary_finder[self.site_name()]()         if itinerary == None:             return "Unknown"         else:             return itinerary     def __get_price_iflychina(self):         ''' Parse the HTML file from iflychina.com to get the price             WARMING: this function is easy to be out of date, since these                 signatures might be changed         '''         pattern = '(?<=<span class=""rsubtitle"">$)[1-9][0-9]{0,2}(,[0-9]{3})*</span> </a>per adult  &nbsp; <nobr>(taxes and fees included)</nobr>'         fhandle = open(self.filename, "rb")         content = fhandle.read()         fhandle.close()         content = content.decode("utf8")         price = re.search(pattern,content)         if price != None:             price = price.group(0)             price = price[:price.find("<")]             price = int(price.replace(",",""))         return price     def __get_iti_iflychina(self):         ''' Parse the HTML file from iflychina.com to get the itinerary             WARMING: this function is easy to be out of date, since these                 signatures might be changed         '''         pattern = re.compile('(?<=()[A-Za-z]{3}(?=))')         sig1 = "We are searching thousands of fares to find the perfect deal for your travel dates"         sig2 = "Leaving on"         fhandle = open(self.filename, "rb")         content = fhandle.read()         fhandle.close()         content = content.decode("utf8")         content = content[content.find(sig1)+len(sig1):]         trip1 = content[:content.find(sig2)]         content = content[content.find(sig2)+len(sig2):]  # Skip the second for the first trip         trip2 = content[:content.find(sig2)]         result = ""         temp = pattern.findall(trip1)         if len(temp) < 2:             result = "N/A"         else:             result += temp[0] + "-" + temp[1] + "; "             temp = pattern.findall(trip2)             if len(temp) < 2:                 result += "N/A"             else:                 result += temp[0] + "-" + temp[1]         return result     def __get_price_onetravel(self):         ''' Parse the HTML file from onetravel.com to get the price             WARMING: this function is easy to be out of date, since these                 signatures might be changed             Actually, it seems changed from my previous version         '''         pattern = '(?<=[0-9].[0-9][0-9]"">)[1-9][0-9]{0,3}(?=</span>)'         fhandle = open(self.filename, "rb")         content = fhandle.read()         fhandle.close()         content = content.decode("utf8")         sig0 = "Sort by:"         sig1 = 'This is an alternate date, please verify the date'         sig2 = 'Total per person'         content = content[content.find(sig0)+len(sig0):]         price = None         while content.find(sig2) != -1:             head = content[:content.find(sig2)]             content = content[content.find(sig2)+len(sig2):]             if head.find(sig1) != -1:                 continue             price = re.search(pattern,content)             break         if price != None:             price = int(price.group(0))         return price     def __get_iti_onetravel(self):         ''' Parse the HTML file from onetravel.com to get the itinerary             WARMING: this function is easy to be out of date, since these                 signatures might be changed         '''         sig = "Available flight deals found from "         fhandle = open(self.filename, "rb")         content = fhandle.read()         fhandle.close()         content = content.decode("utf8")         if content.find(sig) == -1:             result = "N/A"         else:             content = content[content.find(sig)+len(sig):]             result = content[:3] + "-" + content[7:10]         return result     def remove_file(self):         ''' Remove a specific file on OS         '''         while os.path.exists(self.filename):             os.remove(self.filename)             time.sleep(0.3)         return def usage(prog):     ''' print out the usage information of this program     '''     print("Usage:",prog,"folder")     return if __name__=="__main__":     # Check the arguments     if len(sys.argv) != 2 or not os.path.isdir(sys.argv[1]):         usage(sys.argv[0])         sys.exit(-1)     # Prepare for sending emails     if mail_usr == "" or mail_pwd == "":         mail_usr = raw_input("Please input your Gmail acount: ")         mail_pwd = getpass.getpass(prompt="Please input your Gmail password: ")     to_mail = raw_input("Please input the email to recieve notification: ")     try:         mailer = Gmail(mail_usr,mail_pwd)     except:         print "Error in logging in Gmail accoutn."         sys.exit(-2)     del mail_usr,mail_pwd     # Prepare for sending sms     if 'Voice' in globals():    # The Google Voice package is imported         if sms_usr == "" or sms_pwd == "":             sms_usr = raw_input("Please input your Google Voice acount [press enter to skip]: ")             if sms_usr != "":                 sms_pwd = getpass.getpass(prompt="Please input your Google Voice password: ")     else:       # If we do not have the package, how could we use usr and pwd?         sms_usr = ""         sms_pwd = ""     if sms_usr != "":         to_sms = raw_input("Please input the phone number to recieve notification: ")         try:             messenger = SMS(sms_usr, sms_pwd)         except:             print "Error in logging in Google Voice accoutn."             sys.exit(-3)     del sms_usr,sms_pwd     pre_price = {}     try:         while True:             for root, dirs, files in os.walk(sys.argv[1]):                 for file in files:                     # parse each file with flight quote information                     flight_quote = Quote(os.path.join(root,file))                     site = flight_quote.site_name()                     if not site in pre_price:                         pre_price[site] = {}                     when = flight_quote.check_time()                     itinerary = flight_quote.get_itinerary()                     if not itinerary in pre_price[site]:                         pre_price[site][itinerary] = -1                     quote = flight_quote.get_price()                     flight_quote.remove_file()                     if math.isnan(quote) or quote == pre_price[site][itinerary]:                         # No change or still meet mistake                         continue                     pre_price[site][itinerary] = quote                     newprice = "Get new price as: "+str(quote)+                                " in "+site+" at "+when+" ("+itinerary+")"                     print(newprice)                     if quote <= THRESHOLD_EMAIL:                         # Acceptable price                         mailer.send(to_mail,"Price Alert!",newprice)                     if quote <= THRESHOLD_SMS and "messenger" in locals():                         # Great price                         messenger.send(to_sms,newprice )             time.sleep(INTERVAL)     except KeyboardInterrupt:         print "Good luck!"         sys.exit(0)

   
   The directory structure on my computer is like the following:
   
   1 2 3 4 5 6 7 8 9 10

   E:FLIGHTS │  flight_monitor_gv.py │ └─temp     ├─flychina     │      extract20131110231222.csv     │     └─onetravel             extract20131110230536.csv             extract20131110230901.csv

   
   Then you need to run the Python script, such as: “E:flights>c:Python27python.exe flight_monitor_gv.py temp”. The UI should be like this:
   
   1 2 3 4 5 6 7 8 9 10

   E:flights>c:Python27python.exe flight_monitor_gv.py temp Please input your Gmail acount: XXXXXXX@gmail.com Please input your Gmail password: Please input the email to recieve notification: XXXXXXX@gmail.com Please input your Google Voice acount: XXXXXXX@gmail.com Please input your Google Voice password: Please input the phone number to recieve notification: 1234567890 Get new price as: 1588 in flychina at 2013-11-09 00:01:15 (LAX-SFO; LAS-LAX) Get new price as: 1446 in onetravel at 2013-11-08 23:57:44 (LAX-SFO) Get new price as: 1493 in onetravel at 2013-11-09 00:04:39 (LAX-SFO)

6. Have a cup of coffee!

There is a big space to improve for this solution. The price and iteration parser should be designed as plug-in, so that we could add/remove/change them without modifying the main program.