Web scrape weather table


I am trying to web scrape the weather data, and need to take the table and convert to csv format. But not all the entries from table have same number of columns filled. So when i take the input in this format

for h in airports:

    for i in range(1,3):
          if(i==1):
              for j in range(1,32):
                  url="https://www.wunderground.com/history/airport/"+str(h)+"/2018/"+str(i)+"/"+str(j)+"/DailyHistory.html?req_city=&req_state=&req_statename=&reqdb.zip=&reqdb.magic=&reqdb.wmo="
                  www= urllib3.PoolManager()
                  page=www.urlopen("GET",url)
                  bs= BeautifulSoup(page.data,"lxml")
                  x=bs.find('div',{"class":"high-res"})
                  for tr in x.findAll('tr'):
                         weather.append([td for td in tr.stripped_strings])

          else: 
              for k in range(1,29):
                  url="https://www.wunderground.com/history/airport/"+str(h)+"/2018/"+str(i)+"/"+str(k)+"/DailyHistory.html?req_city=&req_state=&req_statename=&reqdb.zip=&reqdb.magic=&reqdb.wmo="
                  www= urllib3.PoolManager()
                  page=www.urlopen("GET",url)
                  bs= BeautifulSoup(page.data,"lxml")
                  x=bs.find('div',{"class":"high-res"})
                  for tr in x.findAll('tr'):
                          weather.append([td for td in tr.stripped_strings])

The out put csv file is all over the place, the comma seperated variables each are going into a new column irrespective of the headers. is there an easy way to do this and get the date in a clearer way? The table is in this format.

So i am continuously appending a list with rows of the table, irrespective of the columns. How to make sure that the data in the columns is under the right header?This is the csv output i got.

This iss what i used to write the data into the csv file:

with open ('weather.csv','a') as file:
   writer=csv.writer(file)
   for row in weather:
      writer.writerow(row)  

Answers:


So guys the following seemed to fix my problem of getting the right data under the right column header:

for tr in x.findAll('tr'):
                     cols=tr.findAll('td')
                     cols=[ele.text.strip() for ele in cols]
                     weather.append([ele for ele in cols if ele])
                      
result=pd.DataFrame(weather,columns=["Time(EST)","Temp.","Windchill","Dew Point","Humidity","Pressure","Visibility","Wind Dir","Wind Speed","Gust Speed","Precip","Events","Conditions"])