{"id":8500,"date":"2022-03-21T17:06:38","date_gmt":"2022-03-21T17:06:38","guid":{"rendered":"https:\/\/wealthrevelation.com\/data-science\/2022\/03\/21\/air-quality-forecasting-python-project\/"},"modified":"2022-03-21T17:06:38","modified_gmt":"2022-03-21T17:06:38","slug":"air-quality-forecasting-python-project","status":"publish","type":"post","link":"https:\/\/wealthrevelation.com\/data-science\/2022\/03\/21\/air-quality-forecasting-python-project\/","title":{"rendered":"Air Quality Forecasting Python Project"},"content":{"rendered":"
\n

You will find the full python code and all visuals for this article here in this gitlab repository<\/a>. The repository contains a series of analysis, transforms and forecasting models frequently used when dealing with time series. The aim of this repository is to showcase how to model time series from the scratch, for this we are using a real usecase dataset
<\/em><\/p>\n

This project foreca<\/span>st the Carbon Dioxide<\/strong> (Co2<\/strong>) emission levels yearly. Most of the<\/span> organizations have to follow<\/span> government<\/span> norms with respect to Co2 emissions and they h<\/span>ave to pay<\/span> charges accordingly,<\/span> so this<\/span> project<\/span> will<\/span> forecast<\/span> the Co2 levels so that organizations can follow<\/span> the<\/span> norms and pay in advance based on the forecasted values. In any data science<\/span> project<\/span> t<\/span>he main<\/span> component is data, for<\/span> this project the data was provided by the company, from here time series<\/span> concept<\/span> comes into the picture. The dataset for this project<\/span> contains 215 entries and two co<\/span>mponents<\/span> which are Year and Co2<\/span> emissions which is univariate time series as there is only one dependent<\/span> variable Co2 which depends on time. from year 1800 to year 2014 Co2 lev<\/span>els were present in the<\/span> dataset.<\/span><\/span><\/p>\n

The dataset used: The dataset contains yearly Co2 emmisions levels. data from 1800 to 2014 sampled every 1 year. The dataset is non stationary so we have to use differenced time series for forecasting.<\/em><\/p>\n

After ge<\/span>tting da<\/span>ta the next step is to analyze the time series data. This process is done <\/span>by using Python<\/strong>. The data was present in excel file so first we need to read that excel file. This tas<\/span>k is<\/span> done by<\/span> using Pandas<\/strong> which<\/span> is python libraries to creates Pandas Data Frame<\/strong><\/span>. After that<\/span> preprocessing like changing data types of time from object to DateTime performed for the coding<\/span> purpose<\/span>. Time s<\/span>eries contain 4 main components<\/span> Level, Trend, Seasonality and Noise. To study this <\/span>component,<\/span> we need to decompose our time series so<\/span> that we can batter understand our time series<\/span> and we can choose the forecasting model acco<\/span>rdingly because each<\/span> component<\/span> behave different on<\/span> the model. also by decomposing we can identify that the time series<\/strong> is multiplicative or additive.<\/span><\/span><\/p>\n

\"\"<\/p>\n

CO2 emissions \u2013 plotted via python pandas \/ matplotlib<\/p>\n<\/div>\n

Decomposing ti<\/span>me series using python<\/span> states<\/span>models<\/span> libraries we get to know trend, seasonality and<\/span> residual component separately.<\/span> the components multiply together to make the time series<\/span> multiplicative and in additive time series components added<\/span> together<\/span>.<\/span><\/span> Taking the deep dive to understand the trend component, moving average of 10 steps<\/span> were applied which shows nonlinear upward trend, fit the linear regression model<\/strong> to check the<\/span> trend<\/span> which shows upward trend.<\/span> talking about seasonality there were combinatio<\/span>n of multiple patterns<\/span> over time period which is common in real world time series data. capturing the white noise is difficult<\/span> in this type of data.<\/span> the time series contains values<\/span> from 1800 where the Co2 values<\/span> are less then 1<\/span> because of no human activitie<\/span>s so levels were<\/span> decreasing<\/span>. By the time numbers of industries and <\/span>human activities are<\/span> rapidly<\/span> increasing<\/span> which causes Co2 levels rapidly<\/span> increasing<\/span>. In time<\/span> series the<\/span> highest Co2 emission<\/span> level was 18.7 in 1979. It was challenging to decide<\/span> whether<\/span> to con<\/span>sider this<\/span> values which are less then 0.5 as white noise or not because 30% of the Co2 values were less then 1,<\/span> in real world looking at current scenar<\/span>io the chances of Co2 emission<\/span> level being 0 is near to<\/span> impossible still there are chances that Co2 leve<\/span>ls can be 0.0005. So considering each data point as a <\/span>valuable information we refused to remove that entries.<\/span><\/span>
<\/span><\/p>\n

Next step is to create Lag plot so we can see the correlation between the current year <\/span>Co2 level and previous year Co2 level. the plot was linear<\/span> which shows high correlation so we can say<\/span> that the current<\/span> Co2 levels and previous levels<\/span> have strong relationship. the randomness of the data<\/span> were<\/span> measured<\/span> by<\/span> plotting<\/span> autocorrelation graph. the autocorrelation graph shows smooth curves<\/span> which<\/span> indicates<\/span> the<\/span> time series is non<\/span>\u2013<\/span>stationary thus next step is to make time series stationary. in <\/span>non<\/span>\u2013<\/span>stationary<\/span> time series, summary<\/span> statistics<\/span> like mean and variance change over time. <\/span><\/span><\/p>\n

To make <\/span><\/span>time<\/span> series stationary we have to remove trend and seasonality from it. Before<\/span> that we use dickey <\/span>fuller test to make sure our time series is non<\/span>\u2013<\/span>stationar<\/span>y. the test was<\/span> done by<\/span> using<\/span> python, and the<\/span> test gives p<\/span>\u2013<\/span>value as output. here the null hypothesis<\/strong> is that the data is non<\/span>\u2013<\/span>stationary while<\/span> alternate<\/span> hypothesis is that the dat<\/span>a is stationary, in this case the<\/span> significance<\/span> values is 0.05 and the p<\/strong><\/span>\u2013<\/span><\/strong>value<\/strong>s<\/span> which is given by dickey fuller tes<\/span>t is<\/span> greater<\/span> than 0.05 hence we<\/span> failed to reject null hypothesis so<\/span> we can say the time series is non<\/span>\u2013<\/span>stationery<\/span>. Differencing is one of the tec<\/span>hniques to make time series <\/span>stationary. On this time series, first order differencing<\/span> t<\/span>echnique<\/span> applied to make the time series <\/span>stationar<\/span>y. In first order differencing<\/span> we have to subtract previous value from current value for all<\/span> the<\/span> data points<\/span>. also diffe<\/span>rent transformations like log, sqrt and reciprocal were applied in the context<\/span> of making the time series stationary. Smoothing techniques like simple moving<\/span> average<\/span>,<\/span> exponential<\/span> weighted<\/span> moving average, simple exponential<\/span> smoothing<\/span> and double exponential s<\/span>moothing<\/span> techniques can be applied to remove the variation between time<\/span> stamps<\/span> and to see the smooth curves.<\/span>
<\/span><\/p>\n

Smoothing techniques also used to observe trend in time series as well a<\/span>s to predict the future values. B<\/span>ut performance of other models was good co<\/span>mpared to smoothing<\/span> techniques<\/span>.<\/span><\/span> First 200 entries taken to train the model and remaining last for testing the<\/span> performance of the model. performance of different models me<\/span>a<\/span>sured by Root Mean Squared<\/span> Error (RMSE<\/strong>) and Mean Absolute Error<\/strong> (MAE<\/strong>) as we are predictin<\/span>g future Co2 emissions<\/span> so basically<\/span> it is regression problem. RMSE is calculated by root of the average of squared difference between<\/span> actual values and predicted values by the model on testing data. Here RMSE values were calculated<\/span> using python sklearn lib<\/span>rary.<\/span> For model building two<\/span> approaches<\/span> are there, one is data<\/span>\u2013<\/span>driven and<\/span> another one is model based. models from both the<\/span> approaches<\/span> were<\/span> applied<\/span> to find the best fitted<\/span> model. ARIMA model gives the best results for this kind of dataset as the model were<\/span> trained<\/span> on<\/span> differenced<\/span> time series. The ARIMA model predicts a given time series based on its own past values.<\/span> It can be used for any non<\/span>\u2013<\/span>seasonal series of numbers that exhibits patterns and is not a series of<\/span> random events. ARIMA<\/strong> takes 3 parameters which a<\/span>re AR, M<\/span>A and the order of difference.<\/span> Hyper<\/span> parameter<\/span> tuning<\/span> technique<\/span> gives best parameters for the model by trying different sets of <\/span>parameters. Although The autocorrelation and partial autocorrelation plots can be use to decide AR <\/span>and MA parameter beca<\/span>use partial autocorrelat<\/span>ion function s<\/span>hows the partial<\/span> correlation of a<\/span> stationary time series with its own lagged values so using PACF we can decide the value of AR and<\/span> from ACF we can decide the value of MA parameter as ACF shows how data points in a ti<\/span>me series<\/span> are related.<\/span><\/span><\/p>\n

\"\"<\/p>\n

Yearly difference of CO2 emissions \u2013 ARIMA Prediction<\/p>\n<\/div>\n

Apart from ARIMA, few other model were trained which are AR, ARMA, Simple Linear <\/span>Regression, Quadratic method, Holts winter exponential smoothing, Ridge<\/strong> and Lasso<\/strong><\/span> Regression<\/span><\/strong>,<\/span> LGBM and XGboost methods, Recurrent neural network<\/strong> (RNN)<\/strong><\/span> \u2013<\/span> Long S<\/span>hort Term<\/span> <\/strong>Memory<\/strong> (LSTM<\/strong>) and<\/span> Fbprophet. I would like to mention my<\/span> experience<\/span> with LSTM here because it is another model which<\/span> gives<\/span> good<\/span> result as ARIMA. the<\/span> reason<\/span> for not choosing LSTM as final model is its complexity. As<\/span> ARIMA is giving<\/span> appropriate<\/span> results<\/span> and it is simple to understand<\/span> and<\/span> requires<\/span> less dependencies.<\/span> while using lstm, lot of data preprocessing and other dependencies required, the dataset was small<\/span> thus we used to train the model on CPU, oth<\/span>e<\/span>rwise gpu is required to train the LSTM model. we f<\/span>ace<\/span> one more challenge in deployment part. the challenge<\/span> is to get the data into original form<\/span> because<\/span> the<\/span> model was trained on differenced time series, so it will predict the future values in differenced format.<\/span> After lot of research on the internet and<\/span> by deeply understanding<\/span> mathemat<\/span>ical<\/span> concepts finally we got the solution for it. solution for this issue is we have to add previous value from the original data from into first order differencing and then we have to add the last value of this time series into predicted values. <\/span>To create the user interface streamlit was used, it is commonly used python library. the pickle file of the ARIMA model were used to predict the future values based on user input. The limit for forecasting is the year 2050. The project was uploaded on google cloud platform. so the flow is, first the starting year from which user want to forecast was taken and the end year till which year user want to forecast was taken and then according to the range of this inputs the prediction takes place. so by taking the inputs the pickle file will produce the future Co2 emissions in differenced format, then the values will be converted to original format and then the original values will be displayed on the user interface as well as the interactive line graph were displayed on the interface.<\/span>
<\/span><\/span><\/p>\n

You will find the full python code and all visuals for this article here in this gitlab repository<\/a>.<\/em><\/p>\n

\n

Shivani Padaya<\/a><\/h3>\n
<\/div>\n

I’m an IT graduate and data science enthusiast who loves to execute data driven solutions and find hidden insights from the data. I enjoy analyzing time series data. I like to read and write data science blogs.<\/p>\n<\/div>\n<\/div>\n","protected":false},"excerpt":{"rendered":"

https:\/\/data-science-blog.com\/en\/blog\/2022\/03\/20\/air-quality-forecasting-python-project\/<\/p>\n","protected":false},"author":0,"featured_media":8501,"comment_status":"open","ping_status":"open","sticky":false,"template":"","format":"standard","meta":[],"categories":[2],"tags":[],"_links":{"self":[{"href":"https:\/\/wealthrevelation.com\/data-science\/wp-json\/wp\/v2\/posts\/8500"}],"collection":[{"href":"https:\/\/wealthrevelation.com\/data-science\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/wealthrevelation.com\/data-science\/wp-json\/wp\/v2\/types\/post"}],"replies":[{"embeddable":true,"href":"https:\/\/wealthrevelation.com\/data-science\/wp-json\/wp\/v2\/comments?post=8500"}],"version-history":[{"count":0,"href":"https:\/\/wealthrevelation.com\/data-science\/wp-json\/wp\/v2\/posts\/8500\/revisions"}],"wp:featuredmedia":[{"embeddable":true,"href":"https:\/\/wealthrevelation.com\/data-science\/wp-json\/wp\/v2\/media\/8501"}],"wp:attachment":[{"href":"https:\/\/wealthrevelation.com\/data-science\/wp-json\/wp\/v2\/media?parent=8500"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/wealthrevelation.com\/data-science\/wp-json\/wp\/v2\/categories?post=8500"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/wealthrevelation.com\/data-science\/wp-json\/wp\/v2\/tags?post=8500"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}