{"id":8320,"date":"2021-06-09T12:14:12","date_gmt":"2021-06-09T12:14:12","guid":{"rendered":"https:\/\/wealthrevelation.com\/data-science\/2021\/06\/09\/this-data-visualization-is-the-first-step-for-effective-feature-selection\/"},"modified":"2021-06-09T12:14:12","modified_gmt":"2021-06-09T12:14:12","slug":"this-data-visualization-is-the-first-step-for-effective-feature-selection","status":"publish","type":"post","link":"https:\/\/wealthrevelation.com\/data-science\/2021\/06\/09\/this-data-visualization-is-the-first-step-for-effective-feature-selection\/","title":{"rendered":"This Data Visualization is the First Step for Effective Feature Selection"},"content":{"rendered":"
\n <\/p>\n

\"\"<\/p>\n

Image by Benjamin O. Tayo.<\/em><\/p>\n

The scatter pairplot is a visualization of pairwise relationships in a dataset and is the first step for effective feature selection. It provides a qualitative analysis of the pairwise correlation between features and is a powerful tool for feature selection and dimensionality reduction. For an introduction of the pairplot using the seaborn package, see this link: https:\/\/seaborn.pydata.org\/generated\/seaborn.pairplot.html<\/a><\/p>\n

In this article, we will analyze a portfolio of stocks to examine the ones that are strongly correlated to the overall market. The portfolio contains 22 stocks (see Table 1<\/strong>) from different sectors such as Healthcare, Real Estate, Consumer Discretionary, Energy, Industrials, Telecommunication Services, Information Technology, Consumer Staples, and Financials.<\/p>\n

\u00a0<\/p>\n\n\n\n\n\n\n\n\n\n\n\n
Symbol<\/strong><\/td>\nName<\/strong><\/td>\nSymbol<\/strong><\/td>\nName<\/strong><\/td>\nSymbol<\/strong><\/td>\nName<\/strong><\/td>\n<\/tr>\n
AAL<\/td>\nAmerican Airlines<\/td>\nEDIT<\/td>\nEditas Medicine<\/td>\nUAL<\/td>\nUnited Airlines<\/td>\n<\/tr>\n
AAPL<\/td>\nApple<\/td>\nHPP<\/td>\nHudson Pacific Properties<\/td>\nWEN<\/td>\nWendy<\/td>\n<\/tr>\n
ABT<\/td>\nAbbott Laboratories<\/td>\nJNJ<\/td>\nJohnson & Johnson<\/td>\nWFC<\/td>\nWells Fargo<\/td>\n<\/tr>\n
BNTX<\/td>\nBioNTech<\/td>\nMRNA<\/td>\nModerna<\/td>\nWMT<\/td>\nWalmart<\/td>\n<\/tr>\n
BXP<\/td>\nBoston Properties<\/td>\nMRO<\/td>\nMarathon Oil Corporation<\/td>\nXOM<\/td>\nExxon Mobile<\/td>\n<\/tr>\n
CCL<\/td>\nCarnival Corporation<\/td>\nPFE<\/td>\nPfizer<\/td>\nSP500<\/td>\nStock Market Index<\/td>\n<\/tr>\n
DAL<\/td>\nDelta Airlines<\/td>\nSLG<\/td>\nSL Green Realty<\/td>\n<\/td>\n<\/td>\n<\/tr>\n
DVN<\/td>\nDevon Energy<\/td>\nTSLA<\/td>\nTesla<\/td>\n<\/td>\n<\/td>\n<\/tr>\n<\/tbody>\n<\/table>\n

Table <\/strong>1<\/strong>. Portfolio of 22 stocks from diverse sectors.<\/p>\n

Our goal is to answer the question: which stocks in the portfolio correlate strongly with the stock market? We will use the S&P 500 index as a measure of the total stock market. We will assume a threshold correlation coefficient of 70% for a stock to be considered to be strongly correlated to the S&P 500.<\/p>\n

\u00a0<\/p>\n

Data Collection and Processing<\/h3>\n

\u00a0<\/p>\n

Raw data were obtained from the yahoo finance website: https:\/\/finance.yahoo.com\/<\/a><\/p>\n

The historical data for each stock has information on daily open price, high price, low price, and closing price. The CSV file was downloaded for each stock, and then the column \u201cclose\u201d was extracted and combined to create the dataset, which can be found here: portfolio.csv<\/a><\/p>\n

\u00a0<\/p>\n

Generate Scatter Pairplot<\/h3>\n

\u00a0<\/p>\n

\n
import numpy as np\r\nimport pandas as pd\r\nimport pylab\r\nimport matplotlib.pyplot as plt\r\nimport seaborn as sns\r\n\r\nurl = 'https:\/\/raw.githubusercontent.com\/bot13956\/datasets\/master\/portfolio.csv'\r\ndata = pd.read_csv(url)\r\ndata.head()\r\n\r\ncols = data.columns[1:24]\r\nsns.pairplot(data[cols], height=2.0)\r\n\r\n\r\n<\/pre>\n<\/div>\n
\u00a0<\/p>\n
Calculate Covariance Matrix<\/h3>\n
\u00a0<\/p>\n
The scatter pairplot is the first step, which provides a qualitative analysis of pairwise correlations between features. To quantify the degree of correlation, the covariance matrix has to be computed.<\/p>\n
\n
from sklearn.preprocessing import StandardScaler\r\nstdsc = StandardScaler()\r\nX_std = stdsc.fit_transform(data[cols].iloc[:,range(0,23)].values)\r\n\r\ncov_mat = np.cov(X_std.T, bias= True)\r\n\r\nimport seaborn as sns\r\nplt.figure(figsize=(13,13))\r\nsns.set(font_scale=1.2)\r\nhm = sns.heatmap(cov_mat,\r\n                 cbar=True,\r\n                 annot=True,\r\n                 square=True,\r\n                 fmt='.2f',\r\n                 annot_kws={'size': 12},\r\n                 yticklabels=cols,\r\n                 xticklabels=cols)\r\nplt.title('Covariance matrix showing correlation coefficients')\r\nplt.tight_layout()\r\nplt.show()\r\n\r\n\r\n<\/pre>\n<\/div>\n
\u00a0<\/p>\n
\u00a0<\/p>\n
Compressed Output Showing Pairplots and Correlation Coefficients<\/h3>\n
\u00a0<\/p>\n
Since we are only interested in the correlations between the 22 stocks in the portfolio with the S&P 500, Figure 1<\/strong> below shows the final output from our analysis.<\/p>\n
\"\"<\/p>\n
Figure <\/strong>1<\/strong>. Scatter pairplots and correlation coefficients between portfolio stocks and the S&P 500.<\/em><\/p>\n
Figure 1<\/strong> shows that out of the 22 stocks, 8 have a correlation coefficient less than 70%. Interestingly, except for WEN stock, all the other stocks have a positive correlation with the S&P 500 index.<\/p>\n
The full covariance matrix is shown in Figure 2<\/strong>.<\/p>\n
\"\"<\/p>\n
Figure 2<\/strong>. Covariance matrix visualization.<\/em><\/p>\n
In summary, we\u2019ve shown how the scatter pairplot can be used as a first step for feature selection. Other advanced methods for feature selection and dimensionality reduction include the following: PCA (Principal Component Analysis<\/a>) and LDA (Linear Discriminant Analysis<\/a>).<\/p>\n
\u00a0<\/p>\n
Related:<\/b><\/p>\n<\/p><\/div>\n","protected":false},"excerpt":{"rendered":"
https:\/\/www.kdnuggets.com\/2021\/06\/data-visualization-feature-selection.html<\/p>\n","protected":false},"author":0,"featured_media":8321,"comment_status":"open","ping_status":"open","sticky":false,"template":"","format":"standard","meta":[],"categories":[2],"tags":[],"_links":{"self":[{"href":"https:\/\/wealthrevelation.com\/data-science\/wp-json\/wp\/v2\/posts\/8320"}],"collection":[{"href":"https:\/\/wealthrevelation.com\/data-science\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/wealthrevelation.com\/data-science\/wp-json\/wp\/v2\/types\/post"}],"replies":[{"embeddable":true,"href":"https:\/\/wealthrevelation.com\/data-science\/wp-json\/wp\/v2\/comments?post=8320"}],"version-history":[{"count":0,"href":"https:\/\/wealthrevelation.com\/data-science\/wp-json\/wp\/v2\/posts\/8320\/revisions"}],"wp:featuredmedia":[{"embeddable":true,"href":"https:\/\/wealthrevelation.com\/data-science\/wp-json\/wp\/v2\/media\/8321"}],"wp:attachment":[{"href":"https:\/\/wealthrevelation.com\/data-science\/wp-json\/wp\/v2\/media?parent=8320"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/wealthrevelation.com\/data-science\/wp-json\/wp\/v2\/categories?post=8320"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/wealthrevelation.com\/data-science\/wp-json\/wp\/v2\/tags?post=8320"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}