Stock Market Prices Prediction Using Machine Learning and Deep Learning

AishwaryaSingh 13 Aug, 2024
12 min read

Introduction

Predicting how the stock market will perform is one of the most difficult things. There are so many factors involved in the prediction—physical factors vs. psychological factors, rational and irrational behavior, etc. All these aspects combine to make share prices volatile and very difficult to predict with a high degree of accuracy.

Can we use machine learning as a game-changer in this domain? Using features like the latest announcements about an organization, their quarterly revenue results, etc., machine learning techniques can unearth patterns and insights we didn’t see before, and these can be used to make unerringly accurate predictions.

stock price prediction, LSTM, machine learning

In this article, we will work with historical data about the stock prices of a publicly listed company. We will implement a mix of machine learning algorithms to predict the future stock price of this company, starting with simple algorithms like averaging and linear regression, and then move on to advanced techniques like Auto ARIMA and LSTM. The above-stated machine learning algorithms can be easily learned from this ML Course online.

The core idea behind this article is to showcase how these algorithms are implemented. I will briefly describe the technique and provide relevant links to brush up on the concepts as and when necessary. In case you’re a newcomer to the world of time series, I suggest going through the following articles first:

Are you a beginner looking for a place to start your data science journey? We’re presenting a comprehensive course, full of knowledge and data science learning, curated just for you! This course covers everything from the basics of Machine Learning to Advanced concepts of ML, Deep Learning, and Time series.

Understanding the Problem Statement

We’ll dive into the implementation part of this article soon, but first, it’s important to establish what we aim to solve. Broadly, stock market machine learning analysis is divided into Fundamental Analysis and Technical Analysis.

  • Fundamental Analysis involves analyzing a company’s future profitability based on its current business environment and financial performance.
  • Technical Analysis, on the other hand, includes reading the charts and using statistical figures to identify the trends in the stock market.

As you might have guessed, our focus will be on the technical analysis part. We’ll be using a dataset from Quandl (you can find historical data for various stocks here) and for this particular project, I have used the data for ‘Tata Global Beverages’. Time to dive in!

Note: Here is the dataset I used for the code: Download

We will first load the dataset and define the target variable for the problem:

Python Code:

The dataset contains multiple variables: date, open, high, low, last, close, total_trade_quantity, and turnover.

  • The columns Open and Close represent the starting and final price at which the stock is traded on a particular day.
  • High, Low and Last represent the maximum, minimum, and last price of the share for the day.
  • Total Trade Quantity is the number of shares bought or sold in the day and Turnover (Lacs) is the turnover of the particular company on a given date.

Another important thing to note is that the market is closed on weekends and public holidays. Notice the above table again; some date values are missing—2/10/2018, 6/10/2018, and 7/10/2018. Of these dates, the 2nd is a national holiday, while the 6th and 7th fall on a weekend.

The profit or loss calculation is usually determined by the closing price of a stock for the day, hence we will consider the closing price as the target variable. Let’s plot the target variable to understand how it’s shaping up in our data:

#setting index as date
df['Date'] = pd.to_datetime(df.Date,format='%Y-%m-%d')
df.index = df['Date']

#plot
plt.figure(figsize=(16,8))
plt.plot(df['Close'], label='Close Price history')
Stock market machine learning

In the upcoming sections, we will explore these variables and use different techniques to predict the stock’s daily closing price.

Moving Average

Introduction

‘Average’ is easily one of the most common things we use in our daily lives. Calculating the average marks to determine overall performance or finding the average temperature of the past few days to get an idea about today’s temperature are all routine tasks we do on a regular basis. So, this is a good starting point to use on our dataset for making predictions.

The predicted closing price for each day will be the average of a set of previously observed values. Instead of using the simple average, we will use the moving average technique, which uses the latest set of values for each prediction. In other words, for each subsequent step, the predicted values are taken into consideration while removing the oldest observed value from the set. Here is a simple figure that will help you understand this more clearly.

Stock market machine learning

We will implement this technique on our dataset. The first step is to create a dataframe that contains only the Date and Close price columns, then split it into train and validation sets to verify our predictions.

Implementation

Just checking the RMSE does not help us understand how the model performed. Let’s visualize this to get a more intuitive understanding. Here is a plot of the predicted values along with the actual values.

#plot
valid['Predictions'] = 0
valid['Predictions'] = preds
plt.plot(train['Close'])
plt.plot(valid[['Close', 'Predictions']])

Inference

The RMSE value is close to 105, but the results are not very promising (as shown in the plot). The predicted values are of the same range as the observed values in the train set (initially, there is an increasing trend and then a slight decrease).

In the next section, we will examine two commonly used machine learning techniques—linear Regression and kNN—and see how they perform on our stock market machine learning data.

Linear Regression

Introduction

The most basic machine learning algorithm that can be implemented on this data is linear regression. The linear regression model returns an equation determining the relationship between the independent and dependent variables.

The equation for linear regression can be written as:

Here, x1, x2,….xn represent the independent variables while the coefficients θ1, θ2, …. θn  represent the weights. You can refer to the following article to study linear regression in more detail:

For our problem statement, we do not have a set of independent variables. Instead, we have only the dates. Let us use the date column to extract features like day, month, year, Monday/Friday, etc., and then fit a linear regression model.

Implementation

We will first sort the dataset in ascending order and then create a separate dataset so that any new feature created does not affect the original data.

#setting index as date values
df['Date'] = pd.to_datetime(df.Date,format='%Y-%m-%d')
df.index = df['Date']

#sorting
data = df.sort_index(ascending=True, axis=0)

#creating a separate dataset
new_data = pd.DataFrame(index=range(0,len(df)),columns=['Date', 'Close'])

for i in range(0,len(data)):
    new_data['Date'][i] = data['Date'][i]
    new_data['Close'][i] = data['Close'][i]
#create features
from fastai.structured import  add_datepart
add_datepart(new_data, 'Date')
new_data.drop('Elapsed', axis=1, inplace=True)  #elapsed will be the time stamp

This creates features such as:

‘Year’, ‘Month’, ‘Week’, ‘Day’, ‘Dayofweek’, ‘Dayofyear’, ‘Is_month_end’, ‘Is_month_start’, ‘Is_quarter_end’, ‘Is_quarter_start’,  ‘Is_year_end’, and  ‘Is_year_start’.

Note: I have used add_datepart from the fastai library. If you do not have it installed, you can simply use the command pip install fastai. Otherwise, you can create these features using simple for loops in Python. I have shown an example below.

Apart from this, we can add features that we believe would be relevant to the predictions. For instance, I hypothesize that the first and last days of the week could affect the stock’s closing price far more than the other days. So, I have created a feature that identifies whether a given day is Monday/Friday or Tuesday/Wednesday/Thursday. This can be done using the following lines of code:

new_data['mon_fri'] = 0
for i in range(0,len(new_data)):
    if (new_data['Dayofweek'][i] == 0 or new_data['Dayofweek'][i] == 4):
        new_data['mon_fri'][i] = 1
    else:
        new_data['mon_fri'][i] = 0

If the day of the week is equal to 0 or 4, the column value will be 1; otherwise, it will be 0. Similarly, you can create multiple features. If you have ideas for features to help predict stock prices, please share them in the comment section.

We will now split the data into train and validation sets to check the model’s performance.

#split into train and validation
train = new_data[:987]
valid = new_data[987:]

x_train = train.drop('Close', axis=1)
y_train = train['Close']
x_valid = valid.drop('Close', axis=1)
y_valid = valid['Close']

#implement linear regression
from sklearn.linear_model import LinearRegression
model = LinearRegression()
model.fit(x_train,y_train)

Results

#make predictions and find the rmse
preds = model.predict(x_valid)
rms=np.sqrt(np.mean(np.power((np.array(y_valid)-np.array(preds)),2)))
rms
121.16291596523156

The RMSE value is higher than the previous technique, showing that linear regression has performed poorly. Let’s look at the plot and understand why linear regression has not done well:

#plot
valid['Predictions'] = 0
valid['Predictions'] = preds

valid.index = new_data[987:].index
train.index = new_data[:987].index

plt.plot(train['Close'])
plt.plot(valid[['Close', 'Predictions']])

Inference

Linear regression is a simple technique and quite easy to interpret, but there are a few obvious disadvantages. One problem with using regression algorithms is that the model overfits the date and month column. Instead of taking into account the previous values from the point of prediction, the model will consider the value from the same date a month ago or the same date/month a year ago.

As seen from the plot above, the stock price dropped in January 2016 and January 2017. The model predicted the same for January 2018. A linear regression technique can perform well for problems such as Big Mart sales, where the independent features are useful for determining the target value.

k-Nearest Neighbours

Introduction

Another interesting ML algorithm for stock market prediction machine learning that one can use here is kNN (k nearest neighbors). Based on the independent variables, kNN finds the similarity between new and old data points. Let me explain this with a simple example.

Consider the height and age of 11 people. Based on given features (‘Age’ and ‘Height’), the table can be represented in a graphical format as shown below:

To determine the weight for ID #11, kNN considers the weight of the nearest neighbors of this ID. The weight of ID #11 is predicted to be the average of it’s neighbors. If we consider three neighbours (k=3) for now, the weight for ID#11 would be = (77+72+60)/3 = 69.66 kg.

For a detailed understanding of kNN, you can refer to the following articles:

Implementation

#importing libraries
from sklearn import neighbors
from sklearn.model_selection import GridSearchCV
from sklearn.preprocessing import MinMaxScaler
scaler = MinMaxScaler(feature_range=(0, 1))

Using the same train and validation set from the last section:

#scaling data
x_train_scaled = scaler.fit_transform(x_train)
x_train = pd.DataFrame(x_train_scaled)
x_valid_scaled = scaler.fit_transform(x_valid)
x_valid = pd.DataFrame(x_valid_scaled)

#using gridsearch to find the best parameter
params = {'n_neighbors':[2,3,4,5,6,7,8,9]}
knn = neighbors.KNeighborsRegressor()
model = GridSearchCV(knn, params, cv=5)

#fit the model and make predictions
model.fit(x_train,y_train)
preds = model.predict(x_valid)

Results

#rmse
rms=np.sqrt(np.mean(np.power((np.array(y_valid)-np.array(preds)),2)))
rms
115.17086550026721

The RMSE value does not differ greatly, but a plot of the predicted and actual values should provide a clearer picture.

#plot
valid['Predictions'] = 0
valid['Predictions'] = preds
plt.plot(valid[['Close', 'Predictions']])
plt.plot(train['Close'])
Stock market machine learning

Inference

The RMSE value is almost similar to the linear regression model, and the plot shows the same pattern. Like linear regression, kNN also identified a drop in January 2018 since that has been the pattern for years. We can safely say that regression algorithms have not performed well on this dataset.

Let’s examine some time series forecasting techniques to see how they perform when faced with this stock price prediction challenge.

Auto ARIMA

Introduction

ARIMA is a very popular statistical method for time series forecasting. ARIMA models take into account the past values to predict the future values. There are three important parameters in ARIMA:

  • p (past values used for forecasting the next value)
  • q (past forecast errors used to predict the future values)
  • d (order of differencing)

Parameter tuning for ARIMA consumes a lot of time. So we will use auto ARIMA, which automatically selects the best combination of (p,q,d) to provide the least error. To read more about how auto ARIMA works, refer to this article:

Implementation

from pyramid.arima import auto_arima

data = df.sort_index(ascending=True, axis=0)

train = data[:987]
valid = data[987:]

training = train['Close']
validation = valid['Close']

model = auto_arima(training, start_p=1, start_q=1,max_p=3, max_q=3, m=12,start_P=0, seasonal=True,d=1, D=1, trace=True,error_action='ignore',suppress_warnings=True)
model.fit(training)

forecast = model.predict(n_periods=248)
forecast = pd.DataFrame(forecast,index = valid.index,columns=['Prediction'])

Results

rms=np.sqrt(np.mean(np.power((np.array(valid['Close'])-np.array(forecast['Prediction'])),2)))
rms
44.954584993246954
#plot
plt.plot(train['Close'])
plt.plot(valid['Close'])
plt.plot(forecast['Prediction'])

Inference

As we saw earlier, an auto ARIMA model uses past data to understand the pattern in the time series. Using these values, the model captured an increasing trend in the series. Although the stock market predictions using this machine learning are far better than those of the previously implemented machine learning models, these predictions are still not close to the real values.

As the plot shows, the model has captured a trend in the series but does not focus on the seasonality. In the next section, we will implement a time series model that takes both trend and seasonality into account.

Prophet

Introduction

Several time series techniques can be implemented on the stock prediction machine learning dataset, but most of these techniques require extensive data preprocessing before fitting the model. Prophet, designed and pioneered by Facebook, is a time series forecasting library that requires no data preprocessing and is extremely simple to implement. The input for Prophet is a dataframe with two columns: date and target (ds and y).

Prophet tries to capture the seasonality in the past data and works well when the dataset is large. Here is an interesting article that explains Prophet simply and intuitively:

Implementation

#importing prophet
from fbprophet import Prophet

#creating dataframe
new_data = pd.DataFrame(index=range(0,len(df)),columns=['Date', 'Close'])

for i in range(0,len(data)):
    new_data['Date'][i] = data['Date'][i]
    new_data['Close'][i] = data['Close'][i]

new_data['Date'] = pd.to_datetime(new_data.Date,format='%Y-%m-%d')
new_data.index = new_data['Date']

#preparing data
new_data.rename(columns={'Close': 'y', 'Date': 'ds'}, inplace=True)

#train and validation
train = new_data[:987]
valid = new_data[987:]

#fit the model
model = Prophet()
model.fit(train)

#predictions
close_prices = model.make_future_dataframe(periods=len(valid))
forecast = model.predict(close_prices)

Results

#rmse
forecast_valid = forecast['yhat'][987:]
rms=np.sqrt(np.mean(np.power((np.array(valid['y'])-np.array(forecast_valid)),2)))
rms
57.494461930575149
#plot
valid['Predictions'] = 0
valid['Predictions'] = forecast_valid.values

plt.plot(train['y'])
plt.plot(valid[['y', 'Predictions']])
Stock market machine learning

Inference

Prophet (like most time series forecasting techniques) tries to capture the trend and seasonality from past data. This model usually performs well on time series datasets but fails to live up to its reputation in this case.

As it turns out, stock prices do not have a particular trend or seasonality. They depend highly on what is currently going on in the market, and thus, the prices rise and fall. Hence, forecasting techniques like ARIMA, SARIMA, and Prophet would not show good results for this particular problem.

Let us go ahead and try another advanced technique – Long Short Term Memory (LSTM).

Long Short Term Memory (LSTM)

Introduction

LSTMs are widely used for sequence prediction problems and have proven extremely effective. They work so well because LSTM can store past important information and forget the information that is not. LSTM has three gates:

  • The input gate: The input gate adds information to the cell state
  • The forget gate: It removes the information that the model no longer requires
  • The output gate: Output Gate at LSTM selects the information to be shown as output

For a more detailed understanding of LSTM and its architecture, you can go through the below article:

For now, let us implement LSTM as a black box and check its performance on our particular data.

Implementation

#importing required libraries
from sklearn.preprocessing import MinMaxScaler
from keras.models import Sequential
from keras.layers import Dense, Dropout, LSTM

#creating dataframe
data = df.sort_index(ascending=True, axis=0)
new_data = pd.DataFrame(index=range(0,len(df)),columns=['Date', 'Close'])
for i in range(0,len(data)):
    new_data['Date'][i] = data['Date'][i]
    new_data['Close'][i] = data['Close'][i]

#setting index
new_data.index = new_data.Date
new_data.drop('Date', axis=1, inplace=True)

#creating train and test sets
dataset = new_data.values

train = dataset[0:987,:]
valid = dataset[987:,:]

#converting dataset into x_train and y_train
scaler = MinMaxScaler(feature_range=(0, 1))
scaled_data = scaler.fit_transform(dataset)

x_train, y_train = [], []
for i in range(60,len(train)):
    x_train.append(scaled_data[i-60:i,0])
    y_train.append(scaled_data[i,0])
x_train, y_train = np.array(x_train), np.array(y_train)

x_train = np.reshape(x_train, (x_train.shape[0],x_train.shape[1],1))

# create and fit the LSTM network
model = Sequential()
model.add(LSTM(units=50, return_sequences=True, input_shape=(x_train.shape[1],1)))
model.add(LSTM(units=50))
model.add(Dense(1))

model.compile(loss='mean_squared_error', optimizer='adam')
model.fit(x_train, y_train, epochs=1, batch_size=1, verbose=2)

#predicting 246 values, using past 60 from the train data
inputs = new_data[len(new_data) - len(valid) - 60:].values
inputs = inputs.reshape(-1,1)
inputs  = scaler.transform(inputs)

X_test = []
for i in range(60,inputs.shape[0]):
    X_test.append(inputs[i-60:i,0])
X_test = np.array(X_test)

X_test = np.reshape(X_test, (X_test.shape[0],X_test.shape[1],1))
closing_price = model.predict(X_test)
closing_price = scaler.inverse_transform(closing_price)

Results

rms=np.sqrt(np.mean(np.power((valid-closing_price),2)))
rms
11.772259608962642
#for plotting
train = new_data[:987]
valid = new_data[987:]
valid['Predictions'] = closing_price
plt.plot(train['Close'])
plt.plot(valid[['Close','Predictions']])
Stock market machine learning

Inference

Wow! The LSTM model can be tuned for various parameters, such as changing the number of LSTM layers, adding a dropout value, or increasing the number of epochs. But are the predictions from LSTM enough to identify whether the stock price will increase or decrease? Certainly not!

As I mentioned at the start of the article, stock price is affected by news about the company and other factors like demonetization or merger/demerger. Certain intangible factors as well can often be impossible to predict beforehand.

Conclusion

Time series forecasting is a very intriguing field, as I have realized while writing these articles. The community perceives it as a complex field, and while there is a grain of truth in that, it’s not so difficult once you get the hang of the basic techniques.

Frequently Asked Questions

Q1. Is it possible to predict the stock market with Deep Learning?

A. Yes, it is possible to predict the stock market with Deep Learning algorithms such as moving average, linear regression, Auto ARIMA, LSTM, and more.

Q2. What can you use to predict stock prices in Deep Learning?

A. Moving average, linear regression, KNN (k-nearest neighbor), Auto ARIMA, and LSTM (Long Short Term Memory) are some of the most common Deep Learning algorithms that predict stock prices.

Q3. What are the two methods to predict stock price?

A. Fundamental Analysis and Technical Analysis are the two ways of analyzing and predicting stock prices.

AishwaryaSingh 13 Aug, 2024

Frequently Asked Questions

Lorem ipsum dolor sit amet, consectetur adipiscing elit,

Responses From Readers

Clear

James Verdant
James Verdant 25 Oct, 2018

Isn't the LSTM model using your "validation" data as part of its modeling to generate its predictions since it only goes back 60 days. Your other techniques are only using the "training" data and don't have the benefit of looking back 60 days from the target prediction day. Is this a fair comparison?

Jay
Jay 25 Oct, 2018

Getting index error - --------------------------------------------------------------------------- IndexError Traceback (most recent call last) in 1 #Results ----> 2 rms=np.sqrt(np.mean(np.power((np.array(valid['Close']) - np.array(valid['Predictions'])),2))) 3 rms IndexError: only integers, slices (`:`), ellipsis (`...`), numpy.newaxis (`None`) and integer or boolean arrays are valid indices

Pankaj
Pankaj 26 Oct, 2018

After running the following codes- train['Date'].min(), train['Date'].max(), valid['Date'].min(), valid['Date'].max() (Timestamp('2013-10-08 00:00:00'), Timestamp('2017-10-06 00:00:00'), Timestamp('2017-10-09 00:00:00'), Timestamp('2018-10-08 00:00:00')) I am getting the following error : name 'Timestamp' is not defined Please help.

Zarief Marzuki Lao
Zarief Marzuki Lao 26 Oct, 2018

Hi Aishwarya, Just curious. LSTM works just TOO well !! Is splitting dataset to train & valid step carry out after the normalizing step ?! i.e. #converting dataset into x_train and y_train scaler = MinMaxScaler(feature_range=(0, 1)) scaled_data = scaler.fit_transform(dataset) then dataset = new_data train = dataset[:987] valid = dataset[987:] Then this x_train, y_train = [], [] for i in range(60,len(train )): # <- replace dataset with train ?! x_train.append(scaled_data[i-60:i,0]) y_train.append(scaled_data[i,0]) x_train, y_train = np.array(x_train), np.array(y_train) Guide me on this. Thanks

rohit
rohit 26 Oct, 2018

hi I'm getting below error... ''">>> #import packages >>> import pandas as pd Traceback (most recent call last): File "", line 1, in import pandas as pd ImportError: No module named pandas""""

Shan
Shan 27 Oct, 2018

Hi.. Thanks for nicely elaborating LSTM implementation in the article. However, in LSTM rms part if you can guide, as I am getting the following error : valid['Predictions'] = 0.0 valid['Predictions'] = closing_price rms=np.sqrt(np.mean(np.power((np.array(valid['Close'])-np.array(valid['Predictions'])),2))) rms ############################################################################## IndexError Traceback (most recent call last) in () ----> 1 valid['Predictions'] = 0.0 2 valid['Predictions'] = closing_price 3 rms=np.sqrt(np.mean(np.power((np.array(valid['Close'])-np.array(valid['Predictions'])),2))) 4 rms IndexError: only integers, slices (`:`), ellipsis (`...`), numpy.newaxis (`None`) and integer or boolean arrays are valid indices

Jay
Jay 27 Oct, 2018

Still same error - --------------------------------------------------------------------------- IndexError Traceback (most recent call last) in ----> 1 valid['Predictions'] = closing_price IndexError: only integers, slices (`:`), ellipsis (`...`), numpy.newaxis (`None`) and integer or boolean arrays are valid indices And also why are we doing valid['Predictions'] = 0 and valid['Predictions'] = closing_price instead of valid['Predictions'] = closing_price

Roberto
Roberto 27 Oct, 2018

Hi, Nice article. I have installed fastai but I am getting the following error: ModuleNotFoundError: No module named 'fastai.structured' Any idea?

Jingmiao
Jingmiao 29 Oct, 2018

Hello AISHWARYA, I dont know what's your motivation to spend such a long time to write this blog But thank you soooooo much!!! Appreciate your time for both the words and codes (easy to follow) !!! Such a great work!!!

Vedamurthy KB
Vedamurthy KB 31 Oct, 2018

new_data.index=data['Date'] valid.index=new_data[987:].index train.index=new_data[:987].index gives AttributeError Traceback (most recent call last) in () 1 new_data.index=data['Date'] ----> 2 valid.index=new_data[987:].index 3 train.index=new_data[:987].index AttributeError: 'numpy.ndarray' object has no attribute 'index' valid['Predictions'] = 0 valid['Predictions'] = closing_price gives IndexError Traceback (most recent call last) in () ----> 1 valid['Predictions'] = 0 2 valid['Predictions'] = closing_price IndexError: only integers, slices (`:`), ellipsis (`...`), numpy.newaxis (`None`) and integer or boolean arrays are valid indices V good article. I am getting above errors. Kindly help solving it

Ravi Shankar
Ravi Shankar 31 Oct, 2018

Hi, Thanks for putting the efforts in writing the article. If you believe LSTM model works this well, try buying few shares of Tata Global beverages and let us know the returns on the same. I guess, you would understand the concept of over-fit. Thanks, Ravi

Moobi
Moobi 11 Nov, 2018

Its nice tutorial, thanks. I need to know how can i predict just tomorrow’s price?

Pabitra
Pabitra 12 Nov, 2018

What is the difference between last and closing price?

Miguel
Miguel 13 Nov, 2018

Hi, thanks for the article. I got this error: rms=np.sqrt(np.mean(np.power((np.array(valid['Close'])-preds),2))) ValueError: operands could not be broadcast together with shapes (1076,) (248,) Can you help me on this?

Alex
Alex 24 Nov, 2018

Thank you so much for the code, you inspire me a lot. I applied your algorithm for my example and it it works fine. Now i have to make it predict the price for the next 5 years, do you know how to achieve that? I should use all the data without splitting into test and train for training and somehow generate new dates in that array, then predict the value for them. I tried to modify your code but i couldn't figure it out. I'm new to ML and it's really hard to understand those functions and classes. Thank you very much in advance

Shabbir Anaswala
Shabbir Anaswala 05 Dec, 2018

Hi, I am getting this error x_train_scaled = scaler.fit_transform(x_train) Traceback (most recent call last): File "", line 1, in x_train_scaled = scaler.fit_transform(x_train) File "C:\ProgramData\Anaconda3\lib\site-packages\sklearn\base.py", line 517, in fit_transform return self.fit(X, **fit_params).transform(X) File "C:\ProgramData\Anaconda3\lib\site-packages\sklearn\preprocessing\data.py", line 308, in fit return self.partial_fit(X, y) File "C:\ProgramData\Anaconda3\lib\site-packages\sklearn\preprocessing\data.py", line 334, in partial_fit estimator=self, dtype=FLOAT_DTYPES) File "C:\ProgramData\Anaconda3\lib\site-packages\sklearn\utils\validation.py", line 433, in check_array array = np.array(array, dtype=dtype, order=order, copy=copy) TypeError: float() argument must be a string or a number, not 'Timestamp'

shourav
shourav 06 Dec, 2018

Hi AISHWARYA, It's a great initiative. Can you please explain the process of getting the future price of a stock ? You told that validation set should have only one row. Can you please explain this ?

Shourav
Shourav 07 Dec, 2018

Hello AISHWARYA , It's a great initiative . Can you please explain how to approach if I want to predict the next day price ? You explained it in the previous comments but I didn't get the full meaning "the validation set should have only 1 row (with past 60 values)" . It will be helpful if you give an example. Thanks in advance .

Sascha
Sascha 14 Dec, 2018

Hello Aishwarya Great article. Thank you so much. I read the above comments but it's still unclear for me what or where I have to change to predict only tomorrow's price for example. Your help is much appreciated. Thanks and regards, Sascha #predicting 246 values, using past 60 from the train data inputs = new_data[len(new_data) - len(valid) - 60:].values inputs = inputs.reshape(-1,1) inputs = scaler.transform(inputs) X_test = [] for i in range(60,inputs.shape[0]): X_test.append(inputs[i-60:i,0]) X_test = np.array(X_test) X_test = np.reshape(X_test, (X_test.shape[0],X_test.shape[1],1)) closing_price = model.predict(X_test) closing_price = scaler.inverse_transform(closing_price)

Sumanth
Sumanth 19 Dec, 2018

Hi Aishwarya, Can you please eloborate on where to change the code for 1 day or 3 days predictions? Thanks, Sumanth

Ricky
Ricky 20 Dec, 2018

Hello AISHWARYA SINGH, Nice article...I had been working FOREX data to use seasonality to predict the next days direction for many weeks and your code under the FastAi part gave me an idea on how to go about it. Thanks for a great article. Have you tried predicting the stock data based bulls and bears only, using classification? I used np.log(df['close']/df.['close'].shift(1)) to find the returns and if its negative I use "np.where" to assign -1 to it and if its positive I assign 1 to it. All I want to predict is if tomorrow would be 1 or -1. I have not been able to get a 54% accuracy from that model. a 60% would be very profitable when automated. would you want to see my attempts?

Bala Sridhar
Bala Sridhar 25 Dec, 2018

Aishwarya: I am neither a quant nor an expert programmer. But I see a problem in how you constructed your models. Instead of LSTM if you were to simply predict tomorrow's price as today's closing price you will end up with a nice graph and what appears to be highly accurate prediction. What is more appropriate is to try to predict whether next day's return is positive or negative and check the correct answer against total predictions. In general, the data needs to be "stationary" and you need to take either returns per day or price differences to come up with the correct model. Let me know if you agree with me.

shubham sharma
shubham sharma 28 Dec, 2018

Thanks for sharing valuable information. a worth reading blog. I try this algorithm for my example and it works excellent.

ice
ice 31 Dec, 2018

Hi Asihwarya, As far as i understand, the model takes in 60 days of real data to predict the next day's value in LSTM. I wonder how the results will be like if you take a predicted value to predict the next value instead. This will allow us to predict say 2 years of data for long term trading.

Samir musa
Samir musa 02 Jan, 2019

Dear Aishwarya, Thank you for this informative article. I am a trader but i am looking for ways to automate my decisions, not necessarily to the point of machine putting the order but as a guide to reduces the burden of decision and pattern recognition. i am experimenting with dynamic time warping to recognise some of the price action behavior, which i have learned after thousands of hours of observation. I am using kmeans, and some combination of euclidean distance to achieve that. What i am hoping is that, one day you produced an article that could show how dtw algorithm could be use to identify identical behaviour in stock price action. You know how dtw works in speak recognition, it could be use to identify one behavior that could happen in hundredth of ways. There are many academic research on the topic but few practical examples. I hope as data science professional, you could be able to write something on this. However, if you know any article that Vidhya have publish about dtw or speech recognition. Please, do point out them to me, as there are many articles to do it.

Jwala
Jwala 02 Jan, 2019

Hi Aishwarya, I am not able to download the dataset getting empty CSV file with header. Could you please help me.

Run Ma
Run Ma 03 Jan, 2019

I am also not able to download the test data (NSE-TATAGLOBAL(1).csv), could you send me? thanks!

Chris
Chris 04 Jan, 2019

Could you send me the full working code, i cant seem to get it to work

Doaa
Doaa 04 Jan, 2019

Hi Aishwarya, Thanks for sharing this great Information. I am new to LSTM so can you help me 1- I confused of how I predict next day price while I do not have (open-close-high - low) for the day I want to predict the price in it. 2- if you can give me some tutorials or books to learn in details how can I use lstm to predict stock price in short term and long term. Thanks in advance

chao
chao 05 Jan, 2019

could you kindly share me the full code, I have strong interest in time series analysis

Doaa
Doaa 06 Jan, 2019

Hi AISHWARYA SINGH, thanks a lot for your great article .

Doaa
Doaa 06 Jan, 2019

please how can I predict sock price for next day or for 200 days ahead while I do not have test data (high - low - open - volume ) for days I want to predict the stock price of It. sorry because I am new in DL. If you can tell me about tutorials explain how can I use LSTM in details for stock price prediction Thanks in advance

chao
chao 06 Jan, 2019

could you share the full code, I have strong interest in time series analysis

Sandeep
Sandeep 07 Jan, 2019

Great article Aishwarya. Kudos to you for taking out to the time to explain it in detail. Can you send me a full working code.

Vidyut
Vidyut 08 Jan, 2019

Hi Aishwarya, when I downloaded the .csv file, the dates are shown as a bunch of '######'. Could you please send me the code and the dataset through email?

Punit
Punit 08 Jan, 2019

Good article to understand LSTM. Thankyou for this tutorial, really appreciable. As I understood from this, if I want to predict the stock price for tomorrow I can use len(valid) = 1 before the following line: inputs = new_data[len(new_data) - len(valid) - 60:].values If I use len(valid) = 2, will it give me price for tomorrow and day after or tomorrow and today ? I am little bit confuse. Also, when I run this model for Reliance Industries stock, it gives me mse ~41. I am using 4000 as training set and 1184 as valid set. Does it make any difference ?

Lyn
Lyn 10 Jan, 2019

Hi ! Thank you for the tutorial ! I wonder is it possible to get Confidence Percentage for each predictions ?

Yibin Ng
Yibin Ng 10 Jan, 2019

Hi, Thanks for the very interesting article. I have a comment with the way you scale the dataset in this line of code: scaled_data = scaler.fit_transform(dataset) Here you are taking the max and min values of the *entire* dataset to do scaling. Later you use the same max and min values to scale the test set in this line of code: inputs = scaler.transform(inputs) However in practice we will not have any prior information about the test set and so we will not know their max and min values. I think if you do fit_transform on the train set is more correct ie. scaled_data = scaler.fit_transform(train).

Robert
Robert 12 Jan, 2019

Hi Aishwarya, The link provided in this article seems to be a lot smaller (min date is 3-2-2017) where in your article is spoken about year 2013... Could you please help me providing te Original CSV file? Thanx!

CoolJJ
CoolJJ 12 Jan, 2019

Could you send me the full working code? Have a nice day!

Lyn
Lyn 14 Jan, 2019

Hi ! Nice tutorial ! I wonder if you know how to get probability for each prediction ? Thank you

Sanat Mishra
Sanat Mishra 14 Jan, 2019

Hello....very good article I am new to this field please guide on 1. how to save the trained model 2. how to predict next day value using this model. what input will be required and how to input same thank ou very much in advance

Rishi
Rishi 17 Jan, 2019

Hi, was wondering with your lstm code how you would do a prediction of a price or prices in the future. I have tried modifying the dataset with future dates and setting the values to 0 to see it predict it however seems like this reduces the accuracy a lot. Also was wondering with this line ***scaled_data = scaler.fit_transform(dataset)*** does that mean we are training our model on the whole dataset because i don't see the variable ***train = dataset[0:987,:]*** be really ever used.

Debasish Chatterjee
Debasish Chatterjee 18 Jan, 2019

Hi...very nice tutorial. I am new to data science, though had some idea on Statistics but pretty new to LSTM. Could you please guide me in how can I predict future values? 1. The data base I have downloaded had 410 rows, so I have changed the training data set to 307 rows (75%). The last value is for date 28/09/18 2. I have followed all your codes and LSTM seems pretty good fit. 3. Getting error while calculating rms (rms=np.sqrt(np.mean(np.power((valid-closing_price),2))) 4. Error received: ValueError: Unable to coerce to DataFrame, shape must be (103, 2): given (103, 1) 5. The code required to predict future values. Say first I want to predict for 29/09/18. After that, If I want to use it for predicting value for 30/09/18 and so on. Thank you in advance. Cheers!

Sergey
Sergey 19 Jan, 2019

As a training material this article is very good. Thank you Aishwarya! As a practical tool this method is useless. It will not even predict where the price will close tomorrow - above or below of today's market close.

Emile
Emile 19 Jan, 2019

Hi AISHWARYA SINGH, Your article is very interesting, specialy the LSTM section. I'am trying to run your code, but I have problems with the libraries in Python 2.7 and in Python 3.6. Could you, please, specify the Python version and the libraries needed with their dependancies and their version number ? Thank you very much in advance.

Emerson Oliveira
Emerson Oliveira 20 Jan, 2019

Why 987 in: train = new_data[:987] valid = new_data[987:]

Emerson Oliveira
Emerson Oliveira 20 Jan, 2019

How can I solve this problem: ModuleNotFoundError: No module named 'fastai.structured' Thanks

Pramesh Bajracharya
Pramesh Bajracharya 22 Jan, 2019

Amazing Article. Very helpful . Thanks.

satya repala
satya repala 23 Jan, 2019

for predicting values, why are you taking values from validation set at each iteration. it should be like this, 987 + 60 --> predict -->987+60+1 th value take 988+59+(1(predicted)) --> predict 988+60+1 th value let me know if i am wrong. if you are taking values from validation set everytime, your RMS error going to be less as expected.so we need to take first 60 values from validation set and predict remaining validation set points.

Dipayan
Dipayan 26 Jan, 2019

Hello Aishwarya By any chance you changed the length of the dataset?? When I download, it gives me the dataset name as 'NSE-BSE' and it contains 411 rows. Splitting at 987 won't leave anything for validation. OR am I not downloading the right dataset ??

C
C 26 Jan, 2019

Hello, I have questions about your model LSTM. I don't understand the building of "X_test". I have understood the aim and the bulding of your variable "inputs", but I believe that your variable "X_test" use the observed values. Imagine that the dataset has 500 values and we use the first 400 for the train and you want to predict the others. For the first day, you will use the values 340 until 400 and you can predict the value 401. Ok no problem ! For the second day, you will use the values 341 until 401. If you want to predict on several months, you have to use the predicted value 401. But your value 401 is the value contained in the list "inputs". So you don't use the predicted variable but you use the observed value. Actually if your model can predict on 2 months, I can replace values in valid and the curve of predictions can't change (because I don't modify the train). I tried to replace all the observed values between 401 and 500 by 0... The predicted values change completely ! I feel that you do predictions using the real value of the day before to compute the actual day but you don't predict the stock market on several months. My question is : what part didn't I understand ? I thank you and have a good day :)

@run
@run 27 Jan, 2019

Hey Aishwarya, your article is super helpful. Thanks a ton. Keep going!!!

Frank
Frank 28 Jan, 2019

Thanks a lot for sharing this. I keep wondering if we just use the previous close as a prediction for next day, would the chart essentially the same as the LTMS? Can LTMS beat that simple prediction?

deep
deep 28 Jan, 2019

Hey Aishwarya, a related question. So say if you need to build a system based on LSTM to predict 50 stocks on any day. Would you create 50 LSTM models and select the right stock model at runtime OR train one LSTM model only for all 50 stocks ?? What would be your approach ?

antoine73
antoine73 29 Jan, 2019

Hello, I do not understand the construction of forecasts in the LSTM method. I note N the first value to predict. The objective is to compare the predicted values N to N + 10 ​​with the observed values. To predict the value N, you take the 60 previous observed values ​​ok. To predict the value N + 1, what values ​​do you take? I don't understand if you use : - 59 previous observed values ​​without taking the value in N - 59 previous values ​​+ the predictive value in N - 59 previous values ​​+ the observed value in N I try to understand the methodology but this little point is blocking me. All the rest of the article is very clear. I thank you :)

Doaa
Doaa 29 Jan, 2019

Hi Aishwarya, please can you tell me about books explain how to use LSTM in stock market in details thanks in advance

MikeMikeMike
MikeMikeMike 30 Jan, 2019

Aishwaryai, I found your article very interesting. I'm not familiar with programing but I was able to follow the reason for each step in the process. I do have a question. You noted that there are many other factors that will ultimately affect the market. As someone who works in marketing and has to be plugged into social media I use many tools that tell me how a company is being perceived. Twitter and Facebook both offer lot of opportunity for social listening and the tools we use in advertising/marketing to measure take the temperature of the market are quite powerful. Would it be possible to incorporate some machine learning to find patterns in the positive/negative sentiment measurements that are constantly active to help predict when stock values are going to change and in which direction? I feel like your article is using current and past market data but doesn't incorporate social perception. http://www.aclweb.org/anthology/W18-3102 The study linked above only focuses on stock related posts made from verified accounts but didn't take public perception measurements into account in their data. With how connected to social media the world has become I wonder if combining the data you used in your article with data from social listening tools used by marketers could create predictive tools that would essentially game the system.

Rahul Saini
Rahul Saini 01 Feb, 2019

Hi AISHWARYA SINGH, You blog is very impressive. Could you please help me for building a system with following requirements: The system will do trade training using huge data (historical/past trades and stocks prices) and incremental data from new daily trades and prices. The system will include a user-friendly interface which means that everyone even people with low knowledge with computers will be able to use this system on any computer/laptop. The user will have the option to run a prediction any time he wants, the user will be able to make the system to do trade training any time he wants. As mentioned above the system will be able to do training based on historical/past data and incremental new daily data, which means data will be updated consistently so the system will be always updated. Thanks in advance

Slavko
Slavko 02 Feb, 2019

# Complete code with yahoo data reader # Calculated dataset length trainig set length and prediction length (plen) # coding: utf-8 # In[1]: #import packages from pandas_datareader import data import fix_yahoo_finance as yf yf.pdr_override() # <== that's all it takes :-) import pandas as pd import numpy as np #to plot within notebook import matplotlib.pyplot as plt get_ipython().run_line_magic('matplotlib', 'inline') #setting figure size from matplotlib.pylab import rcParams rcParams['figure.figsize'] = 20,10 #for normalizing data from sklearn.preprocessing import MinMaxScaler scaler = MinMaxScaler(feature_range=(0, 1)) #read the file #df = pd.read_csv('NSE-BSE.csv') #df['Date'] = df.Date.apply( # lambda x: pd.to_datetime(x).strftime('%Y-%m-%d')) # Define the instruments to download. We would like to see Apple, Microsoft and the S&P500 index. #tickers = 'PYPL' tickers = 'AMD' # We would like all available data from 01/01/2000 until 12/31/2016. start_date = '2015-01-01' end_date = '2019-02-01' # download dataframe df = data.get_data_yahoo(tickers, start=start_date, end=end_date) df.columns = [c.replace(' ', '_') for c in df.columns] #print the tail df.tail() #df.dtypes # In[2]: #plot plt.figure(figsize=(16,8)) plt.plot(df['Close'], label='Close Price history') # In[3]: #importing required libraries from sklearn.preprocessing import MinMaxScaler from keras.models import Sequential from keras.layers import Dense, Dropout, LSTM #creating dataframe data = df.sort_index(ascending=True, axis=0) new_data = pd.DataFrame(index=range(0,len(df)),columns=['Date', 'Close']) for i in range(0,len(data)): new_data['Date'][i] = pd.to_datetime(df.index[i], unit='D').date() new_data['Close'][i] = data['Close'][i] #setting index new_data.index = new_data.Date new_data.drop('Date', axis=1, inplace=True) #Predict number of days plen = 80 #creating train and test sets dataset = new_data.values train = dataset[0:len(data)-plen,:] valid = dataset[len(data)-plen:,:] # Display data set len new_data.shape, train.shape, valid.shape # In[4]: #converting dataset into x_train and y_train scaler = MinMaxScaler(feature_range=(0, 1)) scaled_data = scaler.fit_transform(dataset) x_train, y_train = [], [] for i in range(plen,len(train)): x_train.append(scaled_data[i-plen:i,0]) y_train.append(scaled_data[i,0]) x_train, y_train = np.array(x_train), np.array(y_train) x_train = np.reshape(x_train, (x_train.shape[0],x_train.shape[1],1)) # In[5]: # create and fit the LSTM network model = Sequential() model.add(LSTM(units=50, return_sequences=True, input_shape=(x_train.shape[1],1))) model.add(LSTM(units=50)) model.add(Dense(1)) model.compile(loss='mean_squared_error', optimizer='adam') model.fit(x_train, y_train, epochs=1, batch_size=1, verbose=2) # In[6]: #predicting values, using past plen from the train data inputs = new_data[len(new_data) - len(valid) - plen:].values inputs = inputs.reshape(-1,1) inputs = scaler.transform(inputs) X_test = [] for i in range(plen,inputs.shape[0]): X_test.append(inputs[i-plen:i,0]) X_test = np.array(X_test) X_test = np.reshape(X_test, (X_test.shape[0],X_test.shape[1],1)) closing_price = model.predict(X_test) closing_price = scaler.inverse_transform(closing_price) # In[7]: rms=np.sqrt(np.mean(np.power((valid-closing_price),2))) rms # In[9]: #for plotting train = new_data[:len(data)-plen] valid = new_data[len(data)-plen:] valid['Predictions'] = 0 valid['Predictions'] = closing_price dts=train['Close'] dtv=valid[['Close','Predictions']] plt.figure(figsize=(16,8)) plt.grid() plt.plot(dts[plen:]) plt.plot(dtv[:plen]) dtv.tail()

Harmeet
Harmeet 03 Feb, 2019

Hi there, thanks for the detailed explanation. Just a simple question.. The data set had many features but we chose 'Date' feature to work with. Why didn't we consider other features? What is the approach that we take while choosing the feature set? And, what the need to separate Date feature to do linear regression? Thanks

Pratik
Pratik 04 Feb, 2019

HI Aishwarya my name is pratik am a research engineer i have done similar work not only with stock data but also desigjn sentiment analyzer to get the sentiment of the news as well as the twitter data to get the cumilative score for final prediction i have use cnn and random forest i think ur normalization need to be modified there use EWMA panda techinique to smoothen the results and it will enhance ur prediction i think we have to use reinforcement technique as a reward for dip in prioce and high in price due to news or various factors this will enhace the results i didnt try yet but i feel reinforement is the solution for stock price problem

Alok
Alok 07 Feb, 2019

Hi, Aishwarya I am working on research on the same topic and I have a strong interest in this. SO, could you please send me the full working of this article along with the code. My email id is [email protected]

Archit Jain
Archit Jain 07 Feb, 2019

help me to solve this error rms=np.sqrt(np.mean(np.power((np.array(valid['Close'])-preds),2))) ValueError: operands could not be broadcast together with shapes (0,) (248,)

Sam
Sam 09 Feb, 2019

Hi Aishwarya, could you share the full code please. I am getting the below error: ValueError Traceback (most recent call last) in () 1 #calculate rmse ----> 2 rms=np.sqrt(np.mean(np.power((np.array(valid['Close'])-preds),2))) ValueError: operands could not be broadcast together with shapes (0,) (248,) 1 ​

Muralidharan
Muralidharan 11 Feb, 2019

Hi How I can solve this error? AttributeError: 'DataFrame' object has no attribute 'Date' Thanks

Muralidharan
Muralidharan 11 Feb, 2019

Hi After dropping the date column(Linear Regression method), I am getting an error like this. AttributeError: 'DataFrame' object has no attribute 'Date' How do I resolve this error?

Muralidharan
Muralidharan 11 Feb, 2019

Hi Could please share the entire working code. Thanks

Haroon Balwa
Haroon Balwa 11 Feb, 2019

Thank you for this great blog! I have a question of how to code the following issues in the given dataset of data_stock.csv 1. Which stocks are apparently similar in performance 2. Identify which all stocks are moving together and which all stocks are diff from each other? 3. How many unique patterns that exist in the historical stock data set, based onn fluctuations in price.

Alex
Alex 18 Feb, 2019

Make no mistake, at the expense of LSTM. The model simply takes the data from the previous days and follow trends. This model can't even predict up or down the stock will go. It's easy to check with the following code-the prediction accuracy is only 47%. valid['R_Close'] = valid['Close'].diff()/valid['Close'].shift(1) valid['R_Pred'] = valid['Predictions'].diff()/valid['Predictions'].shift(1) valid['event'] = valid['R_Close']/valid['R_Pred'] print(len(valid[valid['event'] > 0])/len(valid)) plt.plot(valid[['R_Close','R_Pred']])

Tanmay Jain
Tanmay Jain 21 Feb, 2019

--------------------------------------------------------------------------- i am getting this error please help me to debug the error. ValueError Traceback (most recent call last) in () 3 4 #calculate rmse ----> 5 rms=np.sqrt(np.mean(np.power((np.array(valid['Close'])-preds),2))) 6 rms ValueError: operands could not be broadcast together with shapes (0,) (248,)

srujith
srujith 22 Feb, 2019

Hi, thanks for the article. I got this error: rms=np.sqrt(np.mean(np.power((np.array(valid[‘Close’])-preds),2))) ValueError: operands could not be broadcast together with shapes (1076,) (248,) Can you help me on this?

srujith
srujith 22 Feb, 2019

Hi, i got this error can you please help me through this #calculate rmse rms=np.sqrt(np.mean(np.power((np.array(valid['Close'])-preds),2))) print (rms) ValueError Traceback (most recent call last) in () 1 #calculate rmse ----> 2 rms=np.sqrt(np.mean(np.power((np.array(valid['Close'])-preds),2))) 3 print (rms) 4 ValueError: operands could not be broadcast together with shapes (410,) (248,)

MorfiMen
MorfiMen 25 Feb, 2019

How can I predict the future 30 values? thank you so much

Alex
Alex 25 Feb, 2019

You should consider implementing the simplest model you can think of. Since your LSTM uses all data up to a given date to predict just the next date, a good benchmark model would be the following: a model, that simply repeats the value from the last day to predict the value for the following day. In pandas this is simply implemented as: prediction = new_data.shift(1) To get a plot of that model: new_data.join(prediction, lsuffix='_train', rsuffix='_predict').plot()) With that simple model I get a rms of 11.85 (not much worse compared to the 11.77 from LSTM).

Mukesh Kumar
Mukesh Kumar 28 Feb, 2019

Hi, I think you should only use train data to fit MinMaxScaler and then transform train and valid data separately. Thanks

Varun Singh
Varun Singh 02 Mar, 2019

Hey I tried working on this code and i am getting an error, "float() argument must be a string or a number, not 'Timestamp' " Can you please help me solve it

John
John 03 Mar, 2019

Hi Aishwarya, Thanks very much for taking the time to post this article! I have been looking for an intro into time series analysis of stock prices. Your information was a great start and I was able to duplicate your LSTM code output. I apologize for the naive question on Python coding but, regarding your LSTM method to predict the next day closing price based on prior 60 day close prices, how would you output the next day close price prediction? For example, if I print the last day LSTM results from your dataset, I see: Date Actual Close Predicted Close 2019-01-04 213.8 228.571320 If it were real time (I mean if the current day was actually Jan 4th), how would you output the predicted next day close for Jan 5th? Thanks again...nice article!

Alok
Alok 04 Mar, 2019

Hi, Aishwarya could you please tell me that for measurement of the performance of the algorithms what metric are u taking?

Joseph
Joseph 04 Mar, 2019

The error shows up while calculating rms #calculate rmse rms=np.sqrt(np.mean(np.power((np.array(valid['Close'])-preds),2))) rms --------------------------------------------------------------------------- ValueError Traceback (most recent call last) in 1 #calculate rmse ----> 2 rms=np.sqrt(np.mean(np.power((np.array(valid['Close'])-preds),2))) 3 rms ValueError: operands could not be broadcast together with shapes (0,) (248,) When edited according to AISHWARYA SINGH who says, Please use the following command before calculating rmse: valid['Predictions'] = 0 valid['Predictions'] = closing_price After editing the code like this, #calculate rmse valid['Predictions'] = 0 valid['Predictions'] = closing_price rms=np.sqrt(np.mean(np.power((np.array(valid['Close'])-preds),2))) rms --------------------------------------------------------------------------- NameError Traceback (most recent call last) in 1 #calculate rmse 2 valid['Predictions'] = 0 ----> 3 valid['Predictions'] = closing_price 4 rms=np.sqrt(np.mean(np.power((np.array(valid['Close'])-preds),2))) 5 rms NameError: name 'closing_price' is not defined Basically, there's error in both ways. Is it because of that I'm using Python 3? Help would be appreciated!

Marcelo
Marcelo 05 Mar, 2019

Hi Asihwarya, Thanks for sharing this very interesting tutorial. Can you please share the full working code? Regards, Marcelo

yuganshu sanjay tickoo
yuganshu sanjay tickoo 08 Mar, 2019

mam can you help me in how to find the accuracy of the model

Haoran
Haoran 12 Mar, 2019

Hi Aishwarya, could you please share the data with me, Thank you.

Logesh
Logesh 13 Mar, 2019

lets assume we have 1000 instances of data separated into training and validation. Can we predict and get the value of the 1005th instance that is not present in the data?

ml-learner1641
ml-learner1641 19 Mar, 2019

Is there any way to do this with tensor-flow or anything other than keras? If so, can you point me in the right direction? I'm running into problems downloading keras and I haven't found a solution yet. Thanks Aishwarya!

CarreyWong
CarreyWong 21 Mar, 2019

In section LSTM I get this error, could u please tell me why and how to fix this when you are free? thank u very much. TypeError Traceback (most recent call last) in () 55 X_test = np.reshape(X_test, (X_test.shape[0],X_test.shape[1],1)) 56 print(X_test) ---> 57 closing_price = model.predict(X_test) 58 closing_price = scaler.inverse_transform(closing_price) 59 TypeError: data type not understood

Justin
Justin 22 Mar, 2019

Hi, Thanks for the article. Regarding the following line: - #predicting 246 values, using past 60 from the train data inputs = new_data[len(new_data) - len(valid) - 60:].values May I know what value obtained for : - len(new_data) ? len(valid) ? inputs ?

Tobias Wang
Tobias Wang 24 Mar, 2019

Hi Aishwarya, I was suspicious of your program, since it worked it little too well, so I played around with the program, and it after I tried to make it predict some future stocks (by making the validation set go to the future, and filling the rows with zeroes ), and you can imagine my surprise when the prediction said the stocks would drop like a stone. I investigated further, and found that your data leaks in the LSTM implementation, and that's why it works so well.

Justin
Justin 26 Mar, 2019

Hi Aishwarya, #predicting 246 values, using past 60 from the train data inputs = new_data[len(new_data) - len(valid) - 60:].values Regarding the above line, may I know what are the values obtained for the following: len(new_data) ? len(valid) ? inputs ? Thanks.

Diego
Diego 27 Mar, 2019

Hi! Great post! If I want to predict 10 or 15 days instead of 1. Which part of the code I have to change? I can´t get it..thanks!!!!!!!!

Diego Fernandez
Diego Fernandez 27 Mar, 2019

Hi! Great post! I can´t figure how to predict more tan one day. Could you help me? which line do you have to change? greetings!!

varna
varna 28 Mar, 2019

I'm using jupyter notebook in anaconda3 navigator and I'm getting this error ModuleNotFoundError: No module named 'fastai'. can you help me to resolve this

Gaurav
Gaurav 29 Mar, 2019

Hi Aishwarya, I am working on product development for financial domain. If you could share code project with me and along with some material which can help me learning ML basics that would be great. My email id is gcchavda[a]gmail[dt]com.

K N MUZAMIL
K N MUZAMIL 30 Mar, 2019

i want data on which you performed above exercises as i am learning data science course i need to perform projects.over data.....

Rajesh
Rajesh 30 Mar, 2019

Hello, It was a great experience learning from your article, it would have been much better if you explained each line of the code in all the approaches. My doubt is why is the RMSE value varying on each execution on the same data if I execute immediately in a short period of time and secondly correct me if I am wrong the orange line in the predicted graph for each model has to coincide with the original blue line only then we could conclude it to be the right forecast? please reply ASAP

Prashant
Prashant 02 Apr, 2019

Hi Aishwarya Help me to print the predicted values along with plotting of values. and secondly how can we implement the same code for intraday values when the date-time values are like 4/2/2019 2:07:00 AM thirdly what changes are required to see next 5 days values and print them Thanks for the code, it has helped me to start with it.

Marcel Delhez
Marcel Delhez 02 Apr, 2019

Fully agree with Tobias his analysis. By the way, I like the article but not the way LSTM is implemented. Yes, it uses data it is not supposed to know if you want to compare different types of models. And for you information, but here goal is obviously just to compare models, machine learning cannot predict financial markets just based on historical prices.would be too easy. But, as I wrote, article present different models and that is valuable for many people as I see.

Metus
Metus 02 Apr, 2019

Hi Aishwarya, I want you to explain all your code about LSTM. but now i want to know why you use unit = 50 in this below line 'model.add(LSTM(units=50, return_sequences=True, input_shape=(x_train.shape[1],1)))' Thank you

J S Abhishek
J S Abhishek 05 Apr, 2019

Hi Aishwarya, Really an intersting article, Please check if the following is correct. To predict the future values of more than 5 years, I've used the following code to predict next 100 days: pred_price=[] num_pred=100 X_test = np.reshape(X_test, (X_test.shape[0],X_test.shape[1],1)) print(X_test.shape) print(X_test[0,:,0]) X_test = np.array(X_test) for i in range(num_pred): closing_price = model.predict(X_test) print(closing_price) X_test=np.delete(X_test[0,:,0],0) #print(X_test.shape) print(X_test) X_test=np.append(X_test,closing_price) #print(X_test.shape) print(X_test) X_test=np.reshape(X_test,(1,X_test.shape[0],1)) print(X_test.shape) closing_price = scaler.inverse_transform(closing_price) pred_price.append(closing_price) pred_price=np.array(pred_price) print(type(pred_price)) print(pred_price.shape) pred_price=np.reshape(pred_price,(num_pred,1)) print(pred_price.shape) print(pred_price)

J S Abhishek
J S Abhishek 05 Apr, 2019

Hi, Firstly, really an amazing article written by you on AnalyticsVidhya. Thanks a lot for the post. So, I tried to predict more than test_set. The code goes as follows: pred_price=[] num_pred=10 X_test = np.reshape(X_test, (X_test.shape[0],X_test.shape[1],1)) print(X_test.shape) print(X_test[0,:,0]) X_test = np.array(X_test) for i in range(num_pred): closing_price = model.predict(X_test) print(closing_price) X_test=np.delete(X_test[0,:,0],0) #print(X_test.shape) print(X_test) X_test=np.append(X_test,closing_price) #print(X_test.shape) print(X_test) X_test=np.reshape(X_test,(1,X_test.shape[0],1)) print(X_test.shape) closing_price = scaler.inverse_transform(closing_price) pred_price.append(closing_price) pred_price=np.array(pred_price) print(type(pred_price)) print(pred_price.shape) pred_price=np.reshape(pred_price,(num_pred,1)) print(pred_price.shape) print(pred_price) This has shown that the values are always increasing. I've attached the python workbook file and data csv file. Is there anything wrong with the code. Waiting for your response.

Rishab Kumar
Rishab Kumar 06 Apr, 2019

I am having an error: cannot import name 'blosc_version' Please help me.

Ping Xu
Ping Xu 07 Apr, 2019

Thank you for this great work. The prediction results were only 63 rows not like said 248 as the same scripts were use as you posted. Could you please help? Thanks so much. Also I am using Python not Anconda. I could not install pystan to verify the Prophet which needs FBProphet that needs pystan. Any comments? Do I have to use Anconda? When I installed Anconda which supposed have pandas, numpy but error messages were it could not find them, not could it find the access to install. Thanks again.

Arjun
Arjun 07 Apr, 2019

Hi, My name is Arjun, i like to do project using ML. I like to design a prediction model to predict price of spices.Can I use this same code to develop a model?

Impana S
Impana S 10 Apr, 2019

If I could get 'Prediction' value for single date how to write code. I tried for newdata.index but couldn't get date for prediction

Omar El Omeiri
Omar El Omeiri 12 Apr, 2019

I'm not sure why this happened. the second last line of code: closing_price = model.predict(X_test) apparently goes in an infinite loop, hours running but no output. I have a pretty new Macbook Pro, processing power is not an issue. is this normal?

Dev Thakkar
Dev Thakkar 14 Apr, 2019

Hi Aishwarya, I want to use close price and volume both for training the model. I also want to predict prices for next 3 months. Can you please help me with this as I am new to python.

James M. Yunker (please do not post, call me anonymous)
James M. Yunker (please do not post, call me anonymous) 15 Apr, 2019

Please send to me the working python code. Also, please send the workaround I do not have fastai installed.

Sanheen Sethi
Sanheen Sethi 16 Apr, 2019

I train the data , and see the predictions of test data , and it also worked on bitcoin data i used , but How can i predict by giving new values means like i want to predict for 17 apr 2019 , which line of statement can do this ?

Mazhar
Mazhar 01 May, 2019

In KNN it is giving me an error : invalid literal for float() can you please help me with that

Sudeep Dasgupta
Sudeep Dasgupta 03 May, 2019

How you are going to predict future stock price, here you are using the test_X and test_y from the original datasets. that is the reason it is giving accurate prediction using LSTM when you are going to predict future stock price and need to create test_X and test_y and feed the value to predict it wont give good result.

Peter
Peter 11 May, 2019

Hello Aishwarya, Thanks for sharing! If I may ask. 1) Why do you make the cut-off at 987 for training? Why not 988 or any other number? Any rule of thumb to go by here? 2) Why look at 60 days to predict 61st? Would looking at more days back yield a "better" prediction? Any rule of thumb for this? 3) Why set units to 50? As above, is there a rule of thumb? 4) Could you please show me a quick example with what you mean by "You need to create a validation set with only 10 rows as input to the LSTM model." when trying to predict more than 1 day in advance? I cant really see where to make the change(s) to make that work. Many thanks! Peter

Imran
Imran 12 May, 2019

Hi Aish, Thanks for this great article. I tried the but Im getting errors in fastai and other snippets. Can you please send me the code?

Ali Raza
Ali Raza 12 May, 2019

Aishwarya I appreciate your work, I am engineering student and working on Machine Learning, I was trying to implement that code to my own but I found some error, from the LinearRegression (fastai library) It does not work, when I install the library fastai, it shows the error [no module torch] and I can find torch on internet I found pytorch which is not working for that. Please help me...

rekha
rekha 17 May, 2019

hI, Which are the other sequence prediction time series problems where LSTM can be applied

Frank
Frank 21 May, 2019

Hi, The average % error from LSTM model results is around 4-5% per day. If I use previous day close as a prediction for today's close, I got a better prediction with average % error of 2%. The graph will look even prettier than LSTM if we use previous close as a prediction. But we all know using previous day close as a prediction won't make you money.

Pau
Pau 27 May, 2019

Hi, I tried out this algorithm with real data from Binance. It works well, but I'm afraid it does not help predict anything. let's see whether I am missing anything (and I hope so!). I don't want to have any test set because I want to guess the results in the real world, so I trained everything. When predicting, in this case, we have 2 axes x: dates, y: price. We should be able to give the trained model an x coordinate (time units) and then it should return the y (prices) it estimates will be the real one. But after looking into it, I realised prices are being used when predicting instead of dates. Why? In the real world, I don't have these prices! I have only time units! I would like to use these time units in order to predict which prices will be in each one of them, and not using prices I won't have. Am I missing anything? And please, that's what I wanna hear!

Tyler
Tyler 28 May, 2019

Hello, I'm having trouble getting past the fastai section. I can't download the package. Each time I try , I get a memory error (i don't think my server can handle the entire package download...it's almost a GB). I was trying to recreate it like you mentioned using simple loops, and I was able to get something. Here is the code: for i in range(0,len(new_data)): new_data['Year'][i] = new_data['Date'][i].year new_data['Month'][i] = new_data['Date'][i].month new_data['Week'][i] = new_data['Date'][i].strftime("%U") new_data['Day'][i] = new_data['Date'][i].day new_data['Dayofweek'][i] = new_data['Date'][i].strftime("%w") new_data['Dayofyear'][i] = new_data['Date'][i].strftime("%j") if new_data['Date'][i].day == 1: new_data['Is_month_start'][i] = 1 tomorrow = new_data['Date'][i]+timedelta(days=1) if tomorrow.day == 1: new_data['Is_month_end'][i] = 1 if new_data['Date'][i].strftime("%d %m") in q_end: new_data['Is_quarter_end'][i] = 1 if new_data['Date'][i].strftime("%d %m") in q_start: new_data['Is_quarter_start'][i] = 1 if new_data['Date'][i].strftime("%d %m") == '01 01': new_data['Is_year_start'][i] = 1 if new_data['Date'][i].strftime("%d %m") == '31 12': new_data['Is_year_end'][i] = 1 But when I get to the next section I'm having trouble with the LRM. Can you confirm that this adds the proper columns and what not? Many thanks, and thanks for a great guide.

Tyler Tresslar
Tyler Tresslar 28 May, 2019

Hey, Thanks, this is a really great article. I'm having some issues in the linear regression part. I was not able to download the fastai package (memory error..) so I tried to go at it with simple loops instead. Here is my code: for i in range(0,len(new_data)): new_data['Year'][i] = new_data['Date'][i].year new_data['Month'][i] = new_data['Date'][i].month new_data['Week'][i] = new_data['Date'][i].strftime("%U") new_data['Day'][i] = new_data['Date'][i].day new_data['Dayofweek'][i] = new_data['Date'][i].strftime("%w") new_data['Dayofyear'][i] = new_data['Date'][i].strftime("%j") if new_data['Date'][i].day == 1: new_data['Is_month_start'][i] = 1 tomorrow = new_data['Date'][i]+timedelta(days=1) if tomorrow.day == 1: new_data['Is_month_end'][i] = 1 if new_data['Date'][i].strftime("%d %m") in q_end: new_data['Is_quarter_end'][i] = 1 if new_data['Date'][i].strftime("%d %m") in q_start: new_data['Is_quarter_start'][i] = 1 if new_data['Date'][i].strftime("%d %m") == '01 01': new_data['Is_year_start'][i] = 1 if new_data['Date'][i].strftime("%d %m") == '31 12': new_data['Is_year_end'][i] = 1 new_data.head(35) I'm not sure if I made an error here or not, but everything seems to come out ok. I don't know exactly what the output should look like though. Maybe you could include a new_data.head() in your code, just to have an idea of what the data looks like? Anyway, when I try to run the regression, I get this error: TypeError: float() argument must be a string or a number, not 'Timestamp' Thanks again!

Kiran
Kiran 29 May, 2019

Great article, couple of questions (I'm a beginner with DL, dabbled with SkLearn before) .. 1) How do i reproduce the LSTM model on my local Windows machine (or in a Google cloud account for AI, if that's easier)? Do i install Conda on Windows or some other stack? [The Github/fastai clone is supported only on Linux machines] 2) Could you post the daily prediction and actual prices through the validation period? I'm trying to assess the Win-rate (i.e. accuracy of predicted direction) and the Net Profits and Net Losses from a daily investing strategy that invested Long/Short daily based on the prediction. It's hard to tell from the chart - typically, unless there's 60% win-rate and Profit Factor (i.e. Net Profit/Net Loss) of 2x, it's not tradable.

Jonathan Lim
Jonathan Lim 30 May, 2019

Hi Aishwarya, My name is Jonathan Lim, and I have some experience in data science and ML. First of all, thank you so much for this post! I have been trying to figure how to predict the stock market for a while now, and your post ties up the knot between ML and stock market prediction really well! Thus, I would love to learn more about predicting the stock market using ML from you and to hear more about your experiences working in the data science and ML field! :) I have attached my LinkedIn profile along with this comment, which I hope to connect with you. Thank you, and I hope to hear from you soon! Sincerely, Jonathan Lim

Faheem Parach
Faheem Parach 14 Jun, 2019

Plz would anyone send me complete code without errors. ? i am facing a lot of issues.

sachin
sachin 17 Jun, 2019

Thanks Aishwarya, i am getting below error : from fastai.structured import add_datepart --------------------------------------------------------------------------- ModuleNotFoundError Traceback (most recent call last) in ----> 1 from fastai.structured import add_datepart ModuleNotFoundError: No module named 'fastai' i tried pip install fastai still no luck can you please help me with this

Raj
Raj 17 Jun, 2019

Hi, While predicting you use last 60 actual values from train and predict the first value of Pred[0]. Then you use the last 59 actual of train and 1 actual value from valid and get Pred[1] and so on...Thus not using the predicted values to further predict values but rather using all actual values and making separate 1 day predictions. Is my understanding right?

Jay
Jay 17 Jun, 2019

Hi Aishwarya Singh Its really nice article, can you please share me latest dataset Thanks

Tamilselvi
Tamilselvi 19 Jun, 2019

Hi aish, Can u share ideas little deeper on how to predict future sales for two more days. it wil be helpful

Sunil Kelker
Sunil Kelker 19 Jun, 2019

How does one add a feature to be evaluated for predictions. For example Volume, to the LSTM. All help appreciated.

O P MISHRA
O P MISHRA 19 Jan, 2022

Hi Aishwarya, I am writing you on a thread that has no comments for last 20 months. I read the article and found it interesting. I have a few questions as well as a strategy that I want to automate. Seeing your research and interest in the topic, maybe we can collaborate. Plz revert if you would want to discuss it further.

Rajkumar Telukuntla
Rajkumar Telukuntla 19 Jan, 2022

Hi! Thanks a lot for this great article. I'm getting the below error KeyError Traceback (most recent call last) ~\AppData\Local\Temp/ipykernel_23052/3766128780.py in 43 44 #predicting 246 values, using past 60 from the train data ---> 45 inputs = new_data[len(new_data) - len(valid) - 60:].values 46 inputs = inputs.reshape(-1,1) 47 inputs = scaler.transform(inputs) ~\anaconda3\lib\site-packages\pandas\core\frame.py in __getitem__(self, key) 3428 3429 # Do we have a slicer (on rows)? -> 3430 indexer = convert_to_index_sliceable(self, key) 3431 if indexer is not None: 3432 if isinstance(indexer, np.ndarray):

Nahush
Nahush 09 Feb, 2022

Can someone suggest a code to predict values for the next 30 days, whose values are not present in the dataset?

Dasheek Naidu
Dasheek Naidu 20 Feb, 2022

Hi Aishwarya, please may you share the full working code with me. Really appreciate it!

TS
TS 21 Mar, 2022

Hi, please help - how to provide input data (invest 10000) for 10 days using the LSTM model and get the predicted value. Can't find any place to provide those inputs.

Jain Patel
Jain Patel 27 Oct, 2022

Hey my accuracy is in negative for all model. Can you please guide me?