-
Notifications
You must be signed in to change notification settings - Fork 22
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
potential bug in prediction of MIXL models #14
Comments
Hi @alvarogutyerrez , I am glad you find
|
Dear @arteagac, thank you for your quick answer. You are actually right, I forgot to compute the product of choice probabilities that belong to each individual when reconstructing the simulated LL. Unfortunately, once I did it, I still could not retrieve the sample (simulated) LL value produced by Additionally, all of this is because I want to analyze the performance of my model on out-of-sample data. This is why I am doing all this "replication" of the simulated LL value, but it would be much easier if I could simply evaluate a different sample using an already-trained model, but I haven't been able to do so. Actually, even when setting the maximum iterations number to 0, Is it possible that Thank you in advance! Reconstructing LL using product of LL of each individual
Even
|
Hi @alvarogutyerrez, I will take a closer look at the computations of the predict function. Unfortunately, the current version of xlogit does not have a mechanism to run a single step of the log-likelihood function without estimating the model. However, if you need a temporary work-around, you can use the code below, which is simply a function that takes the input data, formats it, and executes a single run of the LL function. You can use this function after estimating the model, and pass the # First estimate the model as usual using MixedLogit.fit
# Then use the function below to run a single execution of the log_likelihood function
def log_lik(model, X, y, varnames, alts, ids, randvars=None, isvars=None, weights=None, avail=None, panels=None, n_draws=1000):
# Pre-process and format inputs
X, y, varnames, alts, isvars, ids, weights, panels, avail, scale_factor \
= model._as_array(X, y, varnames, alts, isvars, ids, weights, panels, avail, None)
model._validate_inputs(X, y, alts, varnames, isvars, ids, weights)
betas, X, y, panels, draws, weights, avail, Xnames, scale = \
model._setup_input_data(X, y, varnames, alts, ids, model.randvars, isvars=isvars, weights=weights, avail=avail,
panels=panels, init_coeff=None, random_state=None, n_draws=n_draws,
halton=True, verbose=1, predict_mode=False, halton_opts=None, scale_factor=None)
from xlogit._choice_model import diff_nonchosen_chosen
Xd, scale_d, avail = diff_nonchosen_chosen(X, y, scale, avail) # Setup Xd as Xij - Xi*
fargs = (Xd, panels, draws, weights, avail, scale_d, None)
# Compute log-likelihood
return -model._loglik_gradient(model.coeff_, *fargs, return_gradient=False)
# LL on original model
ll = log_lik(model,
X=df[varnames],
y=df['CHOICE'],
varnames=varnames,
alts=df['alt'],
ids=df['custom_id'],
avail=df['AV'],
panels=df["ID"],
n_draws=100)
print(f"LL orig. model= {ll}")
# LL on new data
new_data = df[df['ID'] < 20]
ll_new = log_lik(model,
X=new_data[varnames],
y=new_data['CHOICE'],
varnames=varnames,
alts=new_data['alt'],
ids=new_data['custom_id'],
avail=new_data['AV'],
panels=new_data["ID"],
n_draws=100)
print(f"LL new data= {ll_new}")
|
thank you so much for the |
Dear Cristian Arteaga,
First of all, thank you for developing such a great package. I have been using it extensively and I think it works just great!
However, I think I found a weird behavior (most likely a bug) when predicting probabilities from a MIXL model.
I was trying to re-create the Log-likelihood (LL) of the model from its predicted probabilities using equation (6) in your article:
Unfortunately, noticed that I was not able to replicate them, and the differences were rather large. (see the first chunk of code below). Additionally, weirdly enough, when using only 1 draw, I am able to replicate the LL value. (see the second chunk of code below)
I haven't had time to go through your source code, but I think there is something wrong with the
.predict()
method for the MIXL models, since, for MNL models, I checked that my method of retrieving the LL works just fine (not included here).First chunk
Second chunk
Thank you in advance!
Álvaro
The text was updated successfully, but these errors were encountered: