Comments on: Challenge discussion

By: Ku-Chun

Ku-Chun — Mon, 04 Jun 2012 12:14:09 +0000

It is really great that Yahoo is willing to release these datasets. Lacking real world data is a big concern for contextual bandit researchers(at least for me it is).

By: Jeremie

Jeremie — Mon, 04 Jun 2012 11:46:01 +0000

It’ll be done after the workshop on the Yahoo! webscope program

By: Yildiz

Yildiz — Mon, 04 Jun 2012 11:10:55 +0000

Yeah, thank you very much for the great competition!!!

Will you later publish the initial dataset, or the dataset from phase 2, so we can continue offline? Would be great

thanks again

By: Ku-Chun

Ku-Chun — Sun, 03 Jun 2012 11:50:07 +0000

Thank you for hosting such an enjoyable competition, had a great time!

By: Jeremie

Jeremie — Fri, 01 Jun 2012 16:55:38 +0000

Precise time limit is Samoa Standard Time (that means as long as it is June 2th somewhere on earth you can submit).

After the deadline you cannot submit, but you can indicate me your favorite old submission. This submission will replace your “best” one. About “paying attention” this is about not exclude a submission because of a trivial problem on Phase 2 datas.

By: exploreit

exploreit — Fri, 01 Jun 2012 16:46:15 +0000

Hi Jeremie,

Would we get a chance after June 2 to submit code for Phase 2 evaluation (taking care of feature removal, tuning constants etc) ? or would you just use our best submission file for phase 2 data? I did not understand when you said “I’ll pay attention to get them working on phase 2 data”.

Also, what time exactly will the last submission be accepted tomorrow?

By: Jeremie

Jeremie — Tue, 29 May 2012 12:13:42 +0000

Yes it is
The second round is to avoid all possible overfitting.

By: jamh

jamh — Tue, 29 May 2012 12:08:18 +0000

Hi Dr. exploreit.

As I understand, for the first part of the challenge, there is no final round. The winner will be the one with higher score passed the deadline.

For the second round I am not clear now how it will be. As I understand there will be a final round with more data as Jeremie explains.

By: Jeremie

Jeremie — Mon, 28 May 2012 21:22:59 +0000

The final dataset is 4 times bigger so I expect to have lower variance. Anyway if the scores appears to be close, I’ll check with some t-tests

By: exploreit

exploreit — Mon, 28 May 2012 20:16:40 +0000

I am noticing that random seed variations between runs can cause a swing of upto +/-10 points. since there nearly 10 people at the top within a difference of 20, this could be a significant effect. Do you have any suggestions for fixing this issue? Will the best submissions of each participant be run a number of times and the best/average score taken?