Very true, but these are rather trivial concerns compared to A/B testing. With A/B testing you still need a large amount of samples, but you also need to specify the (usually arbitrary) parameters, and decide when to stop the test on top of that.
The challenging aspects seem more of a limitation, but you still would be testing things all the time, ostensibly, rather than test once in the beginning and then stay there. I'd imagine some of the algorithms have a window of data to consider, rather than everything since the beginning.
You could also discard old data to make sure past actions don't impact your performance now.
Overall, the technique isn't a miracle cure, of course, but it's leaps and bounds better than the split tests we have now.
The challenging aspects seem more of a limitation, but you still would be testing things all the time, ostensibly, rather than test once in the beginning and then stay there. I'd imagine some of the algorithms have a window of data to consider, rather than everything since the beginning.
You could also discard old data to make sure past actions don't impact your performance now.
Overall, the technique isn't a miracle cure, of course, but it's leaps and bounds better than the split tests we have now.