Date of Original Version
Proceedings of the Conference on Computational Natural Language Learning (CoNLL)
Copyright 2010 ACL
Abstract or Description
Recent speed-ups for training large-scale models like those found in statistical NLP exploit distributed computing (either on multicore or “cloud” architectures) and rapidly converging online learning algorithms. Here we aim to combine the two. We focus on distributed, “mini-batch” learners that make frequent updates asynchronously (Nedic et al., 2001; Langford et al., 2009). We generalize existing asynchronous algorithms and experiment extensively with structured prediction problems from NLP, including discriminative, unsupervised, and non-convex learning scenarios. Our results show asynchronous learning can provide substantial speedups compared to distributed and singleprocessor mini-batch algorithms with no signs of error arising from the approximate nature of the technique.
Creative Commons License
This work is licensed under a Creative Commons Attribution-Noncommercial-Share Alike 3.0 License.
Proceedings of the Conference on Computational Natural Language Learning (CoNLL), 213-222.