The Seventh Annual Tom Tom Founders Festival is a week-long celebration of innovators, visionaries, and artists who are shaping small cities. The Festival occurs at dozens of venues throughout downtown Charlottesville.
While there is a wide literature on developing neural networks for natural language understanding, the networks all have the same general architecture, determined by basic facts about the nature of linguistic input. In this talk I name and explain the four components (embed, encode, attend, predict), give a brief history of approaches to each subproblem, and explain two sophisticated networks in terms of this framework -- one for text classification, and another for textual entailment. The talk assumes a general knowledge of neural networks and machine learning. The talk should be especially suitable for people who have been working on computer vision or other problems. Just as computer vision models are designed around the fact that images are two or three-dimensional arrays of continuous values, NLP models are designed around the fact that text is a linear sequence of discrete symbols that form a hierarchical structure: letters are grouped into words, which are grouped into larger syntactic units (phrases, clauses, etc), which are grouped into larger discursive structures (utterances, paragraphs, sections, etc). Because the input symbols are discrete (letters, words, etc), the first step is "embed": map the discrete symbols into continuous vector representations. Because the input is a sequence, the second step is "encode": update the vector representation for each symbol given the surrounding context. You can't understand a sentence by looking up each word in the dictionary --- context matters. Because the input is hierarchical, sentences mean more than the sum of their parts. This motivates step three, attend: learn a further mapping from a variable-length matrix to a fixed-width vector, which we can then use to predict some specific information about the meaning of the text.
Matthew Honnibal is the creator of the spaCy NLP library. He started working on NLP research in 2005, and stayed in academia until 2014, before moving from Sydney to Berlin and co-founding Explosion.
S&P Global Inc. (prior to April 2016 McGraw Hill Financial, Inc., and prior to 2013 McGraw Hill Companies) is an American publicly traded corporation headquartered in New York City. Its primary areas of business are financial information and analytics. It is the parent company of... Read More →