Learning Deep Architectures for AI Yoshua Bengio Dept IRO
➤ Gửi thông báo lỗi ⚠️ Báo cáo tài liệu vi phạmNội dung chi tiết: Learning Deep Architectures for AI Yoshua Bengio Dept IRO
Learning Deep Architectures for AI Yoshua Bengio Dept IRO
Learning Deep Architectures for AlYoshua BengloDept. IRO. Université de Montreal c.p. 6128. Montreal. Qc, H3C 3J7, Canada Yoshua. Bengio@umontreai. ca Learning Deep Architectures for AI Yoshua Bengio Dept IROa http://www.iro.umontreal.ca/~bengioyTo appear in Foundations and Trends m Machine LearningAbstractTheoretical results suggest drat in order to learn tire kind of complicated Junctions that can represent high-level abstractions (e.g. in vision, language, and other Al-level tasks). Oise may need dee Learning Deep Architectures for AI Yoshua Bengio Dept IROp architectures. Deep architectures are composed of multiple levels of non-linear operations, such as in neural nets with many hidden layers or in comLearning Deep Architectures for AI Yoshua Bengio Dept IRO
plicated propositional formulae re-using many sub-formulae. Searching the parameter space of deep architectures is a difficult task, but teaming algorLearning Deep Architectures for AlYoshua BengloDept. IRO. Université de Montreal c.p. 6128. Montreal. Qc, H3C 3J7, Canada Yoshua. Bengio@umontreai. ca Learning Deep Architectures for AI Yoshua Bengio Dept IROcertain areas This paper discusses the motivations and principles regarding learning algorithms for deep architectures, in particular those exploiting as building blocks unsupervised learning of single-layer models such as Restricted Boltzmann Machines, used to construct deeper models such as Deep B Learning Deep Architectures for AI Yoshua Bengio Dept IROelief Networks.1 IntroductionAllowing computers to model our wot Id well enough to exhibit what we call intelligence lias been the focus of more thanLearning Deep Architectures for AI Yoshua Bengio Dept IRO
half a century of research. To achieve this, it is clear that a large quantity of information about our world should somehow be stored, explicitly or Learning Deep Architectures for AlYoshua BengloDept. IRO. Université de Montreal c.p. 6128. Montreal. Qc, H3C 3J7, Canada Yoshua. Bengio@umontreai. ca Learning Deep Architectures for AI Yoshua Bengio Dept IROand generalize to new contexts, many researchers have turned to learning algorithms to capture a large fraction of that information. Much progress has been made to understand and improve learning algorithms, but the challenge of artificial intelligence (Al) remains. Do we have algorithms that can un Learning Deep Architectures for AI Yoshua Bengio Dept IROderstand scenes and describe them in natural language0 Not really, except in very limited settings Do we have algorithms that can infer enough semantiLearning Deep Architectures for AI Yoshua Bengio Dept IRO
c concepts to be able to interact with most humans using these concepts? No. If we consider image understanding, one of the best specified of the Al tLearning Deep Architectures for AlYoshua BengloDept. IRO. Université de Montreal c.p. 6128. Montreal. Qc, H3C 3J7, Canada Yoshua. Bengio@umontreai. ca Learning Deep Architectures for AI Yoshua Bengio Dept IROo interpret most images on the web. The situation is similar for other Al tasks.Consider for example the task of interpreting an input image such as the one in Figure 1. When humans try to solve a particular Al task (such as machine vision or natural language processing), they often exploit their in Learning Deep Architectures for AI Yoshua Bengio Dept IROtuition about how to decompose the problem into sub-problems and multiple levels of representation, e g., in object parts and constellation models (WeLearning Deep Architectures for AI Yoshua Bengio Dept IRO
ber. Welling. & Perona. 2000: Niebles & Fei-Fei. 2007; Sudderth. Torralba. Freeman. & Willsky. 2007) where models for pans can be re-used in differentLearning Deep Architectures for AlYoshua BengloDept. IRO. Université de Montreal c.p. 6128. Montreal. Qc, H3C 3J7, Canada Yoshua. Bengio@umontreai. ca Learning Deep Architectures for AI Yoshua Bengio Dept IROear or kernel classifier (Pinto. DiCarlo. & Cox. 200S: Mutch & Lowe. 20081, with intermediate modules mixing engineered transformations and learning, e.g. first extracting low-levelfeatures that are invariant to small geometric variations (such as edge detectors from Gabor filters), transforming the Learning Deep Architectures for AI Yoshua Bengio Dept IROm gradually (eg. to make them invariant to contrast changes and contrast inversion, sometimes by pooling and sub-sampling), and then detecting the mosLearning Deep Architectures for AI Yoshua Bengio Dept IRO
t frequent patterns. A plausible and common way to extract useful information from a natural image involves transforming tile law pixel representationLearning Deep Architectures for AlYoshua BengloDept. IRO. Université de Montreal c.p. 6128. Montreal. Qc, H3C 3J7, Canada Yoshua. Bengio@umontreai. ca Learning Deep Architectures for AI Yoshua Bengio Dept IROntification of abstract categories associated with sub-objects and objects which are parts of the image, and putting all these together to captuie enough understanding of the scene to answer questions about it.Here, we assume that the computational machinery necessary to express complex behaviors (w Learning Deep Architectures for AI Yoshua Bengio Dept IROhich one might label “intelligent”) requires highly vary ing mathematical functions, i.e. mathematical functions that are highly non-linear in terms oLearning Deep Architectures for AI Yoshua Bengio Dept IRO
f raw sensory inputs, and display a very large number of variations (ups and downs) across the domain of interest. We view the law input to the learniLearning Deep Architectures for AlYoshua BengloDept. IRO. Université de Montreal c.p. 6128. Montreal. Qc, H3C 3J7, Canada Yoshua. Bengio@umontreai. ca Learning Deep Architectures for AI Yoshua Bengio Dept IROe, using knowledge of the 3D geometry of solid objects and lighting, we can relate small variations in underlying physical and geometric factors (such as position, orientation, lighting of an object) with changes in pixel intensities for all the pixels in an image. We call these factors of variation Learning Deep Architectures for AI Yoshua Bengio Dept IRO because they are different aspects of the data that can vary separately and often independently. In this case, explicit knowledge of the physical facLearning Deep Architectures for AI Yoshua Bengio Dept IRO
tor-s involved allows one to get a picture of the mathematical form of these dependencies, and of the shape of the set of images (as points in a high-Learning Deep Architectures for AlYoshua BengloDept. IRO. Université de Montreal c.p. 6128. Montreal. Qc, H3C 3J7, Canada Yoshua. Bengio@umontreai. ca Learning Deep Architectures for AI Yoshua Bengio Dept IROin the data, and how they interact to generate the kind of data we observe, we would be able to say that the machine understands those aspects of the world covered by these factors of variation. Unfortunately, in general and for most factors of variation underlying natural images, we do not have an Learning Deep Architectures for AI Yoshua Bengio Dept IROanalytical understanding of these factors of variation. We do not have enough formalized pnor knowledge about the world to explain the observed varietLearning Deep Architectures for AI Yoshua Bengio Dept IRO
y of images, even for such an apparently simple abstraction as MAN. illustrated in Figure I. A high-level abstraction such as MAN has the property thaLearning Deep Architectures for AlYoshua BengloDept. IRO. Université de Montreal c.p. 6128. Montreal. Qc, H3C 3J7, Canada Yoshua. Bengio@umontreai. ca Learning Deep Architectures for AI Yoshua Bengio Dept IROance in the space of pixel intensities. The set of images for which that label could be appropriate forms a highly convoluted region in pixel space that IS not even necessarily a connected region. The MAN category can be seen as a high-level abstraction with respect to the space of images. What we c Learning Deep Architectures for AI Yoshua Bengio Dept IROall abstraction here can be a category (such as the MAN category I or a feature. a function of sensory data, which can be discrete te.g.. the input seLearning Deep Architectures for AI Yoshua Bengio Dept IRO
ntence IS at the past tensei or continuous (eg.. the input video shows an object moving at 2 meter/second I. Many lower-level and intermediate-level cLearning Deep Architectures for AlYoshua BengloDept. IRO. Université de Montreal c.p. 6128. Montreal. Qc, H3C 3J7, Canada Yoshua. Bengio@umontreai. ca Learning Deep Architectures for AI Yoshua Bengio Dept IROlar percepts, whereas higher level ones are what we call "more abstract" because their connection to actual percepts is more remote, and through other, mtemiediate-level abstractions.In addition to the difficulty of coming up with the appropriate intermediate absưactions. the number of visual and se Learning Deep Architectures for AI Yoshua Bengio Dept IROmantic categories (such as MAN) that we would like an ••intelligent” machine to capture is rather lar ge. The focus of deep architecture learning is tLearning Deep Architectures for AI Yoshua Bengio Dept IRO
o automatically discover such abstractions, from the lowest level features to the highest level concepts. Ideally, we would like learning algorithms tLearning Deep Architectures for AlYoshua BengloDept. IRO. Université de Montreal c.p. 6128. Montreal. Qc, H3C 3J7, Canada Yoshua. Bengio@umontreai. ca Learning Deep Architectures for AI Yoshua Bengio Dept IROide a huge set of relevant hand-labeled examples. If these algorithms could tap into the huge resource of text and images on the web. it would certainly help to transfer much of human knowledge into machine-interpretable form.1.1How do We Train Deep Architectures?Deep learning methods aim at learnin Learning Deep Architectures for AI Yoshua Bengio Dept IROg feature hierarchies with features from higher levels of the hierarchy formed by the composition of lower level features. Automatically learning featLearning Deep Architectures for AI Yoshua Bengio Dept IRO
ures at multiple levels of absnaction allows a system to learn complex functions mapping the input to the output directly from data.2very high level rLearning Deep Architectures for AlYoshua BengloDept. IRO. Université de Montreal c.p. 6128. Montreal. Qc, H3C 3J7, Canada Yoshua. Bengio@umontreai. ca Learning Deep Architectures for AI Yoshua Bengio Dept IROould like the raw input image to be transformed into gradually higher levels of representation, representing more and more abstract functions of the raw input, e.g.. edges, local shapes, object parts, etc In practice, we do not know in advance what the ‘Tight” representation should be tor all these Learning Deep Architectures for AI Yoshua Bengio Dept IROlevels of absưactions. although linguistic concepts might help guessing what the higher levels should implicitly represent3without depending completelLearning Deep Architectures for AI Yoshua Bengio Dept IRO
y on human-crafted features. This is especially important for higher-level abstractions. which humans often do not know how to specify explicitly in tGọi ngay
Chat zalo
Facebook