Yoshua Bengio is one of the foremost thinkers in a field within artificial intelligence known as artifical neural networks and deep learning. Although significant progress has been made in recent years due to (among other factors) the combination of the proliferation of data, the decreasing cost of compute, and the tremendous amount of money and talent now devoted to artificial intelligence, Bengio chose this as a field of study during the 1980s, in the throes of what some referred to as the AI winter, seeing through a period when money and enthusiasm for artificial intelligence had dried up.
Bengio is the co-author (with Ian Goodfellow and Aaron Courville) of Deep Learning, a book that Elon Musk referred to as "the definitive textbook on deep learning." On top of his growing influence in this field, he has also been enormously influential in shaping Montreal to become a hotbed for artificial intelligence. Bengio co-founded Element AI in 2016, which has a stated mission to "turn the world's leading AI research into transformative business applications." Element AI aims to foster partnership between the private sector and academia to help push the expansion of AI.
Bengio believes Montreal has emerged as a powerhouse due to the combination of great universities, great companies (including a number of Silicon Valley companies who have established offices in Montreal), and Canada's ethos of cooperation among elite minds. We cover all of the above and more herein.
(To listen to an unabridged audio version of this interview, please visit this link. This is the 16th interview in my artificial intelligence series. Please visit these links to interviews with Mike Rhodin of IBM Watson, Sebastian Thrun of Udacity, Scott Phoenix of Vicarious, Antoine Blondeau of Sentient Technologies, Greg Brockman of OpenAI, Oren Etzioni of the Allen Institute for Artificial Intelligence, Neil Jacobstein of Singularity University, Geoff Hinton of Google, and Nick Bostrom of Oxford University, Jeff Dean of Google, Andrew Moore of Carnegie Mellon, and Andrew Ng of Stanford. To read future posts from either series, please click the link above to follow me on Twitter @PeterAHigh.)
Peter High: Where does the field of deep neural networks currently stand in your estimation?
Yoshua Bengio: We have made amazing progress, but we are far from human level intelligence with computers. Most of the progress has been with supervised learning, which means machines are taught by essentially imitating humans. With supervised learning, humans provide the high-level concepts that the computer learns, which can be tedious and limits the ability of computers to discover things by themselves. Unsupervised learning, or what we call reinforcement learning, is when the learner is not merely passively observing the world, or how humans do things, but interacts with the environment and gets feedback. Humans are good at this. Combining unsupervised deep learning and reinforcement learning is one of the things that I am working on.
High: What steps are needed to reach the more fully realized version of unsupervised learning?
Bengio: First, we have to understand what we have in front of us. That is how science works. It is not just about building new things. It is about understanding the algorithms and the phenomena that we are studying. If we look at the best current deep learning systems, the ones that are trained on millions, or even billions, of examples, the kinds of errors they make tell us a lot about their superficial understanding of the world, or the aspect of the world that they are trained on. Our findings and progress should not be seen as discouraging though because an animal’s understanding of the world is also superficial. As humans, we have a deeper understanding that allows us to survive better than animals and to trick animals into doing things, the same way we are now able to trick neural nets into producing the wrong answers.
How do we move forward? For more than a decade, my research has focused on the notion of learning better representations, which is the heart of deep learning, in particular, representations that have a property called disentangled. Disentangled separates the different concepts and different explanations - we call them factors - that explain the data, that explain what the agent sees around it, and that explain how the agent patrols the world. Disentangled captures some of the causality that explains what we are seeing and what the computer is seeing.
I do not yet know the exact steps that will get us where we want to go. I have a program, but there are a lot of unknowns. When researching, it is important to explore in many directions. That is the difference between basic research, which is long-term and exploratory, and applied research, which is when we take what we have and fine-tune it, so we can build products and solve concrete problems. Applied research is important because AI is greatly impacting the world, both commercially and through things like health care, education, and agriculture.
High: You recently co-founded a company called Element AI. What was the genesis of the enterprise and what is its mission?
Bengio: The genesis was a dream of building an international hub and a new ecosystem for AI in Montreal. To create a new ecosystem that includes both research and innovation elements, we needed great companies to be involved. One of the first companies is Element AI. It occupies an important niche in the ecosystem. There are large multinational companies that, unlike companies such as Google, Facebook, Amazon, Microsoft, IBM, and so on, do not have a team of scientists working on AI. They are rightfully concerned that unless they are helping to chart the course of AI within their organization, their business could be transformed in a direction that they do not want. Element AI helps these companies by developing spinoffs and business deals with them and by connecting them to local startups and companies.
Another important aspect of Element AI is that it is developing a network of internal and external researchers who conduct AI, machine learning, and deep learning research. This network of researchers within universities and Element AI is free to explore any new idea. They also collaborate with the applied researchers at Element AI so that they can be at the forefront of what is going on in the field for their applications. For instance, one concept that Element AI is working on is what we call transfer learning, which is the ability for a machine to improve learning on a new problem by taking advantage of what has been learned before on other datasets or other problems. This means we might not need as large of a dataset to get good results on new problems.
High: Are there certain companies or industries that are particularly well suited to be clients of Element AI?
Bengio: The response from industry, across all sectors, has been amazing. There is more demand than our current team can handle. A lot of what we are doing right now is recruiting new people. Currently, Element AI is not focused on one particular sector, but is exploring all of the possibilities and starting projects in many areas.
High: When an enterprise that is at the beginning of their artificial intelligence journey comes to Element AI, what general things do you advise them to do to begin to ramp up on these topics? What building blocks do you suggest?
Bengio: The first thing is to come up with a strategy that brings together the expertise from various aspects of the company. This includes the people who have the vision for the different products and where they could go in the future and the people who know what the market in this particular niche might be pulling and asking for. The data side is also clearly important. What data have been accumulated? What data could be collected? Finally, we have to examine whether it makes sense, from a deep learning point of view, to tackle these problems. Can we, in a reasonable amount of time, conceive of solutions that would potentially address these hard problems using foreseen data? There has to be a meeting of the minds with people from the organization who have different strengths and expertise with Element AI team members where we come up with a number of recommendations. For large companies, there could be a portfolio of potential projects that has to be prioritized and evaluated before an effort gets going.
High: In addition to being an entrepreneur, you are a professor at the University of Montreal. This seems to mirror the two halves you described before. There is the longer-term, less immediate research you are undertaking, versus the shorter-term commercial implications of what you are doing with Element AI. How do you divide your time between the two efforts?
Bengio: My main job is academic. I am full time at the university. I advise Element AI and other companies. The work that I do with Element AI and companies like Microsoft, for example, is primarily about the long-term aspect. That is my strength and what I can offer. I am not into the nitty-gritty of the commercial things in a company.
I am also developing the Montreal Institute of Learning Algorithms [MILA]. It is like a big startup and has become the heart of the AI ecosystem in Montreal. MILA brings together the machine learning researchers of the University of Montreal and McGill University. We already have about 200 people involved. We expect to double the number of professors and students over the next two to three years.
High: In many ways, much of what you have discussed, with regards to your work at the University of Montreal, with MILA, and with Element AI is about building an ecosystem. Is that an accurate perception?
Bengio: The community aspect is essential. A lot of success is due to the particular culture that has been developed in my lab at MILA. Our culture puts people center stage. Researchers with our expertise are in such high demand that we have to treat them like gold. We give them the freedom to find their place in the community so that they can be as motivated as possible to contribute and create something with their talent. Also, the Quebec culture is less individualistic and competitive than the culture of much of North America. We are more likely to collaborate and work together to build something.
High: Canada is doing interesting and progressive things to attract AI talent. You referred to your vision of making Montreal an artificial intelligence hub. Why has Montreal, in particular, been successful in this area? Is there something about the relationship between academics, the city or province governments, and the business community that has fostered the partnerships that are necessary to bring about an AI revolution?
Bengio: You listed some important factors, but I would say the essential factor is a simple one: critical mass. My group was one of the first to conduct deep learning research. It started with the University of Montreal trusting our vision and allowing us to recruit more professors, in a field that was not yet popular. Then there was a snowball effect. Better researchers brought in better students and better postdocs. This meant we were able to publish better papers, so we attracted more international visibility, which meant we could recruit even better people.
Similarly, when deep learning became more commercially interesting, more companies and investors were attracted to it. This accelerated. As large companies like Microsoft, Google, Facebook, and others came to Montreal, we became even more visible to the scientific and investor communities. All of this stimulated entrepreneurship. More of my students want to start companies, and they are doing it here. Investors used to ask our entrepreneurs to go to the Valley, but now they are happy with them staying in Canada.
As you mentioned, it is not just Montreal. Toronto is moving on a similarly fast track. We have created a collaboration between the Vector Institute in Toronto, the MILA in Montreal, and the Alberta Machine Intelligence Institute in Edmonton. We are forming the AI Consortium for Canada. Our goal is to put Canada on the map scientifically and as the country for AI. We are a small player compared to the U.S. and China, but our critical mass is making a big difference. Silicon Valley is not a big place in terms of the number of people, but it is a center for innovation because of the critical mass effect.
High: I want to turn to AI safety. Elon Musk has tweeted his concerns that AI may lead to World War III. He is not alone. A number of leading entrepreneurs and academics have highlighted the potential threats of artificial intelligence if it is not harnessed appropriately. Where do you stand on this?
Bengio: I am not worried about Terminator scenarios of AI taking over humanity. It is good that some people are investigating those issues, but they are far away. I am concerned with potential short and medium-term issues with the application of AI. AI can be misused. As you mentioned, one concern is use in the military sector. Within the scientific community, there is a lot of support for an international treaty that bans lethal autonomous weapons, or LAWs. Serious discussions are taking place, including in the U.N., about this issue. There are some countries, like the U.S., that are trying to block bans. That is a big mistake. Allowing LAWs to be easily commercialized is bad for everyone's security because it will make it easier for small players to use these tools. It does make sense to conduct military research on defensive techniques against lethal autonomous weapons. However, we need an international treaty that stigmatizes the use of these weapons in the same way that, over the last two decades, we have succeeded in stigmatizing the development of nuclear weapons, chemical weapons, biological weapons, and landmines. Although we may not be able to perfectly and tidily prevent those things, if we can reduce them by making it clearly immoral for everyone, including countries and large companies, to develop them, we can greatly reduce the security risk for everyone on earth.
High: What near term areas of progress with deep learning will impact our everyday lives?
Bengio: The progress in deep learning for natural language is one area. As computers become better at understanding what is written and said, and can respond and generate natural language, the way we interact with computers will be transformed. That will likely have a big impact on the job market, both positively and negatively, and is therefore something that many companies are investing heavily in. This, in part, is why rapid progress is being made. Scientific progress is proportional to the effort that is put in. One reason deep learning has been so successful in the last three years is because there are many more people doing research and then developing things with it. That will continue.
Peter High is President of Metis Strategy, a businessand IT advisory firm. His latest book is Implementing World Class IT Strategy. He is also the author of World Class IT: Why Businesses Succeed When IT Triumphs. Peter moderates the Forum on World Class IT podcast series. He speaks at conferences around the world. Follow him on Twitter @PeterAHigh.
