Week 1 of Udacity's Deep Learning Nanodegree Foundation Program

This past week I began the Deep Learning Nanodegree Foundation Program from Udacity. When I enrolled I was scheduled to begin the second offering of the program, which did not begin until March 29th. Although I was anxious to get started right away I reasoned that the upside was more time to prepare. However, a few days later I received an email stating that there were a few spots remaining in the first cohort. I responded expressing my interest and was accepted. The window of time to prepare went from two months to less than 24 hours. It was go time!

Reflections After The First Week

After completing the few several lessons I am rather appalled at the format, quality, and content of the instruction. An abbreviated list of the issues encountered includes:

  • A significant number of errors in the prose and mathematical content.
  • The material was not presented in the proper order such that new concept introduction was sequenced after deeper examinations. The result was a sense of confusion and suspicion that I had missed something only to discover that the concept was introduced and explained later. As a result, I had to regularly go back and start from the beginning after having grasped the basics. This cycle repeated itself several times.
  • The prerequisites listed in the program's FAQ were "…a basic working knowledge of Python programming. Outside of that Python expectation, it's a very beginner-friendly program." In reality, without a strong mathematical background, an individual will struggle to grasp concepts like backpropagation and gradient descent beyond a superficial level.
  • Minimal supporting courses and content were suggested for those needing to refresh or acquire prerequisite knowledge. Fortunately, other students shared their own lists of resources, including books, other online courses, and videos.
  • Live sessions by the "YouTube star" were rushed, bombastic, tangent laced, and lacking in expository content.
  • More video instruction hours were devoted to low-value live sessions than explaining important, foundational topics. As an example, there was a 5-minute video that barely skimmed the surface of gradient descent and a live session lasting over an hour.
  • The timing of course content release and project due dates was not clearly communicated.

Although it was not advertised as such, Udacity staff explained in the program's Slack channel that they took the approach of opening the program before it was finalized in order to improve and adapt the content based on feedback from real students. This technique is understandable given the high demand for courses on Deep Learning and the fast-moving nature of the field. However, I predict that the backlash by students would have been less severe and accommodative had they been told about the "beta" nature from the outset. All that being said, I am grateful that Udacity created the course, and despite the lack of transparent expectation setting, they have reacted in a very rapid and enthusiastic way to suggestions for improvement submitted by students. I commend them on their response, and by the time I publish this post, many of the early issues will have already been resolved.

The Path Ahead

Given how quickly improvements to the course are being implemented, the biggest hurdle remaining is that truly understanding current approaches to the learning aspect of Deep Learning requires a high degree of comfort with statistics, linear algebra, and the vector calculus sub-branch of multivariable calculus. Although I was fairly proficient in these subjects during high school and college, nearly two decades have elapsed since then during which my usage of this knowledge was scant and infrequent.

To progress towards mastery of this domain of knowledge I will need to build upon a strong foundation in mathematics. To master calculus, one should be proficient in algebra, geometry, and trigonometry as each new topic builds upon the previous. My grades in these subjects were commendable, but they were never my favorite classes. Reflecting now, this had more to do with the lack of connection between the knowledge and practical applications. Since I observe a clear link between my goal to master a new domain and the need to intuitively understand mathematics, it has made the hours spent reinforcing or retraining myself a surprisingly fun experience!

Although there are a few math courses on Udacity, I've found that Khan Academy does a great job of incorporating some of the concepts of deliberate practice into their learning missions. One can read more about deliberate practice and a number of fascinating insights into expert performance in Peak: Secrets from the New Science of Expertise by Anders Ericsson. The short version is that Ericcson's studies found that experts didn't just practice for the popularized 10,000 hours. Simply performing a skill a large number of times is not sufficient to master it. The key insight is that you must continually practice a skill at a level that increases in difficulty as your proficiency increases paired with immediate feedback after each performance. While not a perfect implementation, I've found Khan Academy's approach to video-based instruction, adaptive levels of practice tests, and immediate feedback to be ideal for identifying and practicing the gaps in my knowledge.

This experience has also been a great lesson in the value of an education in a broad set of subjects. I've often remarked in the years since entering the workforce about the number of hours wasted learning subjects that I've never used in my career. I'm now forced to revise that viewpoint as I could have never predicted that linear algebra and calculus would become essential in my quest to better understand machine learning and the sub-field of deep learning.

Takeaways for Future Students

To some extent, I am grateful that the course is difficult. If it were trivially easy then it is unlikely much would be gained by completing the coursework. However, I am frustrated that the course, as initially constructed, does such a poor job of explaining the concepts necessary to tackle the project assignment and failed to inform students in advance.

If anyone happens to stumble upon this before starting a future session of the Deep Learning Nanodegree Foundation Program here are some tips based on my experience after the first week. I'll continue to share insights as the weeks progress in case any of this guidance turns out to be ill-advised.

  • I am beginning to appreciate why several experienced practitioners suggest starting with machine learning first before diving into deep learning. The multi-layered neural networks of deep learning that necessitate complex calculus create a very steep learning curve for someone just getting into the field. If this describes you I would suggest reading the Quora post linked to above before signing up for the Deep Learning Foundation program. A great course to take first is Andrew Ng's Machine Learning on Coursera.
  • Unless you have a strong background in math or physics it will take significantly more than 3 hours for the first week. It is common for people, myself included, to spend 20 hours to build up enough knowledge to successfully complete the first project.
  • Read Khan Academy's prerequisites for AP Calculus AB before diving straight into calculus training. If you feel you need a refresher you can choose to use the links provided to specific algebra, geometry, and trigonometry lessons to strengthen your math skills. If you have the time and interest you can work through the entire instructional and practice test missions for algebra, trigonometry, calculus, and multivariable calculus.
  • Read Andrew Trask's book Grokking Deep Learning in parallel or even before you start the Udacity program. It has a slower ramp up and does a better job at explaining the fundamentals to beginners.

None of these tips will make the Deep Learning Foundation Program easy. It is going to be hard work, but if you are serious about acquiring foundational knowledge use this opportunity to build up rare and valuable skills that will serve you well in the future.