Mining Machine Learning Best Practices

Our project this summer focused on discovering machine learning best practices. There are plenty of novices who try to adapt machine learning models for their own work. Although there are some great libraries and resources for helping people get started, there’s still a lot of mystery surrounding the process of how to find, select, and improve a model. It can also be unclear what makes one model perform better than another, leaving beginner programmers at a loss of how to improve their code. For our project, we built and analyzed a dataset of computational notebooks from Kaggle, a data science competition website. We are still in the process of analysis to identify significant differences. Ultimately, finding differences will allow us to educate ML novices and create a platform which people can interact with to hone their own ML models.

Slide 1: Nadia: Hi everyone My name is Nadia and I am a rising senior at the University of Washington. Jacinta: Hello! My name is Jacinta, and I am a rising senior at the College of William and Mary. Our project this summer focused on discovering machine learning best practices. NEXT PAGE Slide 2 (Motivation/Introduction): There are plenty of novices who try to adapt machine learning models for their own work. Although there are some great libraries and resources for helping people get started, there’s still a lot of mystery surrounding the process of how to find, select, and improve a model. It can also be unclear what makes one model perform better than another, leaving beginner programmers at a loss of how to improve their code. NEXT PAGE Slide 3 (Dataset): To study machine learning best practices, we needed a large set of machine learning code to work with, ideally from many coders of different experience levels. We found exactly what we were looking for on Kaggle, a platform which hosts data science competitions. We chose to use notebooks from the Home Credit Default Risk competition because of the high monetary reward of $70,000, a motivating factor for people to submit proper polished models. In total, we downloaded 165 python notebooks from this competition. NEXT PAGE Slide 4 (Methods): Features Upon acquiring the dataset, we wanted to find a range of features which could be used to distinguish between notebooks and later reveal the best practices. We began by parsing the notebooks for libraries that users imported. Libraries are important to pay attention to because coders, especially ML novices, heavily rely on them to create their models. To extract these libraries we parsed through the notebooks and recorded words in conjunction with the term “import.” We extracted notebook comments in the same way, by scanning the notebooks and recording the line following each pound sign. Using the comments we had extracted, we calculated the number of comments, number of words, and number of characters in each notebook. One of the most important features we extracted were function calls. Using a tracing library we recorded all the calls that each python file was making as it ran. This left us with a long list of random functions ( like print, join, predict, and so on). We then converted this list to some more useful and detailed features, such as consecutive function pairs and binary vectors that represented the entire body of functions within our collection of notebooks. We applied density-based spatial clustering, K-means clustering, and hierarchical clustering to individual features as well as combinations of features. We also applied a regression model to several combinations of features. For each feature we analyzed using hierarchical clustering, we tried standardizing it, applying a principal component analysis, and a TSNE transformation to see if that could help distinguish clusters. The clearest clusters could be seen when we used a TSNE transformation on the libraries and functions as binary vectors prior to clustering. On the bottom left here, you can see the clusters for the libraries, and on the bottom right you can see the clusters for the functions. Although there are sort of clear clusters for the libraries, upon further exploration it did not seem that these correlate with scores. However, there was some correlation for the function clustering. In particular, group 2 for the functions clustering clearly all had high scores and little variation. Another approach that resulted in meaningful groups was K-Means Clustering of two consecutive functions. To extract this feature, we took all the functions used in a notebook, and created consecutive pairs. For example if the notebook contained functions a, b, and c, the pairs would be a-dash-b and b-dash-c. Then, instead of simply creating a binary vector to represent these pairs, we went one step further to create a Term Frequency Inverse Frequency vector or TF IDF in order to give more meaning to pairs that appeared less frequently within the entire set of documents. The K-Means clustering algorithm revealed 4 clear clusters as seen at the bottom left corner. Upon further inspection of the scores behind the clusters, one group was once again distinguishing itself from the rest. Group Zero had the highest average score of over 79%. And It also had the smallest standard deviation, more than 12 times smaller than any of the other groups. Our next step is to deduce what makes these specific notebooks different by performing a sort of “deep dive” into the clustering results and notebooks themselves. Upon filtering the data frame, we were happy to find that the clusters of interest contained the same notebooks. This was a very meaningful step in our research since we got further confirmation that these clusters had a meaningful separation from the rest. We are in the process of inspecting the function and function pair features to see how the clusters formed. At the same time, we are also manually inspecting the notebooks to see any noticeable differences. There are a lot of directions we could take this project in the future, and there is still more work that needs to be done to uncover meaningful differences between beginners’ code and experts’ code. First and foremost, it would be helpful to add more features, especially features that represent code structure and organization. Upon identifying differences, we would want to present these findings in a clear way to new coders. Eventually, a good ultimate goal is to have an easy, interactive platform, which can help people select and improve models depending on their data set.