Data are becoming the new raw material of business
The Economist

Polling and big data in the age of Trump, Brexit, and the Colombian Referendum

Our founder, Michael Li, recently collaborated with his colleague Raymond Perkins, a researcher and PhD candidate at Princeton University, on this piece about big data and polling. You can find the original article at Data Driven Journalism.

globe-1015311_960_720The recent presidential inauguration and the notably momentous election that preceded it has brought about numerous discussions surrounding the accuracy of polling and big data. The US election results paired with those of Brexit, and the Colombian Referendum have left a number of people scratching their heads in confusion. Statisticians, however understand the multitude of sampling biases and statistical errors than can ensue when your data is involving human beings.

“Though big data has the potential to virtually eliminate statistical error, it unfortunately provides no protection against sampling bias and, as we’ve seen, may even compound the problem. This is not to say big data has no place in modern polling, in fact it may provide alternative means to predict election results. However, as we move forward we must consider the limitations of big data and our overconfidence in it as a polling panacea.”

At The Data Incubator, this central misconception about big data is one of the core lessons we try to impart on our students. Apply to be a Fellow today!

 

Editor’s Note: The Data Incubator is a data science education company.  We offer a free eight-week fellowship helping candidates with PhDs and masters degrees enter data science careers.  Companies can hire talented data scientists or enroll employees in our data science corporate training.

Tweet about this on TwitterShare on FacebookShare on LinkedIn

How Employers Judge Data Science Projects

mark-516277_960_720One of the more commonly used screening devices for data science is the portfolio project.  Applicants apply with a project that they have showcasing a piece of data science that they’ve accomplished.  At The Data Incubator, we run a free eight week fellowship helping train and transition people with masters and PhD degrees for careers in data science.  One of the key components of the program is completing a capstone data science project to present to our (hundreds of) hiring employers.  In fact, a major part of the fellowship application process is proposing that very capstone project, with many successful candidates having projects that are substantially far along if not nearly completed.  Based on conversations with partners, here’s our sense of priorities for what makes a good project, ranked roughly in order of importance: 

  1. Completion: While their potential is important, projects are assessed primarily based on the success of analysis performed rather than the promise of future work.  Working in any industry is about getting things done quickly, not perfectly, and projects with many gaps, “I wish I had time for”, or “ future steps” suggests the applicant may not be able to get things done at work.
  2. Practicality: High-impact problems of general interest are more interesting than theoretical discussions on academic research problems. If you solve the problem, will anyone care? Identifying interesting problems is half the challenge, especially for candidates leaving academia who must disprove an inherent “academic” bias.
  3. Creativity: Employers are looking for creative, original thinkers who can identify either (1) new datasets or (2) find novel questions to ask about a dataset. Employers do not want to see the tenth generic presentation on Citibike (or Chicago Crime, Yelp Restaurant Ratings data, NYC Restaurant Inspection DataNYC Taxi, BTS Flight Delay, Amazon Review, Zillow home price, or beating the stock market) data. Similarly, projects that explain a non-obvious thesis supported by concise plots are more compelling than ones that present obvious conclusions (e.g. “more riders use Citibike during the day than at night”). Employers are looking for data scientists who can find trends in the data that they don’t already know. Continue reading
Tweet about this on TwitterShare on FacebookShare on LinkedIn

Search results: Careers in high tech

Screen Shot 2017-01-26 at 11.06.49 AMI was recently interviewed for a piece for ScienceMag about careers in high tech. You can find the original post on ScienceMag.

With big data becoming increasingly popular and relevant,  data scientist jobs are opening up across every industry in virtually every corner of the globe. Unfortunately, the multitude of available positions isn’t making it any easier to actually land a job as a data scientist. Competition is abundant, interviews can be lengthy and arduous, and good ideas aren’t enough to get yourself hired. Michael Li  emphasizes that technical know-how is what hiring managers crave. “No one needs just an ‘ideas’ person. They need someone who can actually get the job done.”

 

This shouldn’t discourage anyone from pursuing a career in data science because it can be both rewarding and profitable. If you’re looking to brush up your skills and jump start your career, consider applying for our free data science fellowship with offerings now in San Francisco, New York, Washington DC, Boston, and Seattle. Learn more and apply on our website.

 

Tweet about this on TwitterShare on FacebookShare on LinkedIn

Tying Together Elegant Models: Alumni Spotlight on Brendan Keller

At The Data Incubator we run a free eight-week data science fellowship to help our Fellows land industry jobs. We love Fellows with diverse academic backgrounds that go beyond what companies traditionally think of when hiring data scientists. Brendan was a Fellow in our Fall 2015 cohort who landed a job with one of our hiring partners, Jolata.

Tell us about your background. How did it set you up to be a great data scientist?144670722818-brendan_keller

I did my PhD research in theoretical condensed matter physics at the University of California, Santa Barbara. The focus of my research was on studying the phase diagram of chains of non-abelian anyons. Because such chains are gapless in most regions of the phase diagram we had to model them using very large matrices in C++. To make this computation more tractable we used hash tables and sparse matrices.  Besides my background in numerics I also took the time to learn Python, Pandas, SQL and MapReduce in Cloudera a few months before starting the fellowship.

What do you think you got out of The Data Incubator?

The Data Incubator gave me a solid foundation in data parsing, large scale data analysis and machine learning. I went into the fellowship already knowing about various concepts like SVM, bag-of-words and cross-validation. But I learned how tie these together into a elegant models that are both modular and easy to modify or upgrade. I also learned how to use Map Reduce on a cluster where the behavior of your program can be quite different then on a single node.  Continue reading

Tweet about this on TwitterShare on FacebookShare on LinkedIn

Continuing Your Data Science Job Hunt Through the Holidays

xmas-wishThe holidays can seem like a tough time to job search, people are out of the office and holiday schedules are hectic.  But there are lots of things you can do to take advantage of this time, keep your search moving forward, and set yourself up for post holiday success. 

  1. Review your skill set

Read through every job description you can, even the ones for jobs that didn’t originally interest you. Where are your skills gaps? If you see fluency in C++ in one or two job descriptions, but not on most, you might be okay not knowing it well. But if you see fluency in C++ listed over and over, the next few weeks are a great time for you to work on learning it.

  1. Take on a new project

One of the best things you can do to really master those new skills (and demonstrate your knowledge) is to apply them. We’ve been publishing links to lots of publically available data sets on our blog. Take one and treat it as a case study, what problem might this company or organization have, and how can you use data science to solve it? You can add your work to your github, blog about it, or share it on your LinkedIn! These are publically available data sets, so definitely show off your work.

Continue reading

Tweet about this on TwitterShare on FacebookShare on LinkedIn


The Details of Data Science: Alumni Spotlight on Ellen Ambrose

At The Data Incubator we run a free eight-week data science fellowship to help our Fellows land industry jobs. We love Fellows with diverse academic backgrounds that go beyond what companies traditionally think of when hiring data scientists. Ellen was a Fellow in our Spring 2016 cohort who landed a job with one of our hiring partners, Protenus.

Tell us about your background. How did it set you up to be a great data scientist? 

The title data scientist implies both technical skills, as well as the ability to ask and answer questions scientifically. My academic decisions were driven by the desire to develop the second of these skills. I studied math in college because I was attracted to the logical way of thinking and proving concepts that I encountered in math class. I went on to do a PhD in neuroscience in part because I wanted to pursue a question to an extreme level of detail, leveraging the logic that I learned doing math. (I also wanted to understand how learning works at the level of neural ensembles, but that’s another story.)

As result of my academic trajectory I also learned to write analysis code. I enjoyed coding, and at first I considered it a perk of my particular field of neuroscience that a lot of coding (mostly in Matlab) was necessary for analyzing the large datasets I was collecting. However, I eventually came to appreciate coding in it’s own right and started taking steps to learn new languages and to improve my analyses by incorporating better tools. By this time I had realized that I would be happy doing coding full time, so The Data Incubator was a great segue way to new concepts and tools in the world of data science.
Continue reading

Tweet about this on TwitterShare on FacebookShare on LinkedIn

Crunching Yelp Data to a Job at Crunchbase: Alumni Spotlight on Newton Le

At The Data Incubator we run a free eight-week data science fellowship to help our Fellows land industry jobs. We love Fellows with diverse academic backgrounds that go beyond what companies traditionally think of when hiring data scientists. Newton was a Fellow in our Summer 2016 cohort who landed a job with one of our hiring partners, Crunchbase.

Tell us about your background. How did it set you up to be a great data scientist? 

I have an electrical engineering and computer science degree from UC Berkeley, which gave me a strong coding foundation. I am also almost done with a PhD in structural engineering at UC Davis, which gave me a lot of experience solving analytical problems computationally.

What do you think you got out of The Data Incubator?

Beyond the miniprojects and learning the data science tools available, The Data Incubator really made me see my potential. Being challenged to do something interesting with data, I came up with a fun idea that I had no idea how to execute, but I pushed myself to learn new tools and produce some useful within a few days. Seeing what others could do every week, I was inspired to add significant improvements to my capstone project and learned a lot each step of the way. Super smart instructors and talented peers really inspired me as well. Everyone had their strengths and I could learn something different from each person. The environment didn’t feel competitive at all and was actually very collaborative. I actually miss coming in every day to work with my pod. Go Team Lannister!  Continue reading

Tweet about this on TwitterShare on FacebookShare on LinkedIn

Data Sources for Cool Data Science Projects Part 6

startup-593324_960_720Links to Part 1Part 2Part 3Part 4, Part 5

At The Data Incubator, we run a free eight week data science fellowship to help our Fellows land industry jobs. Our hiring partners love considering Fellows who don’t mind getting their hands dirty with data.  That’s why our Fellows work on cool capstone projects that showcase those skills.  One of the biggest obstacles to successful projects has been getting access to interesting data.  Here are a few cool public data sources you can use for your next project:

Continue reading

Tweet about this on TwitterShare on FacebookShare on LinkedIn

Data Science Project Ideas

We love data science and cool data science projects.  If you’re a applying for our free data science fellowship and looking to propose a data science project, here are four project ideas.

startup-849805__340GitHub

GitHub is a great source of data on how engineers write code.  A recent post found discrimination against Pull Requests submitted by women on GitHub, although perhaps that study could have been better.  But there are lots of other ideas to pursue.  We can easily learn an n-gram classifier on whether a line of code is a comment or not and search for commented out code.  Are repos by academics more likely to have commented out code?  Are they more likely to violate lint rules?  Additionally, it would be interesting to analyze commits that are in response to bug fixes to predict in which lines of code bugs are more likely to occur.

Continue reading

Tweet about this on TwitterShare on FacebookShare on LinkedIn