Python on TUTYSARA'S SPACE

Machine Learning Flashcards from Twitter -- Part 2 Data Analysis and Download

Sun, 11 Feb 2018 00:00:00 +0000

This is the analysis part where we do a small analysis to find

Which are the most important/popular tweets
Whether older materials covered important concepts than recent tweets

Load the necessary libs

%load_ext autoreload
%autoreload 2
%matplotlib inline


import pandas as pd
pd.set_option("display.width", 150)
import requests
import re
import matplotlib.pyplot as plt
import matplotlib.image as mpimg
from pathlib import Path

Load csv

df = pd.read_csv("chrisalbon_mlflashcards.csv")
df['text'].fillna('', inplace=True)

df.tail()

 id likes replies retweets text timestamp url img_url
236 94607825069... 19 1 0 Bayes Error 2017-12-27T... https://twi... https://pbs...
237 94575108497... 47 3 13 Occams Razor 2017-12-26T... https://twi... https://pbs...
238 94571793723... 8 0 1 K-Fold Cros... 2017-12-26T... https://twi... https://pbs...
239 94538342129... 18 1 1 Extrema 2017-12-25T... https://twi... https://pbs...
240 94536381783... 34 1 7 Softmax Act... 2017-12-25T... https://twi... https://pbs...

Check for missing values

df.count(axis=0)

id 241
likes 241
replies 241
retweets 241
text 241
timestamp 241
url 241
img_url 237
dtype: int64

We see we have few img_urls missing. Let’s check what they are

Machine Learning Flashcards from Twitter -- Part 1 Data Collection and Preprocessing

Fri, 09 Feb 2018 00:00:00 +0000

I was searching the net for mlflashcards, I found this incredible machine learning flashcard tweet series from Chris Albon. It looks pretty and covers a lot of ground, Got a thought – why not download them for later use? I thought it would be a fun exercise to start the weekend and jumped into action.

Step 1 – Collect/Scrape data from twitter

I evaluated using twitter api using tweetpy, but it has its own limitation aka we can search only a week worth of data which is not good for our use case. We shoud be able to get data spread across months since the tweets we are interested are spread across a wide time range.

Themes and Extensions for Jupyter notebook

Wed, 07 Feb 2018 00:00:00 +0000

I am taking the wonderful fast.ai course thought by Jermey Howard and Rachel. We use jupyter notebooks in classes and it is always nice to see how Jermey’s notebook looks.

It has few tweaks like

collapsible headings
better tables and fonts

Learnt that he uses a themes for the visual customizations and an extensions to allow collapsing of headers from his reply in forums

The theme script can be installed from jupyter-themes repo

Use Skipgrams with sklearn CountVectorizer and TfidfVectorizer

Sun, 07 Jan 2018 00:00:00 +0000

We don’t have an implementation for skipgrams in sklearn. This post covers how to use the skipgram function in nltk with sklearn’s CountVectorizer and TfidfVectorizer

We are going to create a skipgram tokenizer that can be passed to the tokenizer parameter of the vectorizer.

Create a basic tokenizer that can split the original strings to tokesn. Tokenizer can be just .split() or a function to filter non-alpahbets etc. We can use tokenizer as below

Django development on OSX using emacs

Wed, 30 Oct 2013 13:16:00 +0000

I started following the Django tutorials by arun. I wanted to know about the related classes and quickly look into docs for the things that I code, I wanted an environment that is nice to newbies, helping with code completion and easy documentation lookup. I searched and found a huge list of options. Since I was begining with Django I wanted an environment that is much easier to setup, my first choice was pycharm, but unfortunately the community edition of pycharm doesn’t support Django - see this.