ran a little experiment in using yahoo term extraction yesterday and it worked well enough. here’s some code to pass some text to yahoo and get back an array of terms
i’ve got to say mongodb is such an easy tool for working with json data. these 20 odd lines insert a text json tweet stream into mongo. so simple, why can’t all code be this easy…
this week i’m between jobs so i have (a little) more time than usual to hack.
i’ve got a list of pending things to do but can’t decide what to do next, here’s my list in (sort of) priority order…
decisions, decisions….
experiment 13; a test of semi supervised naive bayes for text classification is complete.
semi supervised algorithms seem to work pretty well and i can see how they are a huge benefit for text classification where you can have an enormous corpus but not enough time to label it all…
after quite a bit of hacking the statistical synonyms idea doesn’t seem to give terribly interesting results so i’m going onto do something else.
for the record here’s what I did do though….
viola! click this image for a bigger version
some interesting result. few of the more complex things i was trying were working. they were mainly based on trying to incorporate the frequencies of terms but it seemed the simplest gave the best result (i think it’s because my assumptions about how to use the data were wrong).
here’s the code, feel free to read my notes, correct my incorrect terrible statistical assumptions and make a better image!
here’s a great lecture from tom mitchell about document classification using a semi supervised version of naive bayes.
semi supervised algorithms only require some of the training examples to be labeled and are able to make use of any unlabelled ones, very common when we have a huge corpus.
i’ve started an experiment brewing to test this out by porting some previous naive bayes work i did to use this semi supervised scheme and will published it when it’s done.
cool stuff!!
i’ve been doing some reading for my statistical synonyms project and have uncovered a heap of cool papers. most of them are around an idea (from the 1950’s!) called the distributional hypothesis that simply states that words that appear in similar contexts often have similar meanings.
the coolest paper so far is ‘Web-Scale Distributional Similarity and Entity Set Expansion’ by Pantel,Crestan,Borkovsky et al which has introduced me to an area of research i didn’t really know existed; entity set expansion.
entity set expansion is a bit like thesaurus building for proper nouns; given a seed set of related items can you expand the set to include other semantically similiar items?
an example might be brands of japanese motorbikes. starting with ‘yamaha’ and ‘kawasaki’ we might expect the set to be expanded to include ‘honda’
i started hacking around in pig but today switched back to ruby for slightly quicker prototyping. who knows, i might give piglet a go!
the code is on github
i’ve had an idea brewing in my head for awhile now seeded by a great talk by peter norvig about statistically approaches to find patterns in data.
one thing he alludes to is the generation of synoyms based on n-gram models.
the basic intuition is this; if a corpus contains occurrences of the phrases ‘a x b’ and ‘a y b’ then to some degree x and y are synonymous.
the question becomes how do we calculate the strength of the relationship? how is it a function of the frequencies of a, b, x, y, ‘a x b’, ‘a y b’, ‘a ? b’ in the corpus. what else can we take into account?
pig demo from Mat Kelcey on Vimeo.
based on a talk i gave at work recently
people tweet about all sorts of stuff.
sometimes it’s really important ground breaking world changing stuff…
but most of the time it’s ridiculous waste of time stuff like ‘i ate some cheese’
in fact how much do people actually tweet about cheese?
and when they do, what are the most important cheese related topics?
lets gather some data…
just recently discovered xargs has a parallelise option!
i have 20 files, sample.01.gz to sample.20.gz, each ~100mb in size that i need to run a script over
one option is
zcat sample*gz | ./script.rb > output
but this will process the files sequentially on a single core.
to get some parallel action going i could generate a temp script that produces
zcat sample.01.gz | ./script.rb > sample.01.out & zcat sample.02.gz | ./script.rb > sample.02.out & ... zcat sample.20.gz | ./script.rb > sample.20.out &
and run that but this will have all 20 running at the same time and produce contention
(though with only 20 files this might not be a problem)
instead i can make a temp script, parse.sh
zcat $1 | ./script.rb > $1.out
and run
find sample*gz | xargs -n1 -P4 sh parse.sh cat *out > output
what is this xargs command doing?
100% on all cores (and only because the disk can keep up)
awesome!
