Tweetalysis Blog

Today I worked on and off for about 12 hours trying to understand and customize Phirehose to my application.  I got a stream of posts related to starbucks to run in command line which was cool to see and test.  The response time is impressive, I sent out my own tweet with the word starbucks in it and before I could even tab over to the command line window it was there.

Screenshot of php interface to twitter streaming API

The hardest thing for me to figure out was how to decouple the collection and processing components.  It turns out writing the JSON encoded responses straight to a text file is the easiest way but that took me a long time to implement.  Currently I’m trying to adapt this slightly out of date code so that new files are generated for each hour.  My code isn’t throwing any errors but it also isn’t creating the text files I want.

<?php
require_once('../lib/Phirehose.php');

class SampleConsumer extends Phirehose
{
protected $trackWords = array('starbucks');

/*
* Enqueue each status
*/
public function enqueueStatus($status)
{
/*
* So the JSON decode here is just for debuggin purposes, the final ersion won't need that line, infact shouldn't have that line.
* We are going to make a file every hour, so that we can simply run a CRON task hourly at like 5 after to process the data
*/
$data = json_decode($status, true);
print $data['user']['screen_name'] . ': ' . urldecode($data['text']) . "\n";
$time = date("YmdH");
if ($newTime!=$time)
{
@fclose($fp);
$fp = fopen("{$time}.txt","a");
}
fwrite($fp,$data);
$newTime = $time;
}
}

// Start streaming
$sc = new SampleConsumer('username', 'pass', Phirehose::METHOD_FILTER);
$sc->consume();
?>

In the coming weeks I hope to begin working on the actual sentiment analysis phase of the project.  It has been suggested to use WEKA, I’ve started exploring the software but was focused on getting usable data into a database today and didn’t get very far.

This entry was posted on Tuesday, November 3rd, 2009 at 7:14 pm and is filed under Uncategorized. You can follow any responses to this entry through the RSS 2.0 feed. You can leave a response, or trackback from your own site.