RSS pubDate to python date

In RSS feeds one of the field is pubDate, which is common across any feed. This date is required to be in RFC 822 - Standard for ARPA Internet Text Messages.

Sample - Sat, 17 Sep 2012 00:00:01 GMT

if you want to convert it to python date time, get help from email.utils of python
e.g.

>>> import rfc822

>>> rfc822.parsedate_tz('Thu, 26 Jul 2012 13:30:52 EDT')

(2012, 7, 26, 13, 30, 52, 0, 1, 0, -14400)

Though above gives tuple, which you will need to convert to datetime.

If you dont care about timezone (e.g. EDT, GMT etc.), use below -

>>> from datetime import datetime

>>> datetime.strptime('Thu, 26 Jul 2012 13:30:52 EDT'[:-4], '%a, %d %b %Y %H:%M:%S')

datetime.datetime(2012, 7, 26, 13, 30, 52)

I am sure there are many other ways to do this, suggest if you come across any good one :)

Boilerpipe integration in python

Boilerpipe is a library for boilerplate removal and full text extraction from HTML. In most of the scenarios it works pretty amazing, you can try out here.

We wanted to use it with python, and so tried out the python wrapper for Boilerpipe.

It requires JPype install prerequisites, which is available here.

Once its downloaded, run 'sudo python setup.py install'

While installation I run into different errors, and following are the steps for it -

1. Install JPype.

command gcc fail error -

error: command 'gcc-4.2' failed with exit status 1

I followed the explanation here to get it install. Basically you need to update the javaHome path in setup.py based on your machine (Mac, Windows or other Linux based on your platform change appropriate method.) Next step is find out the Java path on your machine, and add it in .bash_profile if its not already set.

I did following changes in my setup.py–

def setupMacOSX(self):

        self.javaHome = '/Developer/SDKs/MacOSX10.7.sdk/System/Library/Frameworks/JavaVM.framework'

        self.jdkInclude = ""

        self.libraries = ["dl"]

        self.libraryDir = [self.javaHome+"/Libraries"]

        self.macros = [('MACOSX',1)]

def setupInclusion(self):

        self.includeDirs = [

            self.javaHome+"/Headers",

            self.javaHome+"/Headers/"+self.jdkInclude,

            "src/native/common/include", 

            "src/native/python/include",

        ]

It should do the trick, and JPype should be installed fine.

2. Run intallation of boilerpipe wrapper, that will the boilerpipe jars and chardet (universal encoding detector) as well to your environment as its also one of the required package for the boilerpipe.

Ran ‘sudo python setup.py install’ on https://github.com/misja/python-boilerpipe to get java boilerpipe wrapper install on your machine.

3. Start running your app as provided instruction on documentation.

I got following error upon running the app -

java.lang.Exception: Class de.l3s.boilerpipe.sax.HTMLHighlighter not found

Its because your $JAVA_HOME is not setup correctly. And another part of the reason was boilerpipe jar were missing from the path. (probably python-boilerpipe install didn't bring boilerpipe java jars, so I brought it manually.) After getting those it works fine.

Please suggest if any other good tool for extraction of the main content out of the web page.

django JSON with DateTime

Django json.dump(data) throws error ‘datetime (...) is not JSON serialzable.’ because default it only does queryset json serialization. Use below to serialize dates.

from django.core.serializers.json import DjangoJSONEncoder

def test(request, title):

    …
    data =  json.dumps(qset, cls=DjangoJSONEncoder)

Which is similar to extending the default JSONEncoder and check for the datetime and return it with extra code to handle it.

This should resolve the issue.

Graph of App Model Django

Generate App Model diagram/graph of you Django app -

Use Django-extensions tool - https://github.com/django-extensions/django-extensions

Its really powerful -  here is the quick guide on the commands you can use with Django-extensions: http://ericholscher.com/blog/2008/sep/12/screencast-django-command-extensions/

To generate the model, follow below installation steps –

sudo pip install django-extensions

Get graphviz from - http://www.graphviz.org/Download..php

another alternative to get it install using below command

brew install graphviz

(graphviz is graph visualization open source software by AT&T) 

sudo pip install pygraphviz

add Django-extensions in settings file of your project

INSTALLED_APPS = (    

                                              ...    

                                              'django_extensions',

                                         )

And you are ready to get app model graph, run following command,

$ ./manage.py graph_models -a -g -o my_project_visualized.png

Bingo!

Unsafe statement written to the binary log using statement format since BINLOG_FORMAT = STATEMENT.

Today while accessing django-admin to update certain information I was keep getting this error on Save - 

Unsafe statement written to the binary log using statement format since BINLOG_FORMAT = STATEMENT. Statement is unsafe because it accesses a non-transactional table after accessing a transactional table within the same transaction.

Platform - django 1.3.1, mysql 5.5

While doing some initial research I came across its happening because of, when you are trying to insert something from admin into the table, first its inserting into temp table or you are reading from temp table and trying to insert into main table. It considers temp table as transactional table. according to following article -

http://bugs.mysql.com/bug.php?id=53259

However, it wasn't much of help. 

You should read about different types of binlog format - http://dev.mysql.com/doc/refman/5.1/en/binary-log-setting.html

you can simply switch to 'MIXED' and that might solve your problem -

SET GLOBAL binlog_format = 'MIXED'; 

If it doesn't then, read further and see different scenarios why it was happening.

I did further research and come across another thread where it was mentioned that, it might be happening because of certain tables are belog to MyISAM Engine, whereas certain tables belong to InnoDB.  I checked my db, and found out that it was correct! 

Tables for which I had downloaded script from server (which runs under mysql 5.0) had default Engine as MyISAM, and my local mysql is 5.5 which has default Engine listed as InnoDB. 

Unless you have specific table that needs a MyISAM specific feature such as fulltext indexes, you should always user InnoDB as Engine as per the most of the recommendations online and by mysql.

As of MySQL 5.5, InnoDB is the default storage Engine. and previous versions had MyISAM as default storage Engine. (reference http://dev.mysql.com/doc/refman/5.5/en/innodb-default-se.html)

to check if your tables using InnoDB as default engine, you can try below command -

SHOW VARIABLES LIKE 'have_innodb';

Below will show all engines for your mysqldb, and which one is default:

  

SHOW ENGINES;

You can check by generating create table script to check if its running under InnoDB or MyISAM.

CREATE TABLE `table_name` (

  `id` int(11) NOT NULL AUTO_INCREMENT,

  `name` varchar(500) NOT NULL

  PRIMARY KEY (`id`) 

) ENGINE=MyISAM AUTO_INCREMENT=11 DEFAULT CHARSET=latin1$$

That means its running under MyISAM engine.

In order to change it to run under innodb, you can execute below alter statement -

ALTER TABLE table_name ENGINE=InnoDB;

On running above you might run into,

0 20:55:09 ALTER TABLE table_name ENGINE=InnoDB Error Code: 1217. Cannot delete or update a parent row: a foreign key constraint fails

This might be because foreign key been used by other tables, if we are updating the Engine for the table, you might want to go in order of foreign key reference.

Find out which tables are on InnoDB -


SELECT table_schema, table_name
FROM INFORMATION_SCHEMA.TABLES
WHERE engine = 'innodb'

If you running into this issue, follow above steps and it should be fine! Comments, Feedback welcome.