Tuesday, June 21, 2016

Django 1.7, 1.8 - queryset.extra is deprecated, how to do group by on datetime with date

In Django 1.5/1.6 version -

signup_count = list(User.objects.filter(
            profile__user_type='learner').order_by(
                '-id').extra({
                             'date_only': "date(date_joined)"}).values(
                                 'date_only').annotate(
                                     signup_count=Count('id'))[:40])

It used to return list with
[{ 'date_only': , 'signup_count': }]

Now, as the extra is deprecated in newer version of Django, here the work around to get the same -

from django.db.models.expressions import Func

# Create custom sql function
class ExtractDateFunction(Func):

    function = "DATE"

signup_signup = list(User.objects.filter(
            profile__user_type='learner').order_by('-id')annotate(
date_only=ExtractDateFunction("date_joined")).values(
'date_only').annotate(
teacher_count=Count('id'))[:40])

This should give you the same results as before. 

Monday, June 6, 2016

There is no South database module 'south.db.mysql' for your database - Django


Django 1.8+

Recently I come across this error, while running my Django application 
python manage.py runserver

There is no South database module 'south.db.mysql' for your database. Please either choose a supported database, check for SOUTH_DATABASE_ADAPTER[S] settings, or remove South from INSTALLED_APPS.

To fix it, you would try to lookup for south.db.mysql or try to search if you SOUTH_DATABASE_ADAPTERS. But you won't find it in your solution. To fix it you have two choices -

1. Manually downgrade to lower Django version i.e.1.6 or so. 
pip install Django==1.6.10

2. Uninstall South from your environment (virtual environment) and move to built-in migration process. 
pip uninstall south


Good luck!

Saturday, January 16, 2016

Search in Django with Haystack using Solr or Elastic Search

Lets say you want to provide search on your Django application. In specific model, or file search on your media files or data files uploaded by users.

Here are tech solution for it -

HayStack - Modular search for Django
It allows querying on top of any search engine from following - Solr, ElasticSearch, Xapian, Whoosh.

Solr and ElasticSearch is built on top of powerful search server Apache Lucene. Both are free, and under Apache License 2.

Interesting presentation on Solr vs ElasticSearch

ElasticSearch is distributed, some functions in Solr doesn't not allow distributed execution. Easy cloud support with third party. easy to scale, add/remove nodes. ES is realtime and distributed.

Solr and ElasticSearch both provides admin page, in ES its called ElasticSearch-Head. ES also provides concept of GateWay, which allows index recovering if the system crash in any case.

Use ES if - index is big and real time, several indices, multi tenancy requirement, want to save administrative effort and cost.
Don’t use ES if - your company is relatively new, and already using Solr, or no real-time search indexing required,  relatively small indices

Utility other than ElasticSearch-Head, is ElasticSearch-bigdesk which provides analytics and charts.

Solr There are some concern when real time index updates and search queries been performed. For plain vanilla search Solr out performs and works much better than ES.

You can find more comparison here.

Solr is older than ElasticSearch, so it got bigger community and help available online. At the same time ElasticSearch was built in order to overcome the scaling limitation of Solr. ES is stable, though Solr is more mature.In terms of scalability, ElasticSearch is easy to scale compare to Solr, but with Solr 4.0 that limitation will be gone as per the documentation.

Sematext provides support for both Solr and ElasticSearch, you can find good overview and comparisons on various categories in this series of blog post by them.

and now the competition is joined by Amazon CloudSearch, applications which use AWS for hosting also seems widely using CloudSearch. Here is comparison between CloudSearch and Solr. There is no clear winner! Make choice based on requirement of your environment. Try to keep it simple, unless its really required.

Wednesday, January 13, 2016

Python 32bit or 64bit ?

Recently I moved my application from centOS 5 to centOS 7, which had 64bit python installed. It end up crashing my django application because some of the packages I was using were compiled in 32bit python and they weren't compatible.

First thing you need to check whether the python you are running is 32 bit or 64 bit. Here is the simple command to check it -

$ python
Python 2.7.5 (default, Nov 20 2015, 02:00:19) 
[GCC 4.8.5 20150623 (Red Hat 4.8.5-4)] on linux2
Type "help", "copyright", "credits" or "license" for more information.
>>> import struct
>>> print struct.calcsize("P") * 8
64

That means its 64bit!

Friday, January 8, 2016

Redis server installation on webfaction/ on shared server

I come across redis while using sentry. With the latest version 7.7.4, you need redis to run sentry.

Here are the steps to install Redis server on shared server of webfaction -

First fetch redis installation on home directory (of your account)
$ wget "http://download.redis.io/releases/redis-3.0.6.tar.gz"

Extract it and remove the version name from the directory.
$ tar -xzf redis-3.0.6.tar.gz
$ mv redis-3.0.6 redis
(to keep it clean remove the redis-3.0.6.tar.gz)

Run installation
$ make
Run test to verify that installation was correct
$ make test
Now go to webfaction and create custom app, so that we can get port number and we can use it at various places on the configuration.

It will get you the new port information and creates the directory based on the name you have given inside the webapps directory in your account.

As per above, your app name is redis_server and custom port for it - 19957

Now copy redis.conf from extraction to the webapps.

$ cd ~/webapps/redis_server/
$ cp ~/redis/src/redis-server .
$ cp ~/redis/src/redis-cli .
$ cp ~/redis/redis.conf .
Update three items in the redis.conf file.

daemonize no > daemonize yes
pidfile /var/run/redis.pid > pidfile /home//webapps/redis_server/redis.pid
port 6379 -> port 19957
Now try to run it manually to verify once. (ideally we want to run it in background)
./redis-server redis.conf

Once it’s running, you can test if its running fine or now by going to cli
./redis-cli -p 19957

It should prompt -
127.0.0.1:19957>

Otherwise it will say-
not connected >

you can get out of the cli mode by Ctl + d

You can automate this commands by creating Makefile

vi Makefile
client cli:
       ./redis-cli -p 19957
start restart:
       ./redis-server redis.conf
stop:
       cat redis.pid | xargs kill

In order to start now, you can use
make start

to stop
make stop

you can manually search and find the redis process
ps -u $USER -o pid,command | grep redis

and kill it manually

Would still prefer the clean way of

Wednesday, January 6, 2016

Sentry - That page number is less than 1 [error]

While running sentry, when you click on your project, it throws Internal server error with some random code. If you look at the log, it shows something like following. Only thing stands out is "That page number is less than 1"  

  File "/home/User/.virtualenvs/sentry/lib/python2.7/site-packages/sentry-6.4.4-py2.7.egg/sentry/templatetags/sentry_helpers.py", line 217, in paginator
    result = paginate_func(request, queryset_or_list, per_page, endless=True)
  File "/home/User/.virtualenvs/sentry/lib/python2.7/site-packages/paging/helpers.py", line 24, in paginate
    'paginator': paginator.get_context(page),
  File "/home/User/.virtualenvs/sentry/lib/python2.7/site-packages/paging/paginators.py", line 96, in get_context
    'previous_page': paginator.previous_page_number(),
  File "/home/User/.virtualenvs/sentry/lib/python2.7/site-packages/Django-1.5.8-py2.7.egg/django/core/paginator.py", line 143, in previous_page_number
    return self.paginator.validate_number(self.number - 1)
  File "/home/User/.virtualenvs/sentry/lib/python2.7/site-packages/Django-1.5.8-py2.7.egg/django/core/paginator.py", line 30, in validate_number
    raise EmptyPage('That page number is less than 1')

EmptyPage: That page number is less than 1

This error is because of the django-paging version, you may have version lower than 0.2.5. you need >=0.2.5 

Run the following command -

pip install django-paging==0.2.5

And restart your sentry server (web). 

Thursday, December 31, 2015

ReferenceError: Promise is not defined on gulp build

While going thru getting started guide of polymer starter kit, I run into following issue -

> gulp
[15:12:04] Using gulpfile ~/Apps/js/polymer-playground/example/polymer-starter-kit-1.2.1/gulpfile.js
[15:12:04] Starting 'clean'...
[15:12:04] Finished 'clean' after 18 ms
[15:12:04] Starting 'default'...
[15:12:04] Starting 'copy'...
[15:12:04] Starting 'styles'...

/Users/UserName/Apps/js/polymer-playground/example/polymer-starter-kit-1.2.1/node_modules/gulp-autoprefixer/node_modules/postcss/lib/lazy-result.js:157
        this.processing = new Promise(function (resolve, reject) {
                              ^
ReferenceError: Promise is not defined

    at LazyResult.async (/Users/UserName/Apps/js/polymer-playground/example/polymer-starter-kit-1.2.1/node_modules/gulp-autoprefixer/node_modules/postcss/lib/lazy-result.js:157:31)

The issue is my node version, it needs to be 0.12.5 (it was 0.10.3)

Here are the steps to upgrade -
$ sudo npm cache clean -f
$ sudo npm install -g n
$ node -v
v0.10.35

It still says 0.10.35

Here is the way to switch between versions -

$ sudo n stable
$ node -v
v5.2.0

means Now we can move to 0.12.5 in easy steps -
$ sudo n 0.12.5
$ node -v
0.12.5

Now running gulp build works fine!