Haystack (ElastiSearch backend) on Heroku Deployment Error


#1

Asking if the myhellowebapp production Heroku setup (such as settings_porudction.py, ‘whitenoise’ or wsgi.py) could cause the haystack API to not ‘GET’ the complete URL?

Background

I have been coding for 2 days trying to get the haystack module to work on Heroku deployment. Long story short it works on local but not on development. The queries work using ‘request’ on the Heroku shell.

My testing seems to have pinned it down to the backend. This is the result of a test to ‘get all’ from a search query. It doesn’t appear to be requesting the full API URL.

Here is the error after following the instructions for debugging here link
http://django-haystack.readthedocs.io/en/master/debugging.html#no-results-found-on-the-web-page

from haystack.query import SearchQuerySet
sqs = SearchQuerySet().all()
sqs.count()

GET /haystack/modelresult/_search?_source=true [status:400 request:0.007s]
Failed to query Elasticsearch using ‘:’: TransportError(400, ‘parsing_exception’, ‘no [query] registered for [filtered]’)

Compared to using the Heroku Python shell it seems to be missing the URL part.

response = requests.get(‘http://[myAPI]@nori-us-east-1.searchly.com/haystack/_search?q=text:the’)
response.raise_for_status()
response.text → [gives info I want]

My question is did we put any settings in the production Heroku settings that would cause this?

Anyway, can I can debug this?

settings_production.py

    # Inherit from standard settings file for defaults
    from hellowebapp.settings import *
    #from boto.s3.connection import S3Connection
    
    # Everything below will override our standard settings:
    import os
    from urllib.parse import urlparse
    # Parse database configuration from $DATABASE_URL
    import dj_database_url
    DATABASES['default'] =  dj_database_url.config()
    
    # Honor the 'X-Forwarded-Proto' header for request.is_secure()
    SECURE_PROXY_SSL_HEADER = ('HTTP_X_FORWARDED_PROTO', 'https')
    
    # Allow all host headers
    ALLOWED_HOSTS = ['*']
    
    # Set debug to False
    DEBUG = False 
    
    # Static asset configuration
    STATICFILES_STORAGE = 'whitenoise.django.GzipManifestStaticFilesStorage'

es = urlparse(os.environ.get('SEARCHBOX_URL') or 'http://127.0.0.1:9200/')
#es = urlparse(os.environ.get('SEARCHBOX_SSL_URL'))
#es = urlparse(os.environ.get('SEARCHBOX_URL'))

port = es.port or 80

HAYSTACK_CONNECTIONS = {
    'default': {
       #'ENGINE': 'haystack.backends.elasticsearch_backend.ElasticsearchSearchEngine',
       'ENGINE': 'haystack.backends.elasticsearch2_backend.Elasticsearch2SearchEngine',
       'URL': es.scheme + '://' + es.hostname + ':' + str(port),
       'INDEX_NAME': 'haystack',
    },
}

if es.username:
    HAYSTACK_CONNECTIONS['default']['KWARGS'] = {"http_auth": es.username + ':' + es.password}

HAYSTACK_SIGNAL_PROCESSOR = 'haystack.signals.RealtimeSignalProcessor'

wsgi.py

import os
os.environ.setdefault("DJANGO_SETTINGS_MODULE", "hellowebapp.settings_production")
from django.core.wsgi import get_wsgi_application
from whitenoise.django import DjangoWhiteNoise

application = get_wsgi_application()
application = DjangoWhiteNoise(application)

and local settings.py

"""
Django settings for hellowebapp project.

Generated by 'django-admin startproject' using Django 1.9.6.

For more information on this file, see
https://docs.djangoproject.com/en/1.9/topics/settings/

For the full list of settings and their values, see
https://docs.djangoproject.com/en/1.9/ref/settings/
"""

import os
# Build paths inside the project like this: os.path.join(BASE_DIR, ...)
BASE_DIR = os.path.dirname(os.path.dirname(os.path.abspath(__file__)))


# Quick-start development settings - unsuitable for production
# See https://docs.djangoproject.com/en/1.9/howto/deployment/checklist/

# SECURITY WARNING: keep the secret key used in production secret!
SECRET_KEY = os.environ['DJANGO_SECRET_KEY']

# SECURITY WARNING: don't run with debug turned on in production!
DEBUG = True

#UPDATE ME BEFORE LAUNCH
ALLOWED_HOSTS = ['*']


# Application definition

INSTALLED_APPS = [
    'collection', # this is the app we just added 20170519
    'django.contrib.admin',
    'django.contrib.auth',
    'django.contrib.contenttypes',
    'django.contrib.sessions',
    'django.contrib.messages',
    'django.contrib.staticfiles',
    'django.contrib.humanize',
    'django.contrib.sitemaps',
    'django.contrib.sites',
    'rest_framework',
    'debug_toolbar',
    'registration', # 3rd party app
    'elasticsearch',
    'haystack',
]

MIDDLEWARE_CLASSES = [
    'django.middleware.security.SecurityMiddleware',
    'django.contrib.sessions.middleware.SessionMiddleware',
    'django.middleware.common.CommonMiddleware',
    'django.middleware.csrf.CsrfViewMiddleware',
    'django.contrib.auth.middleware.AuthenticationMiddleware',
    'django.contrib.auth.middleware.SessionAuthenticationMiddleware',
    'django.contrib.messages.middleware.MessageMiddleware',
    'django.middleware.clickjacking.XFrameOptionsMiddleware',
    'debug_toolbar.middleware.DebugToolbarMiddleware',
]

ROOT_URLCONF = 'hellowebapp.urls'

TEMPLATES = [
    {
        'BACKEND': 'django.template.backends.django.DjangoTemplates',
        'DIRS': [],
        'APP_DIRS': True,
        'OPTIONS': {
            'context_processors': [
                'django.template.context_processors.debug',
                'django.template.context_processors.request',
                'django.contrib.auth.context_processors.auth',
                'django.contrib.messages.context_processors.messages',
            ],
        },
    },
]

WSGI_APPLICATION = 'hellowebapp.wsgi.application'


# Database
# https://docs.djangoproject.com/en/1.9/ref/settings/#databases

DATABASES = {
    'default': {
        'ENGINE': 'django.db.backends.sqlite3',
        'NAME': os.path.join(BASE_DIR, 'db.sqlite3'),
    }
}


# Password validation
# https://docs.djangoproject.com/en/1.9/ref/settings/#auth-password-validators

AUTH_PASSWORD_VALIDATORS = [
    {
        'NAME': 'django.contrib.auth.password_validation.UserAttributeSimilarityValidator',
    },
    {
        'NAME': 'django.contrib.auth.password_validation.MinimumLengthValidator',
    },
    {
        'NAME': 'django.contrib.auth.password_validation.CommonPasswordValidator',
    },
    {
        'NAME': 'django.contrib.auth.password_validation.NumericPasswordValidator',
    },
]

# Users must register within 7 days of initial email
ACCOUNT_ACTIVATION_DAYS = 7

# Internationalization
# https://docs.djangoproject.com/en/1.9/topics/i18n/

LANGUAGE_CODE = 'en-uk'

TIME_ZONE = 'Australia/Adelaide'

USE_I18N = True

USE_L10N = True

USE_TZ = True

# static files
STATIC_URL = '/static/'
STATIC_ROOT = 'staticfiles'
STATICFILES_DIRS = (os.path.join(BASE_DIR, 'static'),)

# media files
MEDIA_ROOT = os.path.join(BASE_DIR, 'media')
MEDIA_URL = '/media/'

# EMAIL SETTINGS

# admin emails
SERVER_EMAIL ='[email protected]'
ADMINS = [
    ('Jared', '[email protected]'),
    ]

EMAIL_BACKEND = 'django.core.mail.backends.console.EmailBackend'
DEFAULT_FROM_EMAIL = '[email protected]'
EMAIL_HOST_USER = ''
EMAIL_HOST_PASSWORD = ''
EMAIL_USE_TLS = False
EMAIL_PORT = 1025

# GMAIL setup
# http://andrewtorkbaker.com/using-environment-variables-with-django-settings
#EMAIL_HOST = 'smtp.gmail.com'
#EMAIL_HOST_USER = os.environ['EMAIL_ACCOUNT']
#EMAIL_HOST_PASSWORD = os.environ['EMAIL_PASSWORD']
#EMAIL_PORT = 587
#EMAIL_USE_TLS = True

# END EMAIL SETTINGS

# Stripe #http://andrewtorkbaker.com/using-environment-variables-with-django-settings
STRIPE_SECRET = os.environ['SECRET_KEY']
STRIPE_PUBLISHABLE = os.environ['PUBLISHABLE_KEY']

# REST API
REST_FRAMEWORK = {
    # Use Django's standard 'django.contrib.auth' permissions,
    # or allow read-only access for unauthenticated users.
    'DEFAULT_PERMISSION_CLASSES': [
    # we're going to use this because we're just showing data
    'rest_framework.permissions.AllowAny',
    # BUT use this one or another restricted permission if you
    # update your API to allow update and deleting
    # 'rest_framework.permissions.IsAuthenticated',
    ],
}

# haystack search engine configurations
# http://django-haystack.readthedocs.io/en/master/tutorial.html#installation
HAYSTACK_CONNECTIONS = {
    'default': {
        'ENGINE': 'haystack.backends.elasticsearch2_backend.Elasticsearch2SearchEngine',
        'URL': 'http://127.0.0.1:9200/',
        'INDEX_NAME': 'haystack',
    },
}

# session data
import django.conf.global_settings as D_SETTINGS
TEMPLATE_CONTEXT_PROCESSORS = D_SETTINGS.TEMPLATE_CONTEXT_PROCESSORS + ['django.core.context_processors.request']

LOGIN_REDIRECT_URL = "home"

Example project github (it also appears to be based on hellowebapp!) → https://github.com/searchly/searchly-django-haystack-sample/blob/master/django_haystack_sample/settings.py

2 days of furious debugging has left me a bit helpless on this

I can see from the elastisearch backend that running manage.py rebuild_index / update_index has worked on Heroku deployment as well as the local deployment

Here is a pastebin of the traceback from heroku python shell
https://pastebin.com/b8K6JKCC


#2

Thanks for taking the time to write this out! I’m personally unfamiliar with Haystack but I’ll ask around my local network and see if they can respond to this post.


#3

Solved by moving from ‘searchly’ Heroku addon to Heroku ‘bonsai’ :/

a lot of troubleshooting for a relatively simple fix. had to clear index before changing from searchly to bonsai, then rebuid index.

I don’t know the actual code fix but for my purposes this is fine.


#4

Thanks for the update and sorry I wasn’t successful in getting outside help!