Skip to Content

New Blog Post: How to keep all your websites in sync with scraping technology

Technology Blog

Technology Blog

Django Apps + Development

Imaginary enjoys sharing our love for the dynamic Django open-source web framework. Our posts talk about applications that allow ease of development for complex, database-driven websites. We discuss reusability and plug-ins components, rapid development, Django programming techniques, and Django's configuration, code, templates, and data models.

Implementing dependency management with Python Poetry

One of the biggest issues with Python (our preferred development language) is dependency management.  Any individual deployment can result in minor differences in the versions of the files and libraries which make up the application.  And these minor variations introduce uncertainty and randomness into our deployment process. So, we've investigated…

Migrate Away from cmsplugin-filer in a Few Easy Steps

Companion code for this post: https://github.com/ImaginaryLandscape/deprecate_cmsplugin_filer If you've been building projects using django CMS for any length of time, chances are you're familiar with Divio's cmsplugin-filer application which provided image, link, file, folder and video plugins for interacting with django-filer. And if you're here, chances are you're aware that cmsplugin-filer has now...

Website Search using Django and PostgreSQL Trigrams

Over the years I've become increasingly wary of the word "easy" in software documentation. Pick a software project at random, and there's a good chance the documentation will lead off with something like "Booloogent makes the process of frobnifying your wakalixes easy!" And then you try to use the package...

Multipage Forms in Django

Introduction Most online forms fit on a single page.  Think of a "join our forum" or "contact us" form into which the user enters a name, email address, and maybe a few other pieces of information.  If you're building this kind of functionality into a Django site, you can take...

Django shell_plus with Pandas, and Jupyter Notebook

The Django shell provides an environment where developers can interact with the database via Django's ORM. While the shell is great for basic interactions, it quickly becomes laborious to work in, be it due to manual imports, having to scroll through shell history to repeat commands, or working with / viewing queries returning more than, say, 20 records. These issues, and more, can be remedied by interacting with the ORM in Jupyter notebooks, using Pandas.

Step-by-Step Guide to Setting up Django-RQ

Modern websites are complicated pieces of machinery. The requests they initiate can be intensive and take more time to complete than a typical HTTP request-response cycle. This is typically refererd to as a blocking request, in that the browser (and user) waits until the response is received. If the request...

Scraping pdf, doc, and docx with Scrapy

In February 2017, Google announced its plans to discontinue its Google Site Search product. Those clients of Imaginary Landscape who had relied on Google to provide their users with a search engine service for their website looked to us for a new solution. Finding no obvious equivalent replacement, we decided to create our own website scraper and accompanying search app.

The case for a Django upgrade

It boils down to this. An upgrade costs money, sometimes a lot of money, but the result has no visible outcome. In fact, in many cases the only outcome is an assurance that you've reduced the probability of attack, intrusion, breach and related unpleasantness. By any measure, that's a tough...

Writing Functional Python

The recent buzz surrounding Functional Programming (FP for short) is well deserved. Languages like Haskell and Clojure in use by big name companies such as Barclays Captial, Atlassian, and Walmart have helped FP ideas gain a major foothold in industry. I am a Clojurist hobbyist who has been interested in FP for...

End User Focused Django Tutorials

Django is being used by corporations worldwide.  Every day, Imaginary does its part to increase this footprint by deploying Django websites and applications to our clients. As a result, Django is being exposed to greater numbers of end users. I'm not talking end user developers.  I'm talking Samantha in marketing...

Braintree Integration with Django

Braintree Integration in Django Introduction Much has been written about how to implement online payment using the services available from Braintree, but not much has been written about how to do it within a Django framework. Combining the two does not require any special contortions, but as Braintree’s own online...

Django Class-Based Views: The Basics

Class-based views are my favorite feature of Django, so there is an obvious bias when discussing function views and class-based views.  I’ll throw my two cents in here and move on.  I prefer the structure and predictability that class-based views provide. I recognize that there are many instances when a...

Django and IPython Notebook

The IPython Notebook is a really cool application, and I've always wanted to use it during Django development and debugging.  The only problem is that it requires a lot of dependencies and I feel no need to encumber my production projects with those dependencies for a feature that I only...

Debug Toolbar Primer

The Django Debug Toolbar is a powerful and useful tool for identifying potential trouble spots in your Django application.  Hopefully, this post will serve as a starting point for understanding the information presented by the toolbar.  Also, we will cover a couple of plugins for the toolbar, cache panel, and...

Simple Job Queues with django_rq

The de facto solution for job queues with background workers is Celery and RabbitMQ, but it is not the right fit for every project. RQ is an alternative to Celery, and while not as featureful, does provide a lightweight solution that makes it easy to set up and use. RQ...

Geo Django Quickstart

GeoDjango is a very powerful tool for storing and manipulating geographic data using the Django ORM.  It provides a simple API to determine distances between two points on a map, find areas of polygons, locate the points within a polygon, and much more.   GeoDjango is fairly well documented on...

Django Admin Snippets

At its minimum, the Django Admin is an effective tool for viewing and manipulating data within a Django database.  At its max, it can be a robust application allowing clients and administrators to better manage their web applications.   This article aims to highlight some of the admin customizations that...

Django Form Snippets

Django forms are very convenient tools for Django development, but sometimes I find myself wrangling with them to get them to do what I want them to do.  I’ve compiled a few simple form examples that I find useful for various form-related tasks. Say you have a model that looks...

Top 5 Favorite Django Modules

Of all the Django modules that are out there, several stand out as ones that I keep coming back to.  These are some of my favorites:  Django Debug Toolbar Django Debug Toolbar is an extremely useful Django application for debugging a site from “Django’s perspective.”  It should only be used...

Magic Links

The first time I used the new link attachment tool found on sites like Facebook, I thought I had experienced something magical.  After pasting an external webpage URL into my wall-post form, an image, page title, and page description of the linked webpage appeared beneath my post as though out...