Pillow Resizing images (thumbnail) on Amazon S3


#1

Hi @limedaring and Hello Web app friends
Receive you a warm greetings

I am try resize an image when this is saved, by which I am using the code helper of Hello Web App IC.
Specifically, my images are stored on Amazon S3, then I use the django-storages and boto3 third party applications

When a image is saved, this is stored in my Amazon S3 bucket, having a acces url such as follow https://s3-sa-east-1.amazonaws.com/ihost-project/media/studyoffer_images/algoritmos-para-ensenanza/15061122523583.jpg

The code to save and resize the image is this:

class UploadStudyOffer(models.Model):
    study_offer = models.ForeignKey(StudiesOffert, related_name='uploadsstudyoffer')
    image = models.ImageField(upload_to=get_image_path)
    # images folder per object

    def save(self, *args, **kwargs):
        super(UploadStudyOffer, self).save(*args, **kwargs)
        # We first check to make sure an image exists
        if self.image:
            # Open image and check their size
            image = Image.open(self.image)
            i_width, i_height = image.size
            max_size = (100,100)

            # We resize the image if it's too large
            if i_width > 1000:
                image.thumbnail(max_size, Image.ANTIALIAS)
                image.save(self.image.path) 

When I upload an image, I get this message

Exception Type: NotImplementedError at /host/study-offer/algoritmos-para-ensenanza/edit/images/
Exception Value: This backend doesn't support absolute paths.

And I am not sure, if the error is manage at storages or boto backends or in Pillow.
Then at level of Pillow I found the following options in the momento of save the image, such as follow:

I change the options in image.savesection code, staying of this way:

    def save(self, *args, **kwargs):
    super(UploadStudyOffer, self).save(*args, **kwargs)
    # We first check to make sure an image exists
    if self.image:
        # Open image and check their size
        image = Image.open(self.image)
        i_width, i_height = image.size
        max_size = (100,100)

        # We resize the image if it's too large
        if i_width > 1000:
            image.thumbnail(max_size, Image.ANTIALIAS)
            
           # Saving the image with the name attribute
            image.save(self.image.name) 
            # image.save(self.image.file)
            # image.save(self.image.path) #This backend doesn't support absolute paths.

Then I get this error:

File "/home/bgarcial/workspace/hostayni_platform/hosts/models.py" in save
  542.                 image.save(self.image.name) #[Errno 2] No such file or directory: 'studyoffer_images/ingenieria-de-sistemas/15061122523583.jpg'

File "/home/bgarcial/.virtualenvs/hostayni/lib/python3.6/site-packages/PIL/Image.py" in save
  1725.             fp = builtins.open(filename, "w+b")

Exception Type: FileNotFoundError at /host/study-offer/algoritmos-para-ensenanza/edit/images/
Exception Value: [Errno 2] No such file or directory: 'studyoffer_images/algoritmos-para-ensenanza/1900-X-1080-Wallpapers-022.jpg'

Of course, my image is stored om Amazon S3 and not locally in my project or hard disk, then I use the url parameter of this way:

I change

image.save(self.image.name) 

to

image.save(self.image.url)

And I get this error:

Exception Type: FileNotFoundError at /host/study-offer/algoritmos-para-ensenanza/edit/images/
Exception Value: [Errno 2] No such file or directory: 'https://s3-sa-east-1.amazonaws.com/ihost-project/media/studyoffer_images/algoritmos-para-ensenanza/15061122523583.jpg'

Getting the amazon s3 image URL does not works, even though the url is a valid url https://s3-sa-east-1.amazonaws.com/ihost-project/media/studyoffer_images/algoritmos-para-ensenanza/15061122523583.jpg

Then I change

 image.save(self.image.url)

to:

image.save(self.image.file)

And my image is uploaded without any errors, but is not resized and is uploaded as its original format.

How to can I process a image uploaded from my application and their result was saved on Amazon S3 to after use them?


#2

Hi @bgarcial,

I believe what you would want to do in this situation is to use a buffer object, either a cStringIO object or a BytesIO object, for Python 2 and Python 3 respectively. This buffer object serves as the “file” that PIL will save your image data into before it is saved to S3 by the storage backend.

(By the way, I would suggest not calling the PIL Image object that you use to perform the thumbnail operation as “image”, just because there are so many other things using that name–it could get confusing. Just a suggestion. Maybe “thumbnail_image”?)

An example modified from your code snippet, for python 3 (since I see the python3.6 in your error traceback):

from io import BytesIO
from django.core.files import File
from pathlib import Path  # python 3.6+ only!

# This list of image types is just an example, use what
# you might actually encounter instead.  See PIL docs
# for format names it knows about.

image_types = {
    'jpg': 'JPEG',
    'png': 'PNG',
    'gif': 'GIF',
    'tif': 'TIFF',
}

if i_width > 1000:
    buffer = BytesIO()

    thumbnail_image = Image.open(self.image)
    thumbnail_image.thumbnail(max_size, Image.ANTIALIAS)

    # I am not sure where you get your image's filename from
    # but here I am calling it image_filename.  Please substitute
    # that name with whatever is necessary in your code.  By
    # filename I mean just the filename, not the whole path.
    # Perhaps you can use "Path(self.image.path).name" ?
    filename_suffix = Path(image_filename).suffix[1:]
    image_format = image_types[filename_suffix] 

    # You must tell PIL which format to use because there is no
    # filename in the .save() call for it to guess from, since
    # we are saving the thumbnail data to a virtual buffer.

    # First we save the image data into the buffer
    thumbnail_image.save(buffer, format=image_format)

    file_object = File(buffer)
    # You may wish to set the mimetype of the file:
    file_object.content_type = 'image/jpeg' # perhaps use "mimetypes.guess_type()" ?

    # Now we actually save the data to S3 through the storage backend
    self.image.save(image_filename, file_object)
    
    # If you need to save the whole model object, you must do that separately.
    self.save()

I have adapted this example from my own code, which uses a FileField rather than an ImageField, but since the ImageField is a subclass of FileField with just a few extra attributes to support access to the height and width of the file, it should work the same or very similarly.

I hope this helps you to fix the problem quickly. Best of luck.
P.H.


#3

@phoikoi Thanks for your approach.
This process is perform on the override the save() method really?


#4

Yes, I wrote it to fit into your last code snippet where the if i_width > 1000 test is. (The imports go at the top of the file, of course.)


#5

HI @phoikoi

I’ve changed the PIL Image object name which I am using to perform the thumbnail operation, this now is called thumbnail_image according to your reccomendation.

With respect to this quote that you says me:

# I am not sure where you get your image's filename from
# but here I am calling it image_filename.  Please substitute
# that name with whatever is necessary in your code.  By
# filename I mean just the filename, not the whole path.
# Perhaps you can use "Path(self.image.path).name" ?

I am get the filename using self.image.file which I get the string name of image object (I am in right?)
This in the filename_suffix

Then my class and override save() method stay so:

class LodgingOfferImage(models.Model):
    lodging_offer = models.ForeignKey(LodgingOffer, related_name='lodgingofferimage')
    image = models.ImageField(upload_to=get_image_filename, verbose_name='Imagen',)

    def save(self, *args, **kwargs):
        super(LodgingOfferImage, self).save(*args, **kwargs)
    
        # We check to make sure an image exists
        if self.image:
            thumbnail_image = Image.open(self.image)
            # Open image and check their size
            i_width, i_height = thumbnail_image.size
            max_size=(1000,1000)
    
            image_types = {
                'jpg': 'JPEG',
                'png': 'PNG',
                'gif': 'GIF',
                'tif': 'TIFF',
            }
    
            # We resize the image if it's too large
            if i_width > 100:
                buffer = BytesIO()
    
                thumbnail_image = Image.open(self.image,)
                thumbnail_image.thumbnail(max_size, Image.ANTIALIAS)
    
                filename_suffix = Path(self.image.file).name[1:]
                print(filename_suffix)
                image_format = image_types[filename_suffix]
    
                thumbnail_image.save(buffer, format=image_format)
    
                file_object = File(buffer)
                file_object.content_type = 'image/jpeg'
    
                self.image.save(self.image.file, file_object)
                self.save()

But when I upload a image I get this error:

File "/home/bgarcial/workspace/hostayni_platform/hosts/models.py" in save
  344.                 filename_suffix = Path(self.image.file).name[1:]

File "/usr/local/lib/python3.6/pathlib.py" in __new__
  979.         self = cls._from_parts(args, init=False)

File "/usr/local/lib/python3.6/pathlib.py" in _from_parts
  654.         drv, root, parts = self._parse_args(args)

File "/usr/local/lib/python3.6/pathlib.py" in _parse_args
  638.                 a = os.fspath(a)

Exception Type: TypeError at /host/lodging-offer/apartamento-ochentero-37-38-39-40-41-48-82-83/edit/images/
Exception Value: expected str, bytes or os.PathLike object, not S3BotoStorageFile

I think that the filename_suffix does not getting come value in relation to name or string path of the image.

In this section code I try print it and cannot see nothing value …

filename_suffix = Path(self.image.file).name[1:]
   print(filename_suffix)
   image_format = image_types[filename_suffix]

The image, unless of error, is saved of anyway.
I don’t know If I am doing the proccess of a suited way

UPDATE:
When I change the condition if i_width > 1000: by if i_width > 100: the image is saved and I don’t get any error,
but the resize is not performed, the size of image uploaded to S3, width and height are keep equal.

I think that this helper works to images with specific size, or unless width size.
Is possible take any image size and crop to width and height specific?


#6

Hi @bgarcial,

In regards to the error expected str, bytes or os.PathLike object, not S3BotoStorageFile, I think the self.image.save() call is needing to use Path(self.image.file.name).name instead of self.image.file. The value self.image.file.name includes the parent path string (“parent/filename.jpg”), and so the Path.name property gives you just the filename, with no preceding path (“filename.jpg”).

The issue would be the same in the second error you mention with the filename_suffix, my mistake! Sorry! Make the line as filename_suffix = Path(self.image.file.name).name[1:] and see if it works then.

PIL (Pillow) works on any size image–I have used it for very large images–but perhaps the preceding problems were causing it to fail? See if those help this error as well.


#7

Hello, I have been searching for an answer to this for weeks now but finally came upon this thread and think you may be able to help me out.

I have a Django based blog application that is currently connected to Heroku. Media files are being served from S3 Bucket AWS.

I am also using PIL to resize the image in my models.py file:

from django.db import models
from django.contrib.auth.models import User 
from PIL import Image 
from django.core.files.storage import default_storage as storage

class Profile(models.Model):
	user = models.OneToOneField(User, on_delete=models.CASCADE)
	image = models.ImageField(default='default.jpg', upload_to='profile_pics')

	def __str__(self):
		return f'{self.user.username} Profile'

	def save(self, **kwargs):
		super().save()

		#img = Image.open(self.image.path)
		img = Image.open(storage.open(self.image.name))

		if img.height > 300 or img.width > 300:
			output_size = (300, 300)
			img.thumbnail(output_size)
			img.save(self.image.name)

Unfortunately, when I attempt to update my profile image, I get:


AND notice this code near the bottom of the error:

 /app/users/signals.py in save_profile

    	instance.profile.save()

     ...

â–¶ Local vars
/app/users/models.py in save

    			img.save(self.image.name)

     ...

â–¶ Local vars
/app/.heroku/python/lib/python3.6/site-packages/PIL/Image.py in save

                    fp = builtins.open(filename, "w+b")

Just for context this is my settings.py:

import os

# Build paths inside the project like this: os.path.join(BASE_DIR, ...)
BASE_DIR = os.path.dirname(os.path.dirname(os.path.abspath(__file__)))


# Quick-start development settings - unsuitable for production
# See https://docs.djangoproject.com/en/2.1/howto/deployment/checklist/

# SECURITY WARNING: keep the secret key used in production secret!
SECRET_KEY = '*********'

# SECURITY WARNING: don't run with debug turned on in production!
DEBUG = True

ALLOWED_HOSTS = ['*']


# Application definition

INSTALLED_APPS = [
    'blog.apps.BlogConfig',
    'users.apps.UsersConfig',
    'crispy_forms',
    'django.contrib.admin',
    'django.contrib.auth',
    'django.contrib.contenttypes',
    'django.contrib.sessions',
    'django.contrib.messages',
    'django.contrib.staticfiles',
    'storages',
]

MIDDLEWARE = [
    'django.middleware.security.SecurityMiddleware',
    'django.contrib.sessions.middleware.SessionMiddleware',
    'django.middleware.common.CommonMiddleware',
    'django.middleware.csrf.CsrfViewMiddleware',
    'django.contrib.auth.middleware.AuthenticationMiddleware',
    'django.contrib.messages.middleware.MessageMiddleware',
    'django.middleware.clickjacking.XFrameOptionsMiddleware',
]

ROOT_URLCONF = 'django_blog_project.urls'

TEMPLATES = [
    {
        'BACKEND': 'django.template.backends.django.DjangoTemplates',
        'DIRS': [],
        'APP_DIRS': True,
        'OPTIONS': {
            'context_processors': [
                'django.template.context_processors.debug',
                'django.template.context_processors.request',
                'django.contrib.auth.context_processors.auth',
                'django.contrib.messages.context_processors.messages',
            ],
        },
    },
]

WSGI_APPLICATION = 'django_blog_project.wsgi.application'


# Database
# https://docs.djangoproject.com/en/2.1/ref/settings/#databases

DATABASES = {
    'default': {
        'ENGINE': 'django.db.backends.sqlite3',
        'NAME': os.path.join(BASE_DIR, 'db.sqlite3'),
    }
} 


# Password validation
# https://docs.djangoproject.com/en/2.1/ref/settings/#auth-password-validators

AUTH_PASSWORD_VALIDATORS = [
    {
        'NAME': 'django.contrib.auth.password_validation.UserAttributeSimilarityValidator',
    },
    {
        'NAME': 'django.contrib.auth.password_validation.MinimumLengthValidator',
    },
    {
        'NAME': 'django.contrib.auth.password_validation.CommonPasswordValidator',
    },
    {
        'NAME': 'django.contrib.auth.password_validation.NumericPasswordValidator',
    },
]


# Internationalization
# https://docs.djangoproject.com/en/2.1/topics/i18n/

LANGUAGE_CODE = 'en-us'

TIME_ZONE = 'UTC'

USE_I18N = True

USE_L10N = True

USE_TZ = True


# Static files (CSS, JavaScript, Images)
# https://docs.djangoproject.com/en/2.1/howto/static-files/


STATIC_URL = '/static/'
STATIC_ROOT = os.path.join(BASE_DIR, 'staticfiles')

MEDIA_ROOT = os.path.join(BASE_DIR, 'media') #location where media will be sent 
MEDIA_URL = '/media/' #accessing media from browser 


CRISPY_TEMPLATE_PACK = 'bootstrap4'

# Modified redirect locations for login 
LOGIN_REDIRECT_URL = 'blog-home'
LOGIN_URL = 'login'

EMAIL_BACKEND = 'django.core.mail.backends.smtp.EmailBackend'
EMAIL_HOST = 'smtp.gmail.com'
EMAIL_PORT = 587 
EMAIL_USE_TLS = True 
EMAIL_HOST_USER = os.environ.get('EMAIL_USER')
EMAIL_HOST_PASSWORD = os.environ.get('EMAIL_PASS')

AWS_S3_OBJECT_PARAMETERS = {
    'Expires': 'Thu, 31 Dec 2099 20:00:00 GMT',
    'CacheControl': 'max-age=94608000',
}

AWS_STORAGE_BUCKET_NAME = os.environ.get('S3_BUCKET')
AWS_S3_REGION_NAME = 'us-west-1'
AWS_ACCESS_KEY_ID = os.environ.get('AWS_ACCESS_KEY_ID')
AWS_SECRET_ACCESS_KEY = os.environ.get('AWS_SECRET_ACCESS_KEY')

AWS_S3_CUSTOM_DOMAIN = '%s.s3.amazonaws.com' % AWS_STORAGE_BUCKET_NAME

MEDIAFILES_LOCATION = 'media'
DEFAULT_FILE_STORAGE = 'custom_storages.MediaStorage'


import dj_database_url
db_from_env = dj_database_url.config()
DATABASES['default'].update(db_from_env)

AND this is my urls.py:

from django.contrib import admin
from django.contrib.auth import views as auth_views #anytime you import views specify the type of view 'as'
from django.urls import path, include 
from django.conf import settings
from django.conf.urls.static import static 
from users import views as user_views 

urlpatterns = [
    path('admin/', admin.site.urls),
    path('register/', user_views.register, name='register'),
    path('profile/', user_views.profile, name='profile'),
    path('login/', auth_views.LoginView.as_view(template_name='users/login.html'), name='login'),
    path('logout/', auth_views.LogoutView.as_view(template_name='users/logout.html'), name='logout'),
    path('password-reset/', auth_views.PasswordResetView.as_view(template_name='users/password_reset.html'), name='password_reset'),
    path('password-reset/done/', auth_views.PasswordResetDoneView.as_view(template_name='users/password_reset_done.html'), name='password_reset_done'),
    path('password-reset-confirm/<uidb64>/<token>/', auth_views.PasswordResetConfirmView.as_view(template_name='users/password_reset_confirm.html'), name='password_reset_confirm'),
    path('password-reset-complete/', auth_views.PasswordResetCompleteView.as_view(template_name='users/password_reset_complete.html'), name='password_reset_complete'),
    path('', include('blog.urls')), #empty string will default url to this urls 
    #checks for pattern, passes onto potential include with the rest of url pattern
] 


if settings.DEBUG:
    urlpatterns += static(settings.MEDIAFILES_LOCATION, document_root=settings.DEFAULT_FILE_STORAGE)

Lastly, this is my views.py:

from django.shortcuts import render, redirect
from django.contrib import messages 
from django.contrib.auth.decorators import login_required
from .forms import UserRegisterForm, UserUpdateForm, ProfileUpdateForm

def register(request):
	if request.method == 'POST':
		#UserCreationForm is a built in form from django
		form = UserRegisterForm(request.POST)
		if form.is_valid():
			form.save()
			username = form.cleaned_data.get('username')
			messages.success(request, f'Your account has been created! You are now able to log in!')
			return redirect('login') #return redirect allows browser to bypass 'are you sure you want to reload'
	else:
		form = UserRegisterForm()
	return render(request, 'users/register.html', {'form': form})

@login_required
def profile(request):
	if request.method == 'POST':
		u_form = UserUpdateForm(request.POST, instance=request.user)  #pass in an instance of user that form expects to alter 
		p_form = ProfileUpdateForm(request.POST, request.FILES, instance=request.user.profile)
		if u_form.is_valid() and p_form.is_valid():
			u_form.save()
			p_form.save()
			messages.success(request, f'Your account has been updated!')
			return redirect('profile') 
	else:
		u_form = UserUpdateForm(instance=request.user)
		p_form = ProfileUpdateForm(instance=request.user.profile)

	context = {
		'u_form': u_form, 
		'p_form': p_form
	}

	return render(request, 'users/profile.html', context)

SIDENOTE I also get this error when I am attempting to log into a profile that has an image stored on my s3 bucket. So there seems to be some issue with properly accessing my S3bucket images.

I looked through your above explanation but am not sure if it completely applies to my issue (if it does please let me know). Truly appreciate any help you can give me in troubleshooting this issue. Thank you!


#8

Hi, I think you’re saving to a local file instead of the S3 storage as you intend. Pillow doesn’t know about the S3 storage, so when you call img.save(self.image.name) it tries to save to a file using the name you pass it. You’ll need to use a BytesIO buffer as I showed in my previous answer.


#9

Thank you for the swift response, I am not so familiar with how I should implement BytesIO buffer in my code above. Would I only need to edit the models.py above? Without inconveniencing you too much, would you be able to show me an example based on my code? Very much appreciate your aid.


#10

I wanted to follow up by saying that I got rid of the error by removing the following section of my models.py file:

if img.height > 300 or img.width > 300:
			output_size = (300, 300)
			img.thumbnail(output_size)
			img.save(self.image.name)

As @bgarcial suggested, when he changed the condition, the FileNotFoundError was prevented (though in my case, completely removing the condition). Though I’m not sure why, img.save(self.image.name) was the root of the error.