Lullabot Ideas

We know stuff. We empower you to know stuff too.

Ubercart downloads via Amazon CloudFront

Article by David Burns

We recently started serving our Drupal training video downloads from Amazon CloudFront, a web service for content delivery, through the Lullabot Store. Because of this change we are pleased to say that the download speeds have been drastically improved. The process of integrating CloudFront into your own Ubercart store is fairly simple. However, there is a requirement that your version of PHP is compiled with OpenSSL. If OpenSSL is not enabled you will get errors. We have posted the Ubercart CloudFront module on Drupal.org so that it can be further improved.

Amazon CloudFront

Amazon CloudFront is a web service for content delivery. It integrates with other Amazon Web Services to give developers and businesses an easy way to distribute content to end users with low latency, high data transfer speeds, and no commitments.

Amazon CloudFront delivers your static and streaming content using a global network of edge locations. Requests for your objects are automatically routed to the nearest edge location, so content is delivered with the best possible performance. Amazon CloudFront works seamlessly with Amazon Simple Storage Service (Amazon S3) which durably stores the original, definitive versions of your files. Like other Amazon Web Services, there are no contracts or monthly commitments for using Amazon CloudFront – you pay only for as much or as little content as you actually deliver through the service.

Deciding to use CloudFront

The reason we decided to consider using CloudFront was that a very small but equally important segment of our friends were having troubles downloading these large files over slower connections. At times these slower connections were not due to the type of internet connection but rather the fact that they were trying to download these files from outside of the US. We needed to find a solution and fast.

Since we were already using S3, via the Media Mover module, to serve file downloads from Ubercart it was a simple decision to at least look at the benefits of using CloudFront. Amazon's hosting and file serving services are still quite young but are rapidly growing in features and popularity. Download speeds from any location and security were at the top of our list. Each of these features have been addressed with CloudFront's current configuration.

Setting up CloudFront

Setting up CloudFront is quite simple if you already use S3. It becomes even easier if you use a desktop app like S3Hub for OSX, CloudBerry for Windows, or firefox plugin like S3 Organizer. Each application handles this slightly different, but here are the basic steps.

  • Login to you S3 via the GUI app
  • Right Click the folder you want to setup a CloudFront distribution
  • Choose Manage Distributions
  • Provide a CNAME (if desired - must create this CNAME at your domain registrar and point it to your own xxxxx.cloudfront.net provided by Amazon)
  • Save & Wait for it to be created

Since S3Organizer is a plugin for Firefox it works for Windows, Mac, and Linux. Here's a good tutorial on getting this setup.

Now that CloudFront is setup it's time to configure your own Ubercart/Drupal site to access these files. Download and install the Ubercart CloudFront module then navigate to the Ubercart CloudFront settings page http://example.com/admin/store/settings/cloudfront. Here you will need to store the base URL (either the CNAME or xxxxx.cloudfront.net URL), along with paired SSH keys and KeyPairID all of which are provided by Amazon. On this page you may also set a default expire time to set how long these generated links will be valid.

That's it. From this point the module hooks into the Ubercart download link through a simple hook_file_transfer_alter(). Even if you enable this module on a live store all existing download links will still function, but they will now download the file through CloudFront. You will need to confirm that the files you are selling are currently stored in your S3 account.

Perfomance

Prior to setting this up we have heard from some of our friends that a 1.5GB file could take many hours. From our test on the live server we have consistently seen files this size take less than 20 minutes. This is an outstanding improvement that we now get to offer anyone who decides to checkout our excellent Drupal training videos.

Comments

Excellent news and

Excellent news and contribution. : )

Holy...

I think I might look into using CloudFront... Sounds excellent!

Thanks

Nice info, useful for me... :)

Amazon Web Services make it easy

Thank you for your contribution.

Amazon Web Services make many things easier. We are using it as well. Especially auto scaling to make sites ready for high traffic (peaks).

But also automated backups or software updates are made easy and quick.

We are not using CloudFront for now, but I guess that will change soon :)

Of course, AWS is not really the cheapest solution.

Amazon Web Services make it easy

Thank you for your contribution.

Amazon Web Services make many things easier. We are using it as well. Especially auto scaling to make sites ready for high traffic (peaks).

But also automated backups or software updates are made easy and quick.

We are not using CloudFront for now, but I guess that will change soon :)

Of course, AWS is not really the cheapest solution.

What kind of errors when not

What kind of errors when not using open SSL? for example SSL functionality is proxied and not installed on the actual web-servers when using the RackSpace Cloud and I assume other similar services.

Thanks

BK

Ubercart downloads via Amazon CloudFront

realy interesting) David, thanks!

GREAT Module but we need some insight

I'm excited about this module because we have a need for CloudFront support with our Ubercart store. I spent just about an entire day trying to get this to work without any success though. Because you guys are using it in your store... the module obvisouly works. I think we need some better documentation about how the module operates though.

For example, in reading about CloudFront and how it serves protected files it seems that a policy is needed to generate the signed URL. I don't see in the module where to upload a policy or does it do one in the background?

Also, I noticed it was generating the wrong URL for downloads and placing my sites URL infront of the CloudFront URL.

Example:

http://www.mysite.com/subdomain.cloudfront.com/dir/dir/myfile.mp3

Would you guys be so kind as to ellaborate on how the module works and how we successfully set it up?

Thanks!

ACL not implemented (yet)

First thing you'll want to resolve is

generating the wrong URL for downloads and placing my sites URL infront of the CloudFront URL.

This can be accomplished by including the full URL for your CloudFront CDN or the CNAME you provided for this bucket. For example our CNAME on the store is "http://dl.lullabot.com". This setting is located on https://your-domain.com/admin/store/settings/cloudfront

Even with this setting fixed I believe there's still a permission issue with setting the correct Access Control for CloudFront. There is an open issue for this http://drupal.org/node/804612 which I moved to critical status and currently working on.

For the time being I have been getting around this ACL issue by using CloudBerry Explorer (a Windows Application) and giving "READ" permission to the CloudFront user associated with our S3 account. It's a temporary workaround till ACL is working properly within this module.

Another Observation

Using Cloudberry, we put our files in our bucket with the exact same directory structure that we have on our server. The, we created the distribution. The I installed Media_Mover and uc_cloudfront. I then made a test purchase.

Trying to download a file from the Files tab in the test account gave a "page not found" error from our site because the URLs still had our our site and then the cloudfront url like this:

http://www.mysite.com/d21baubmeeh07j.cloudfront.net

and after the couldfront server it went straight to the "expires" variable and completely dropped the directory structure and file name.

However, when we use the Generate CloudFront URLs function from the Store administration menu, we get useable URLs.

Does the uc_cloudfront module correctly handle files in subdirectories?

Media_mover module stores the

Media_mover module stores the full path of files on S3, so subdirectories should work fine.

Looks like you are having similar issues to an earlier comment. Please refer to my response here.

Media Mover Participation

First of all, Dave, THANK YOU for your response. We're excited to get this module working.

The "Generate CloudFront URLs" is working as long as we don't have spaces in the files names. This brings me to my next question...

I don't really want Media Mover to handle the moving and placing of our files especially since the "Test S3" option in Media Mover will access your bucket, write a file, delete a file, and then DELETE the bucket! That seems like a bug to me and I need to report that today.

So, does CloudFront *depend* on the Media Mover files table?

Would you please elaborate on how CloudFront and MediaMover interact?

If you want to, drop in over at Ubercart.org where I have documented all the steps I've taken. Maybe you'll see something I missed?

http://www.ubercart.org/forum/support/16717/uc_cloudfront_anyone_have_it...

THANKS!!

how limit access to authenticated users

I understand the idea of using ubercart to sell individual files, be they videos or whatever. But what about ubercart only being used to establish an authenticated user, and then giving that authenticated user, post-login, unlimited access to all files hosted on the cloudfront network and linked from the drupal site?

We're running the Drupal book module to manage content, some of which is audio and video. We'd like to start using Cloudfront to host them. But I'm wanting to protect file access to just authenticated users.

Is uc_cloudfront the tool for that?