Getting Uploaded Data *Out* of AWS S3
OK, so you know how to get data into AWS S3, what about getting it out? Previously, we uploaded entries from an imagined photo contest into a bucket. We sent a pair of files, a JSON file with the form data and the image. Let’s presume there’s a Rails app, it’s details don’t matter, but it has a model ContestEntry and we want to populate it from the S3 data. We’re going to write a script to do the import.
That One Stupid dd Trick and the Ballad of SIGINFO
dd
is the *NIX byte copying utility. It’s typically used for copying
disks, creating disk images, or initializing disks from images. It can
also be using to recover damaged files that can’t otherwise be
copied. I mostly use it to make create bootable USB sticks for server
installs. However, it’s also pretty opaque.
Uploading Form Data Directly to AWS S3
A couple of posts back, I walked through uploading an image to AWS S3 without the need for a server of your own. This has the advantage of being a fully standalone, browser client only solution. Of course images by themselves aren’t very useful. Likely we want to collect some additional information.
Direct to S3 Uploads With AJAX Presigning
Previously, I covered uploading to S3 from a Rails app using a presigned-url. This works just fine, but means the data flows from the visitors computer to your server before heading to S3. I’ve previously showed how to upload the file directly to S3, which requires a world-writable bucket.
There’s also a hybrid solution that has the server generate a presigned-url which the browser then uses to go direct to S3, allowing the data to bypass the server. Let’s look at two ways to do this.
Uploading From Rails to AWS S3 With PreSigned URLs
Last time, I walked through directly uploading an image to an S3 bucket from the browser with no server involved. That last bit is important. If you are using a server, then PreSigned URLs are a better choice as they do not require a world-writable bucket.
Uploading Images Directly From the Browser to AWS S3
Want to upload an image directly to S3 from the browser? Here’s what you need.
Advanced Ruby IMAP
Previously, I worked through how to get messages from an IMAP server and work with the message headers. Let’s look at extracting data from those messages.
Ruby and IMAP
There’s a really solid, tried and true, Internet wide messaging queue. It’s been around for decades and has first class support in Ruby. What is it? Email. That’s right, email.
Bullet Journal
Here’s something slightly random and the exact opposite of technical, the Bullet Journal.
I’ve tried just every to-do/productivity app/site out there. OmniFocus, Things, Trello, Evernote, Org Mode to name five. But, I never stick to them. They start of well enough, yet end up cluttered with things I’m never going to do. Or things I sort of did. Or things that are too nebulous to be meaningful.
Stupid ssh-add Tricks
If you are reading my blog, odd are you already know how to use
ssh-add
to manage you SSH keys. If not,
you can read up on it.
Up to speed? ssh-add
has a few other features that people,
including me, forget about. Let’s take a look.