AWS S3 Connector

reshuffle-aws-connectors

npm install reshuffle-aws-connectors

Reshuffle AWS S3 Connector

This Reshuffle connector can be used to manage AWS S3 buckets and objects. Full details on the S3 API can be found here.

The following example creates an API endpoint to list all files in an S3 bucket:

const { HttpConnector, Reshuffle } = require('reshuffle')
const { AWSS3Connector } = require('reshuffle-aws-connectors')

const app = new Reshuffle()

const s3Connector = new AWSS3Connector(app, {
  accessKeyId: process.env.AWS_ACCESS_KEY_ID,
  secretAccessKey: process.env.AWS_SECRET_ACCESS_KEY,
  bucket: process.env.AWS_DEFAULT_BUCKET,
})

const httpConnector = new HttpConnector(app)

httpConnector.on({ method: 'GET', path: '/list' }, async (event) => {
  const keys = await s3Connector.listObjectKeys()
  event.res.json(keys)
})

app.start(8000)

Configuration Configuration options

Connector events:

bucketInitialized Bucket traching initialized

bucketChanged Bucket content changes

objectAdded Object added to bucket

objectModified Object modified in bucket

objectRemoved Object removed from bucket

Connector actions:

listBuckets Get a list of bucket info objects

listBucketNames Get a list of bucket names

createBucket Create a new bucket

deleteBucket Delete a bucket

listObjects Get a list of object info objects

listObjectKeys Get a list of object keys

copyObject Create a copy of an existing object

deleteObject Delete an object

getObject Get the contents of an object

putObject Create a new object

getSignedURL Get a signed URL for a single operation

getSignedObjectGetURL Get a signed download URL

getSignedObjectPutURL Get a signed upload URL

getS3URL Get an S3 object URL

getWebURL Get an web object URL

SDK:

sdk Get direct SDK access

Configuration options

const app = new Reshuffle()
const awsS3Connector = new AWSS3Connector(app, {
  accessKeyId: process.env.AWS_ACCESS_KEY_ID,
  secretAccessKey: process.env.AWS_SECRET_ACCESS_KEY,
  bucket: process.env.AWS_DEFAULT_BUCKET,
})

Connector events

Bucket Initialized event

Example:

async (objects) => {
  console.log(objects)
}

This event is fired when the connector starts tracking a specific S3 bucket. More technically, it is fired when the connector first reads the content of an S3 bucket and does not have a previous record of its object in its internal database.

For example, if a set of scripts is used to synchronize the contents of one bucket to another, this event can be used to read the contents of the target bucket and copy over the missing objects. This prevents the need to copy over every object in case of a database failure.

When this event is fired, neither the bucketChanged event nor the individual objectAdded events are fired for the same objects. Subsequent additions or modification to the tracked bucket will generate those events.

Bucket Changed event

Event parameters:

bucket: string - S3 bucket name

Handler inputs:

objects: object - Bucket state

Example:

async (objects) => {
  console.log('All keys:', Object.keys(objects).join(', '))
}

This event is triggered when one or more of the objects in an S3 buckets change: a new object is created, the content of an existing object is modified or an object is removed.

This event consolidates multiple changes. For each of this changes, the appropriate objectAdded, objectModified or objectRemoved is also fired. Those events are fired one per each object changed.

The objects argument has the following format:

{
  'key 1': {
    key: string, // equals to 'key 1' in this case
    lastModified: Date,
    eTag: '"..."' // 32 character hex string
    size: number // In bytes
  },
  'key 2': { ... },
  ...
}

Object Added event

Event parameters:

bucket: string - S3 bucket name

Handler inputs:

object: object - Object info

Example:

async (object) => {
  console.log('New object added:')
  console.log('  Key:', object.key)
  console.log('  Modified:', object.lastModified)
  console.log('  eTag:', object.eTag)
  console.log('  Size:', object.size, 'bytes')
}

This event is triggered once for each new object added to the bucket.

The object info argument has the following format:

{
  key: string,
  lastModified: Date,
  eTag: '"..."' // 32 character hex string
  size: number // Size in bytes
}

Object Modified event

Event parameters:

bucket: string - S3 bucket name

Handler inputs:

object: object - Object info

Example:

async (object) => {
  console.log('New object added:')
  console.log('  Key:', object.key)
  console.log('  Modified:', object.lastModified)
  console.log('  eTag:', object.eTag)
  console.log('  Size:', object.size, 'bytes')
}

This event is triggered once whenever the content of an object in the bucket is modified.

The object info argument has the following format:

{
  key: string,
  lastModified: Date,
  eTag: '"..."' // 32 character hex string
  size: number // Size in bytes
}

Object Removed event

Event parameters:

bucket: string - S3 bucket name

Handler inputs:

object: object - Object info

Example:

async (object) => {
  console.log('New object added:')
  console.log('  Key:', object.key)
  console.log('  Modified:', object.lastModified)
  console.log('  eTag:', object.eTag)
  console.log('  Size:', object.size, 'bytes')
}

This event is triggered once whenever an object is removed from the bucket.

The object info argument has the following format:

{
  key: string,
  lastModified: Date,
  eTag: '"..."' // 32 character hex string
  size: number // Size in bytes
}

Connector actions

List Buckets action

Definition:

() => object[]

Usage:

const buckets = await awsS3Connector.listBuckets()

Get a list of bucket information objects for accessible buckets.

List Bucket Names action

Definition:

() => string[]

Usage:

const names = await awsS3Connector.listBucketNames()

Get a list of accessible bucket names.

Create Bucket action

Definition:

(
  bucket: string,
  region?: string,
) => void

Usage:

await awsS3Connector.createBucket('my-bucket-name', 'us-west-1')

Create a new bucket.

Delete Bucket action

Definition:

(
  bucket: string,
) => void

Usage:

await awsS3Connector.deleteBucket('my-bucket-name')

Delete a bucket.

List Objects action

Definition:

(
  bucket: string,
) => object[]

Usage:

const objects = await awsS3Connector.listObjects('my-bucket-name')

Get a list of object information objects for objects in the specified bucket.

List Object Keys action

Definition:

(
  bucket: string,
) => string[]

Usage:

const keys = await awsS3Connector.listObjectKeys()

Get a list of object keys in the specified bucket.

Copy Object action

Definition:

(
  sourceBucket: string,
  sourceKey: string,
  targetBucket: string,
  targetKey: string,
) => object

Usage:

const result = await awsS3Connector.copyObject(
  'old-bucket',
  'original.jpg',
  'new-bucket',
  'copy.jpg',
)

Create a new copy of an existing object. Returns a copy result.

Delete Object action

Definition:

(
  bucket: string,
  key: string,
) => void

Usage:

await awsS3Connector.deleteObject('my-bucket-name', 'no-longer-needed.txt')

Delete an object from the specified bucket.

Get Object action

Definition:

(
  bucket: string,
  key: string,
) => object

Usage:

const info = await awsS3Connector.getObject('my-bucket-name', 'image.png')

Get information about an object, including its contents, as defined here.

Put Object action

Definition:

(
  bucket: string,
  key: string,
  buffer: Buffer,
) => object

Usage:

const info = await awsS3Connector.putObject(
  'my-bucket-name',
  'hello.txt',
  Buffer.from('Hello, world!'),
)

Returns information about the new object, as defined here.

Get Signed URL action

Definition:

(
  operation: string,
  key: string,
  expires?: number = 60,
) => string

Usage:

const url = await awsS3Connector.getSignedURL('getObject', 'me.png')

Get a pre-signed URL for a single operation. The URL can be used to access an object without requiring any credentials.

The URL is valid for a limited time, as specified by expires in seconds.

Get Signed Object Get URL action

Definition:

(
  key: string,
  expires?: number = 60,
) => string

Usage:

const url = await awsS3Connector.getSignedObjectGetURL('me.png')

Get a pre-signed URL for downloading an object. The URL can be used to download the content of an object without requiring any credentials.

The URL is valid for a limited time, as specified by expires in seconds.

Get Signed Object Put URL action

Definition:

(
  key: string,
  expires?: number = 60,
) => string

Usage:

const url = await awsS3Connector.getSignedObjectPutURL('you.png')

Get a pre-signed URL for uploading an object. The URL can be used with a PUT HTTP request to create a new object without requiring any credentials.

The URL is valid for a limited time, as specified by expires in seconds.

Get S3 URL action

Definition:

(
  key: string,
  bucket?: string,
) => string

Usage:

const url = await awsS3Connector.getS3URL('image.png')

Get an S3 object URL in the form "s3:////". If bucket is omitted then the valude from the connector options is used.

Get Web URL action

Definition:

(
  key: string,
  bucket?: string,
) => string

Usage:

const url = await awsS3Connector.getWebURL('image.png')

Get an HTTP URL for accessing objet key in bucket. If bucket is omitted then the valude from the connector options is used.

This URL is only valid if the specified S3 bucket is configured for web access. The action does not configure the bucket for web access nor does it validate that such access is enabled.

SDK

SDK action

Definition:

(
  options ?: object,
) => object

Usage:

const s3 = await awsS3Connector.sdk()

Get the underlying SDK object. You can specify additional options to override or add to the required fields in the connector's configuration.

Documentation

Introduction

Classes

Connectors

Deployment

AWS S3 Connector

reshuffle-aws-connectors

Reshuffle AWS S3 Connector

Table of Contents

Configuration options

Connector events

Bucket Initialized event

Bucket Changed event

Object Added event

Object Modified event

Object Removed event

Connector actions

List Buckets action

List Bucket Names action

Create Bucket action

Delete Bucket action

List Objects action

List Object Keys action

Copy Object action

Delete Object action

Get Object action

Put Object action

Get Signed URL action

Get Signed Object Get URL action

Get Signed Object Put URL action

Get S3 URL action

Get Web URL action

SDK

SDK action