Integrate AWS Transfer for SFTP With A Custom Identity Provider

Integrate AWS Transfer for SFTP With A Custom Identity Provider

February 05, 2019 by Volodymyr Rudyi, Artem Yefimenko

AWS Transfers for SFTP is a fully managed service that allows to easily upload/download data to/from AWS S3 using the SFTP protocol. In the previous blog post, we created a managed SFTP endpoint using the public key authentication. Sometimes, a username/password authentication may be required, e.g. if the access should be provided to existing users, e.g. from the AWS Cognito User Pool. In this blog post we are giving step-by-step instructions on how to implement a custom authentication for AWS Transfers for SFTP.

Deployment

We'll be using the Serverless framework to create corresponding infrastructure. We assume the AWS Cognito Userpool already exists to simulate a real-world scenario. The resulting CloudFormation stack contains:

  • API Gateway
  • AWS Lambda function which validates username/password supplied to the SFTP endpoint
  • A custom resource for the AWS Transfers for SFTP since at the moment the blog post was written, it was not present in the CloudFormation.

Here is a diagram with the resulting infrastructure:

Integration Infrastructure

Custom resource definition is provided below:

    SftpServer:
      Type: "Custom::SFTPServer"
      Version: "1.0"
      DependsOn:
        - CustomSftpServerManagerLambdaFunction
        - TransferIdentityProviderRole
        - TransferLoggingRole
        - ApiGatewayRestApi
        - SftpBucket
      Properties:
        ServiceToken: 
          Fn::GetAtt: [ "CustomSftpServerManagerLambdaFunction", "Arn" ]
        InvocationRole:
          Fn::GetAtt: [ "TransferIdentityProviderRole", "Arn" ]
        LoggingRole: 
           Fn::GetAtt: [ "TransferLoggingRole", "Arn" ]
        AuthorizeUrl:
          Fn::Join:
            - ""
            - 
              - https://
              - Ref: ApiGatewayRestApi
              - .execute-api.
              - Ref: 'AWS::Region'
              - .amazonaws.com/dev/

IAM Roles and Permissions

The Lambda execution role is rather standard:

    LambdaExecutionRole:
      Type: AWS::IAM::Role
      Properties:
        Path: /sftp/
        AssumeRolePolicyDocument:
          Version: '2012-10-17'
          Statement:
            - Effect: Allow
              Principal:
                Service:
                  - lambda.amazonaws.com
              Action: "sts:AssumeRole"
        ManagedPolicyArns:
          - arn:aws:iam::aws:policy/service-role/AWSLambdaBasicExecutionRole

For our SFTP endpoint we need to create two roles: TransferIdentityProviderRole(for API Gateway invocation) and TransferLoggingRole(for logging):

    TransferIdentityProviderRole:
      Type: AWS::IAM::Role
      Properties:
        Path: /sftp/
        AssumeRolePolicyDocument:
          Version: '2012-10-17'
          Statement:
            - Effect: Allow
              Principal:
                Service: 
                  - transfer.amazonaws.com
              Action: sts:AssumeRole
        Policies:
          - PolicyName: TransferCanInvokeAuthorizeApi
            PolicyDocument:
              Version: '2012-10-17'
              Statement:
                - Effect: Allow
                  Action:
                    - execute-api:Invoke
                  Resource:
                    Fn::Join:
                      - ":"
                      - 
                        - "arn"
                        - Ref: 'AWS::Partition'
                        - execute-api
                        - Ref: 'AWS::Region'
                        - Ref: 'AWS::AccountId'
                        - Fn::Join:
                            - ""
                            -
                              - Ref: ApiGatewayRestApi
                              - /${self:provider.stage}/GET/*
          - PolicyName: TransferCanReadAuthorizeApi
            PolicyDocument: 
              Version: '2012-10-17'
              Statement:
                - Effect: Allow
                  Action:
                    - apigateway:GET
                  Resource: "*"
    TransferLoggingRole:
      Type: AWS::IAM::Role
      Properties:
        Path: /sftp/
        AssumeRolePolicyDocument:
          Version: '2012-10-17'
          Statement:
            - Effect: Allow
              Principal:
                Service: 
                  - transfer.amazonaws.com
              Action: sts:AssumeRole
        ManagedPolicyArns:
          - arn:aws:iam::aws:policy/service-role/AWSTransferLoggingAccess

Another required role is the common SFTP user role. ARN of this role in the response from Lambda will indicate a successful authorization:

    SftpUserRole:
      Type: AWS::IAM::Role
      Properties:
        Path: /sftp/
        AssumeRolePolicyDocument:
          Version: '2012-10-17'
          Statement:
            - Effect: Allow
              Principal:
                Service:
                  - transfer.amazonaws.com
              Action: "sts:AssumeRole"
        Policies:
          - PolicyName: SftpBucketAccessPolicy
            PolicyDocument:
              Version: '2012-10-17'
              Statement:
                - Effect: Allow
                  Action:
                    - s3:*
                  Resource:
                    Fn::Join:
                      - ":"
                      - 
                        - "arn"
                        - Ref: 'AWS::Partition'
                        - "s3::"
                        - ${env:TRANSFER_BUCKET_NAME}*

Additionally, if you would like to use custom "SFTPServer" resource, you should define a role that allows managing the AWS Transfer for SFTP and also write logs to CloudWatch:

    LambdaSftpManageRole:
      Type: AWS::IAM::Role
      Properties:
        Path: /sftp/
        AssumeRolePolicyDocument:
          Version: '2012-10-17'
          Statement:
            - Effect: Allow
              Principal:
                Service:
                  - lambda.amazonaws.com
              Action: "sts:AssumeRole"
        Policies:
          - PolicyName: LambdaSftpManageRole
            PolicyDocument:
              Version: '2012-10-17'
              Statement:
                - Effect: Allow
                  Action:
                    - transfer:CreateServer
                    - transfer:UpdateServer
                    - transfer:DeleteServer
                    - logs:CreateLogGroup
                    - logs:CreateLogStream
                    - logs:PutLogEvents
                    - iam:PassRole
                    - apigateway:GET
                  Resource: "*"

IMPORTANT: iam:PassRole and apigateway:GET permissions are required, otherwise you will get an error if you try to create an SFTP Server.

Lambdas declaration

First, lets take a look at "SftpAuthorizer" lambda configuration:

  SftpAuthorizer:
    handler: src/handler.authorize
    role: LambdaExecutionRole
    environment:
      COGNITO_USER_POOL_ID: ${env:COGNITO_USER_POOL_ID}
      COGNITO_CLIENT_ID: ${env:COGNITO_CLIENT_ID}
      SERVER_ID: { "Fn::GetAtt": ["SftpServer", "ServerId" ] }
      BUCKET_ARN: { "Fn::GetAtt": ["SftpBucket", "Arn" ] }
      ROLE_ARN: { "Fn::GetAtt": ["SftpUserRole", "Arn" ] }
    events:
      - http:
          path: /servers/{serverId}/users/{user}/config
          method: GET
          authorizer: aws_iam

This lambda is being triggered by a request to AWS API Gateway and is used for authorization event handling. You should provide the following environment variables:

  • COGNITOUSER_POOL_ID and COGNITO_CLIENT_ID – AWS Cognito IDs
  • ROLE_ARN – an ARN of a common role for your SFTP users
  • SERVER_ID – a server id of our SFTP server (we will get it from the custom resource)
  • BUCKET_ARN – an ARN of an S3 bucket which you would like to serve to your users

IMPORTANT NOTE: Please note that you must specify /servers/{serverId}/users/{user}/config URL because it’s a path which AWS Transfers for SFTP uses by convention.

Also, you need to define the "SFTPServer" resource management lambda:

  CustomSftpServerManager:
    handler: src/handler.manageSftpServer
    role: LambdaSftpManageRole
    timeout: 180

Lambdas implementation

Now when we are done with the configuration, the last step is creating handlers for configured lambdas. First, let’s create the “authorize” function. This function will authorize users in the AWS Cognito using amazon-cognito-identity-js library. If the user was successfully authorized it should return the user's role ARN. Optionally, you can also provide Scope-Down Policy which allows restricting access for users based on HomeBucket and HomeDirectory variables, e.g. to define directories that will be available to the SFTP user. For this blog post, we created a policy which allows access only for bucket prefix that equals a username. If te user is unauthorized, the function should return a response with an empty body. An example of an authorized response code:

  const response = {
    headers: {
      "Access-Control-Allow-Origin": "*",
      "Content-Type": "application/json"
    },
    body: JSON.stringify({
      Role: process.env.ROLE_ARN,
      Policy: `{
        "Version": "2012-10-17",
        "Statement": [
            {
                "Sid": "AllowListingOfUserFolder",
                "Action": [
                    "s3:ListBucket"
                ],
                "Effect": "Allow",
                "Resource": [` +
                    '"arn:aws:s3:::${transfer:HomeBucket}"' +
                `],
                "Condition": {
                    "StringLike": {
                        "s3:prefix": [` +
                            '"${transfer:UserName}/*",' +
                            '"${transfer:UserName}"' +
                        `]
                    }
                }
            },
            {
                "Sid": "AWSTransferRequirements",
                "Effect": "Allow",
                "Action": [
                    "s3:ListAllMyBuckets",
                    "s3:GetBucketLocation"
                ],
                "Resource": "*"
            },
            {
                "Sid": "HomeDirObjectAccess",
                "Effect": "Allow",
                "Action": [
                    "s3:PutObject",
                    "s3:GetObject",
                    "s3:DeleteObjectVersion",
                    "s3:DeleteObject",
                    "s3:GetObjectVersion"
                ],` +
                '"Resource": "arn:aws:s3:::${transfer:HomeDirectory}*"' +
             `}
        ]
      }`,    
      HomeDirectory: `/${process.env.BUCKET_ARN.substring("arn:aws:s3:::".length)}/${username}/`,
      HomeBucket: process.env.BUCKET_ARN.substring("arn:aws:s3:::".length)
    }),
    statusCode: 200
  };

IMPORTANT NOTE: Policy parameter must be a string, it doesn’t work with an object type.

The second "manageSftpServer" handler is used for custom SFTPServer resource management. It handles create, update and delete events. There is also a utility "sendResponse" function which sends a response from the custom resource to the CloudFormation.

Summary

As you may noticed, AWS Transfer for SFTP with a custom identity provider configuration is not just "plug-and-play", but it's very flexible and can work with any identity provider.

Additionally, you can specify more conditions for the authorization, for example, grant access only for particular groups or for users, that have certain attribute values.

If you don't want to do all of this configuration by yourself, you can download our project from the GitHub using the link below, provide some environment variables and simply deploy your SFTP server with the "sls deploy" command.

Useful Links

Subscribe to our mailing list

* indicates required

Please select all the ways you would like to hear from AgileVision Sp. z o.o.:

You can unsubscribe at any time by clicking the link in the footer of our emails. For information about our privacy practices, please visit our website.

We use Mailchimp as our marketing platform. By clicking below to subscribe, you acknowledge that your information will be transferred to Mailchimp for processing. Learn more about Mailchimp's privacy practices here.

© 2016 - 2019 AgileVision sp. z o.o.