Mike Slinn

Connoisseur of Technology

Publishing Maven Artifacts to AWS S3

2013-07-07 / All Blog posts

I wanted a simple, flexible and cheap way of publishing proprietary Maven artifacts created by sbt projects such that they could be securely retrieved by authorized individuals. I liked the idea of versioned artifacts, but did not want to use GitHub or BitBucket to host the artifacts because of the hassle of maintaining ever-larger git repos. Instead, I opted for S3's optional versioning mechanism.

The technique described here relies on the fact that publishing to a local file (using sbt publish) generates all the necessary artifacts, which merely need to be copied to the right directory on the artifactory server. The server need not be anything special: a normal web server works fine, as does webdav. A variety of other protocols are also supported by sbt.

I created two test projects on GitHub: testPublishLib and testPublishApp. You could clone them if you would like to try this. Each of them has a README that explains what they do.

I used s3cmd to manage the AWS S3 buckets that hold the repository. You can obtain s3cmd for most OSes. I found I needed to install python-magic:

$ sudo pip install python-magic
  1. Let s3cmd know your s3 keys:
    $ s3cmd --configure
  2. Create the S3 bucket, which must be unique. If you want to repository to be publicly visible, be sure that the bucket name starts with www. If you already have the S3 bucket then just omit this step.
    $ s3cmd mb s3://www.mymavenrepo
  3. If you want to repository to be publicly visible, you need to enable the S3 bucket web site option:
    $ s3cmd ws-create s3://www.mymavenrepo
  4. Publish your library to a local repository. testPublishLib is an example of how to set up a project properly.
    $ sbt publish
  5. Copy your locally published artifacts to S3. You can either type it out longhand:
    $ s3cmd -P sync ~/.ivy2/local/com/micronautics/test_publish_lib \
    s3://www.mymavenrepo/snapshots/com/micronautics/test_publish_lib
    ... (note that the -P option makes the files publicly visible), or you can use s3publish:
    $ s3publish com/micronautics/test_publish_lib
  6. If your repository is not public, you will need to provide authentication in a file which I called ~/.sbt/awsCreds.sbt for convenience:
    credentials += Credentials("AWS Realm", "www.mavenrepo.s3.amazonaws.com",
      "myUserId", "myPassword")
  7. Use the published artifact from your sbt project by including a resolver of the form:
    "AWS Snapshots" at "http://www.mavenrepo.s3.amazonaws.com/snapshots"
    The TestPublishApp project is a working example of how to do that.

Here is the script I wrote to upload to S3, which I called s3publish: It assumes you are always publishing a snapshot; and that the files should be public. I leave it to you to extend this script to handle releases and private content if you have the need.


Contact Mike Slinn

Unless you are a recruiter, in which case you should not try to make contact!

  • Email
  • Direct: 514-418-0156
  • Mobile: 650-678-2285

Disclaimer

The content on this web site is provided for general information purposes only and does not constitute legal or other professional advice or an opinion of any kind. Users of this web site are advised to seek specific legal advice by contacting their own legal counsel regarding any specific legal issues. Michael Slinn does not warrant or guarantee the quality, accuracy or completeness of any information on this web site. The articles published on this web site are current as of their original date of publication, but should not be relied upon as accurate, timely or fit for any particular purpose.

Accessing or using this web site does not create a client relationship. Although your use of the web site may facilitate access to or communications with Michael Slinn via e-mail or otherwise via the web site, receipt of any such communications or transmissions does not create a client relationship. Michael Slinn does not guarantee the security or confidentiality of any communications made by e-mail or otherwise through this web site.

This web site may contain links to third party web sites. Monitoring the vast information disseminated and accessible through those links is beyond Michael Slinn's resources and he does not attempt to do so. Links are provided for convenience only and Michael Slinn does not endorse the information contained in linked web sites nor guarantee its accuracy, timeliness or fitness for a particular purpose.


comments powered by Disqus

© 1976-2020, Michael Slinn. All rights reserved.