Maven Repository on Cloudflare R2 with Gradle

 Here's a challenge with the public availability of the R2 Cloudflare blob storage is it possible to use it as a Maven repository with Gradle as the build system?

The short answer is yes! The longer answer is that it is, but it is a bit disappointing with Gradle, I'll explain why it's disappointing later, but first (and probably the only bit you will read) is how.

Step 1: R2 storage

I'm assuming you have a Cloudflare account and have signed up for the R2 storage option.
  1. Navigate to the R2 Storage option on the R2 page
  2. Click 'Create bucket'

  3. Give the bucket a name, for the purpose of this tutorial I'm calling the bucket 'tutorial-r2-maven-bucket'


  4. Click create and you should have your new bucket all ready for use


  5. Make note of the URL for the bucket, you'll need this later

Step 2: S3 Auth token

The secret sauce with using R2 as a maven repository is to treat it as an S3 bucket and use it's S3 compatible API. To do this you next need an S3 Auth token  for the bucket.
  1. On the R2 Overview page, click the 'Manage R2 API Tokens' link on the top right


  2. Select 'Create API token'

  3. Fill in the details by changing the token name, and setting the permissions to edit as we want to publish to this bucket.


  4. Once created you will be shown the 'Access Key ID' and 'Secret Access Key'. This is the only page where you will be able to see them so ensure you copy them and keep them secure.


Step 3: Configuring Gradle

Now time to edit your Gradle file (I assume you haven one already). There are a number of tutorials that show how to configure the 'maven-publish' plugin to work with an S3 bucket, but in simple terms in the publishing section, configure the repositories as follows:

publishing {
    publications {
        mavenJava(MavenPublication) {
            from components.java
            artifact sourcesJar
            artifact javadocJar
        }
    }

    repositories {
        maven {
            url "s3://tutorial-r2-maven-bucket"
            credentials(AwsCredentials) {
                accessKey = "0000000000000000000000000000";
                secretKey = "aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa";
            }
        }
    }
}

  1. Set the 'url' field to be s3:// followed by your bucket name for example 's3://tutorial-r2-maven-bucket'
  2. Assign the 'Access Key ID' from earlier to the 'accessKey' field
  3. Assign the 'Secret Access Key' from earlier to the 'secretKey' field
IMPORTANT You should never hard code your secrets into build files as shown above.

Step 4: The secret sauce

The eagle eyed amongst you will have noticed that there is nothing in the configuration that tells Gradle to use your endpoint, and you would be right. The secret sauce is in the documentation about how to use the S3 API with a non AWS S3 compatible service. So you need to tell Gradle to use the URL from earlier (the one for the R2 bucket creation in Step 1.4) and unfortunately you have to do it on the command line with a -Dorg.gradle.s3.endpoint flag setting this to the root of the S3 bucket URL, so the command would look like:

./gradlew publish -Dorg.gradle.s3.endpoint=https://1234567890abcdef1234567890abcdef.r2.cloudflarestorage.com/

If you then check the bucket you will find the published artefacts, and that is it!

Why was this disappointing?

It's all Gradle's fault and the maven-publish plugin. It took quite a while to figure the above out mainly because the Gradle maven publish Aws S3 documentation is pretty poor and I eventually discovered what I was looking for on other website. I went down a blind alley of trying to inject in my own implementation to the publishing plugin either to override some of the limiting default behaviour, or with an idea of creating a native R2 implementation. Unfortunately the plugin is in no way extensible and has some pretty ugly limitations around hard coded assumptions between the configuration of the URL and the credentials. And that is why I came away disappointed from trying to use a tool (Gradle) that on the whole is very configurable:
  • Using the AWS S3 API with an alternate endpoint is pretty poorly documented, you can find it if you know what you're looking for but it's tough
  • Using a -D flag to signify an alternate URL is a cop out, this should be configurable as part of the DSL. This would have the side effect of improving the documentation
  • It selects the implementation based on the prefix of the URL in the url field ie.g. s3:// for an AWS S3 bucket or https:// for an HTTP endpoint. which means it doesn't understand that the http url for an R2 bucket should actually be for an S3 compatible API, even though you have told it to use AwsCredentials
  • If you use AwsCredentials with an HTTP url (or the other way round) then it just bombs out when you try to publish, not during the configuration phase
  • If you do try to use HTTP then there is no way to configure an AWS compatible headers for authentication, the HttpHeader credentials only allows specifying a single header.
  • You cannot change/extend the publishing implementation, nor the credentials implementation from those built in to Gradle, so as an example it's impossible to provide an R2 implementation that uses the R2 native API.
  • Finding the right Gradle source code is nearly impossible, only through deliberate misconfiguration and examining stacktraces did I find it.
I should point out that Cloudflare's documentation is spot on and wasn't the cause of my frustration.

Comments

Post a Comment

Popular posts from this blog

Gradle and the parent pom

We've all got used to various maven features that we have come to rely on, and in my effort to migrate to Gradle I have endeavoured to find alternative replacments, one of which is the parent POM functionality. Gradle does not natively have this feature that as Maven users we rely on. Workarounds There are a number of workarounds documented, firstly the most common is to use the apply from to load a file from a URI (you can use a URL) but this is not ideal for many reasons. The alternative is to write your own plugin that alters your build.gradle file. Anorakgirl has a great write-up on how to do this here  Gradle and the parent POM  and is the inspiration for my plugin. Gradle pater-build plugin I love Anorakgirls write up but I was thinking that it really is a workaround and devs would need to start to understand the internal Gradle API, so if the apply from syntax has its short comings as Anorakgirl pointed out then how about subverting or working around them. I created
Read more

Pushing changes from Shippable to Github

If you use Shippable as your Continuous Integration server on an open source project hosted on Github you will find that by default you cannot push any changes back to Github such as when you increment a version number and wish to push the changes back. This is because Shippable uses HTTPS for public Github repositories so trying to push changes back to it will require a username and password. Solution The solution is actually simple and requires two steps, firstly installing the the deployment key in your user or repository and then switching git over to ssh rather than https. Deployment key You can find your shippable deployment key on the right hand side of your account, this is a standard SSH public key, copy this key. You can either add this key to your repository or to your user (or another user if you so choose) that has access to your repositories. Be aware though that Github restricts the re-use of keys so if you add it to your repository then you can only have it
Read more

Goodbye Shippable Hello ... CircleCI

Migration number 1... CircleCI  how did it go and what did I think? Well it was pretty smooth, the free tier worked pretty well. The configuration for a Java project was smaller than what I ended up with with Shippable (I could remove some of the boiler plate setup). The builds were significantly faster than Shippable. Initial implementation was quick to do, I like the way it walked me through creating the first configuration. however trying to refine it into something more robust took a fair amount of time and that is where I came across some of it's shortcomings: Workflows are a bit confusing, and whilst it seemed like they would be ideal for different flows for building and publishing the artefacts this ended up not being the case and it would have been simpler just to use jobs at the top level if it wasn't wanting to inject 'secrets' into the build. Unable to define a global setup step, this looks like what an 'orb' would be used for but ended up just sourci
Read more