0.17: Release Notes
v0.17 was released on March 17. Several important features were implemented in Debian-adapter:
- GPG clearsign signature is now supported
by the adapter, meaning that
Release
index files are now signed with gpg; - Duplicated packages handling: while updating
Packages
index we now check that the package to add is not present in the repository yet. If it does, the old package and metadata are removed and replaced with the new one; - Debian-repository now provides some
sdk
and can be used as a dependency for the debian repository batch-update and indexes generation. For more details checkDebian
interface and itsDebian.Asto
implementation.
We started to restructure Artipie integration tests,
changes are mostly infrastructural: now there are two Docker containers started on each integration test,
Artipie docker image built on package state and client image. Client image differs depending on the
repository we are testing, it could be maven:3.6.3-jdk-11
for maven
repository tests
or centos:centos8
to verify rpm
repository. Thus, our tests work with the image we provide for
our users guaranteeing its efficiency and credibility.
Here is the list of other implemented features and important fixes:
- Helm-adapter now support index merging
- Composer-adapter supports uploading packages via
PUT
HTTP request, also we start implementing remote repository support for composer - Bug with incorrect paths in metadata was fixed in npm-adapter
0.16: Release Notes
v0.16 was released on February 16. Here is the list of other implemented features and important fixes:
- Docker-adapter now supports overwrite permission for tags
- Release index file is now fully supported in Debian-adapter
FileStorage
delete
operation was improved: now we also remove empty directories recursively along with the removed file. This change fixed connected bug in Docker-adapter- Basic authentication was added to Composer-adapter
- Writing big files operation performance was improved in Asto
- Npm-adapter now supports
tgz
upload, to publish the package simplecurl
call can be used:curl PUT your-archive.tgz
. Also,unpublish
command now can remove the specified version of the package, not only the package entirely.
Granular Permissions For Docker Registry
Docker-adapter supports granular permissions, which means that some operations can be granted for specific scope and image. Here is the example docker repository settings yaml:
repo:
type: docker
storage:
type: fs
path: /var/artipie/data
permissions:
alice:
- repository:my-alpine:*
- repository:ubuntu-latest:pull
- repository:some-image:overwrite
- repository:some-image:push
mike:
- repository:some-image:push
- repository:some-image:pull
bob:
- registry:base:*
- registry:catalog:*
Each permission consists of three parts: scope:name:action
, scope
can be either “registry” for
the registry level operations or “repository”. Name
is the access resource name, action
can be
one of pull
, push
, overwrite
or *
(*
stands for any action), overwrite
works
for overwriting existing tags and allows creating new tags by default.
0.15: Release Notes
v0.15 was released on January 29.
We are happy to announce that minimal support for Debian repository
was added to Artipie: you can configure the repository
which apt-get
will be able to work with.
Npm-adapter functionality was extended with the following commands support:
npm dist-tag
for managing tags of package versionsnpm deprecate
for deprecating and un-deprecating the packagenpm unpublish
to un-publish entire package
Docker-adapter evolution:
- In practice, it seems that blobs are always uploaded in single chunk, that’s why we optimized this particular upload case
- Image level permissions are now supported, meaning, that
you can define which user has
read
orwrite
access to particular image
And here is the list of other implemented features:
- Helm-adapter now supports
deleteByName
anddeleteByNameAndVersion
commands and--merge
flag for indexing - Php-adapter became fully non-blocking
- Nuget-adapter endpoints were refactored not to interact with data storage directly
0.14: Release Notes
v0.14 was released on December 30. This month our two main directions were Docker-adapter and management-api transfer. Features added to Docker-adapter:
- Local catalogs and tags listing
- Support for mount blob endpoint in remote and local repositories
- Catalogs and tags support was added for local and proxy repositories
Artipie API and dashboard were transferred to management-api repository and added minimal integration tests for API endpoints in Artipie.
Here is the list of other changes:
- Artipie now supports both .yml and .yaml extensions for configs
- Proper path handling in Npm-Proxy was implemented
- Invalid repository configurations are now properly handled on start, we also added separate setting for repo configs location
- Basic authentication support was added to Gem-adapter
We also started to develop Debian-adapter to support Debian repository.
0.13: Release Notes
v0.13 was released on November 27. Here is the list of the key changes:
-
rpm-adapter: we introduce an instrument to properly verify correctness of
repomd.xml
, any errors with metadataxmls
are properly handled now, added subdirectories support in RPM repository, improved test coverage and finalize incremental update feature -
docker-adapter was migrated to Java 8, configurable Docker storage layout was added to the adapter
-
nuget-adapter became non-blocking
-
management-api repository was created to contain all Artipie API and dashboard related classes
-
helm-adapter: we added verification for
index.yaml
-
pypi-adapter now supports names normalization in proxy
-
We implemented minimal support for Bearer authentication
-
Integration tests for Ruby Gems and Helm were implemented, thus all repository types are now covered with end-to-end tests
Python And Binary Repositories
Being a student I dreamed about something like Artipie, I needed some private centralized storage where I could have kept both binary or text data files and Python packages. We were conducting a research and used to work with a lot of data from experimental measurements and used Python to process and visualise them. My university colleagues from that time were mathematicians, they did not have strong development background and were absolutely fine with passing data and code via flash drive or email. All my efforts to inculcate VCS (GitHub, for example, provides free Pro accounts for students and professors) and some cloud storage usage were always postponed to the brighter future. I’ve quit the postgraduate program and never saw that future to come.
Now, I’d like to show how I could have used Artipie being that student. In the example I’ll use Artipie Central, but the same will work with an Artipie instance running on your private server in the private network.
Python Repository
Artipie supports PyPi repository which
twine
and pip
can perfectly
understand. This means that you can work with Artipie Python repository exactly as you do while
installing or publishing packages from or to PyPI
and TestPyPI repos. Here is my PyPi repository configuration from
Artipie Central:
repo:
type: pypi
storage: default
permissions:
olenagerasimova:
- download
- upload
"*":
- download
Upload is available for me only, but anyone can download any package, more details about permission configuration can be found here.
To make sure this repository exists and works, open index page from your browser, packages list should be displayed there.
I’ve created a small Python example project, source code is available here,
the main idea of the example is from my university times: download three data files from Artipie Central
(the next part is about how these files got there), read the numbers into arrays and
use these arrays to draw a plot. Let’s use pip
to install this example package and run it:
python3 -m pip install --index-url https://central.artipie.com/olenagerasimova/pypi/ pypiexample
python3 -m pypiexample
If you run these commands then you will see a polar plot with three curves, visualisation of our data files.
To publish package to the Artipie Central repository build it with
sdist
and use twine
for upload:
python setup.py sdist bdist_wheel
twine upload --repository-url https://central.artipie.com/olenagerasimova/pypi -u olenagerasimova -p *** dist/*
I’ve set up GitHub action to upload the example to Artipie Central on release, check the script here.
To sum up, in this chapter we set up PyPi repository in Artipie Central, created sample Python project, published and installed it. Now we’ll move on to the binary storage and I’ll show where data files for the example are stored.
Binary Storage
Artipie can be used as a storage for any files,
this repository is used to store experimental data which are visualised by the Python example. I’ve created
binary data
repository on Central, exact configuration is the following:
repo:
type: file
storage: default
permissions:
olenagerasimova:
- upload
- download
"*":
- download
As in the previous example I’ve allowed download operation to anyone. Data files are three text files with some numbers, here is how the first one looks like:
6
3.5
5
4
4.5
3
2.7
5
6
3
1.2
3.2
6
The other two files have the same form, only numbers are different. These files can be uploaded to
the Artipie Central data
repository with simple PUT
http request:
httpie -a olenagerasimova:*** PUT https://central.artipie.com/olenagerasimova/data/y1.dat @data/y1.dat
httpie -a olenagerasimova:*** PUT https://central.artipie.com/olenagerasimova/data/y2.dat @data/y2.dat
httpie -a olenagerasimova:*** PUT https://central.artipie.com/olenagerasimova/data/y3.dat @data/y3.dat
To download the files use GET
request
httpie -a https://central.artipie.com/olenagerasimova/data/y1.dat > ./data/y1.dat
or simply open links one, two and three in the browser. The Python example downloads these files and visualise them on the polar plot.
As this binary repository API is very simple (http PUT
and GET
requests), it’s easy to write
a piece of code in any language to upload and downloads necessary files. What is also important,
authorization is supported in Artipie, so it’s possible to grant rights to upload/download from
the repository how it suits your case.
Using Artipie as a Maven repository
As we know, Apache Maven is one of the most popular build management tool for Java. Many companies that
do Java development have private Maven repositories, where they store all the artifacts. There are some
problems with management when many developers work with a huge repository.
Let’s assume, you need to restrict access to artifacts. So, it’s necessary to split this one into
several small repositories or add authentication for users. But at the same time you want to keep the
ability to share easily and quickly within a team or between multiple teams of developers. Or you are
just wondering how to set up an private repository for a pet project.
Artipie allows you to solve these problems as it is a binary
artifact management tool. In the blog post I will share with you how to configure Artipie and project
for this purpose. Let’s start!
In the simplest case, you need a file with the contents below. We want to use file storage therefore we
specify fs
as the type. At first, create a separate directory with the following artipie.yaml
file
inside:
meta:
storage:
type: fs
path: /var/artipie/configs
After that you can install and run Artipie with Docker:
$ docker run -d --name artipie \
-v $(pwd)/artipie.yaml:/etc/artipie.yml \
-v $(pwd):/var/artipie \
-p 8080:80 \
artipie/artipie:latest
In the previous step, we created the settings file for Artipie. Now let’s see what the configuration
file for the repository should look like. For this, we need to create configs
directory and the
following my-maven.yaml
file inside:
repo:
type: maven
storage:
type: fs
path: /var/artipie/data
As a results, you will have the following files structure:
.
├── artipie.yaml
├── configs
└── my-maven.yaml
At this step, I will show you how to configure your project to use created Artipie Maven repository. If
you want to mvn deploy
your project to Artipie, add the following to project’s pom.xml
:
<distributionManagement>
<repository>
<id>artipie</id>
<url>http://localhost:8080/my-maven</url>
</repository>
</distributionManagement>
In case you want to mvn install
deployed packages from Artipie then add the following to project’s
pom.xml
:
<repositories>
<repository>
<id>artipie</id>
<url>http://localhost:8080/my-maven</url>
</repository>
</repositories>
If you have completed all the instructions correctly now you should be able to use the repository: deploy and install Maven artifacts.
Configure Artipie with authentication
If you want to use authentication for users then you need to add a section with credentials to the
artipie.yaml
. In this case, the file with settings should look like this:
meta:
storage:
type: fs
path: /var/artipie/configs
credentials:
type: file
path: _credentials.yml
Also, it’s necessary to create _credentials.yml
file in the configs
directory. Each node is a
user name. The supported types of password can be found here
The file with credentials should look like this:
credentials:
alice:
type: plain
pass: 123
bob:
type: plain
pass: qwerty
As a results, you will have the following files structure:
.
├── artipie.yaml
├── configs
├── my-maven.yaml
└── _credentials.yml
Don’t forget to run Artipie server with Docker and to update pom.xml
of your project as shown above.
If all the steps were done correctly you should be able to use the repository with authentication:
deploy and install Maven artifacts.
Apart from demonstrated possibilities, Artipie Maven supports: group repositories, proxies and user groups. Check out the corresponding documentation section for more details.
In this blog post I explained how to set up private Maven repository with Artipie and how to configure a Maven project to use it. I hope now it will be easier and less time-consuming to manage the repository by using Artipie.
0.12: Release Notes
v0.12 was released on October 29. In October, we focused on extending and adding proxy for our adapters. First, we implemented basic and bearer authentication in our http-client, then supported authentication for proxy in different adapters. Here is the list of the proxy adapters, authentication and caching in the local storage can be configured for each proxy:
- docker-proxy
(also supports
push
operation to the local storage) - maven-proxy (multiple remotes are supported)
- npm-proxy
- pypi-proxy
- files-proxy
We also worked on another features in Artipie:
- Artipie is now compatible with
pip search
command - field
time
is supported now on update in npm-adapter - we introduced different local storage cache mechanisms in asto, these implementations are used in all our proxy-adapters
- authentication mechanisms in http was simplified and improved
- integration tests were implemented in artipie for maven-proxy, NuGet, npm and pypi-proxy