Python And Binary Repositories
Being a student I dreamed about something like Artipie, I needed some private centralized storage where I could have kept both binary or text data files and Python packages. We were conducting a research and used to work with a lot of data from experimental measurements and used Python to process and visualise them. My university colleagues from that time were mathematicians, they did not have strong development background and were absolutely fine with passing data and code via flash drive or email. All my efforts to inculcate VCS (GitHub, for example, provides free Pro accounts for students and professors) and some cloud storage usage were always postponed to the brighter future. I’ve quit the postgraduate program and never saw that future to come.
Now, I’d like to show how I could have used Artipie being that student. In the example I’ll use Artipie Central, but the same will work with an Artipie instance running on your private server in the private network.
Python Repository
Artipie supports PyPi repository which
twine
and pip
can perfectly
understand. This means that you can work with Artipie Python repository exactly as you do while
installing or publishing packages from or to PyPI
and TestPyPI repos. Here is my PyPi repository configuration from
Artipie Central:
repo:
type: pypi
storage: default
permissions:
olenagerasimova:
- download
- upload
"*":
- download
Upload is available for me only, but anyone can download any package, more details about permission configuration can be found here.
To make sure this repository exists and works, open index page from your browser, packages list should be displayed there.
I’ve created a small Python example project, source code is available here,
the main idea of the example is from my university times: download three data files from Artipie Central
(the next part is about how these files got there), read the numbers into arrays and
use these arrays to draw a plot. Let’s use pip
to install this example package and run it:
python3 -m pip install --index-url https://central.artipie.com/olenagerasimova/pypi/ pypiexample
python3 -m pypiexample
If you run these commands then you will see a polar plot with three curves, visualisation of our data files.
To publish package to the Artipie Central repository build it with
sdist
and use twine
for upload:
python setup.py sdist bdist_wheel
twine upload --repository-url https://central.artipie.com/olenagerasimova/pypi -u olenagerasimova -p *** dist/*
I’ve set up GitHub action to upload the example to Artipie Central on release, check the script here.
To sum up, in this chapter we set up PyPi repository in Artipie Central, created sample Python project, published and installed it. Now we’ll move on to the binary storage and I’ll show where data files for the example are stored.
Binary Storage
Artipie can be used as a storage for any files,
this repository is used to store experimental data which are visualised by the Python example. I’ve created
binary data
repository on Central, exact configuration is the following:
repo:
type: file
storage: default
permissions:
olenagerasimova:
- upload
- download
"*":
- download
As in the previous example I’ve allowed download operation to anyone. Data files are three text files with some numbers, here is how the first one looks like:
6
3.5
5
4
4.5
3
2.7
5
6
3
1.2
3.2
6
The other two files have the same form, only numbers are different. These files can be uploaded to
the Artipie Central data
repository with simple PUT
http request:
httpie -a olenagerasimova:*** PUT https://central.artipie.com/olenagerasimova/data/y1.dat @data/y1.dat
httpie -a olenagerasimova:*** PUT https://central.artipie.com/olenagerasimova/data/y2.dat @data/y2.dat
httpie -a olenagerasimova:*** PUT https://central.artipie.com/olenagerasimova/data/y3.dat @data/y3.dat
To download the files use GET
request
httpie -a https://central.artipie.com/olenagerasimova/data/y1.dat > ./data/y1.dat
or simply open links one, two and three in the browser. The Python example downloads these files and visualise them on the polar plot.
As this binary repository API is very simple (http PUT
and GET
requests), it’s easy to write
a piece of code in any language to upload and downloads necessary files. What is also important,
authorization is supported in Artipie, so it’s possible to grant rights to upload/download from
the repository how it suits your case.