tsetup.md - amprolla - devuan's apt repo merger
(HTM) git clone https://git.parazyd.org/amprolla
(DIR) Log
(DIR) Files
(DIR) Refs
(DIR) README
(DIR) LICENSE
---
tsetup.md (5347B)
---
1 amprolla setup
2 ==============
3
4 amprolla should be able to run on any system supporting Python 3, rsync
5 and GnuPG. However it has only been tested on Devuan and Gentoo.
6
7
8 Installation
9 ------------
10
11 The recommended way to install Python 3 and the needded modules, along
12 with the extra needed dependencies is using your package manager.
13
14 You will need the following:
15
16 ```
17 python3, python-requests, gnupg2, rsync
18 ```
19
20 After installing the required dependencies, clone the amprolla git repo
21 using git to a place of your preference.
22
23 You will also need to setup a valid gnupg directory structure, along with
24 a key you shall use for signing the Release files if you require to do so.
25
26
27 Configuration
28 -------------
29
30 To configure amprolla, a default configuration file is provided in
31 `lib/config.def.py`. Copy the file to `lib/config.py` and edit it to
32 your needs. The configuration file contains all the information needed
33 to properly merge the required repositories. The default configuration
34 is also working as long as you provide a valid gpg fingerprint used to
35 sign the Release files. If you do not wish to sign Release file, make
36 sure to disable it in the configuration file.
37
38 The `*dir` variables in the configuration file are the directories
39 where the files that are being merged are kept and the merges itself
40 are done. They can be either absolute or relative paths to the root
41 amprolla directory. The prefered way is to actually have absolute paths
42 as this will cause less trouble.
43
44 `banpkgs` is a set of package names that amprolla will refuse to merge
45 if they are found either in the dependencies of a package or if they
46 are the package itself.
47
48 `repo_order` is a list that holds that is ordered in the priority the
49 packages are prefered. The preference is ordered first to last.
50 The dict `repos` itself holds the required information for them.
51
52 ### repos dict structure
53
54 * The variable is a normal dict where the key is a string that should
55 be contained in `repo_order`. They key's values are another dict with
56 the following fields:
57
58 - `name`: the name of the repository used for rewriting the path to
59 the package's upstream (deb file). It is used in the nginx config
60 provided in the `contrib` directory.
61
62 - `host`: the upstream URL where the repository is being held. Used
63 to know from where to download the necessary Packages/Release files
64 and how to rewrite certain values.
65
66 - `dists`: the root directory where the suites are held. It is appended
67 to the above URL.
68
69 - `aliases`: if True, when downloading/rewriting, will look for the
70 suite's alias defined later in the configuration file.
71
72 - `skipmissing`: a hack helpful to skip missing suites if the repo
73 we are merging does not contain them.
74
75
76 `suites` hold our release names and their suites.
77
78 `aliases` hold the suites we want to merge as aliases in case we know a
79 certain repository doesn't use the same name for it.
80
81 `release_aliases` list our stable and testing branch aliases.
82
83 `categories` hold the package categories we wish to actually merge. apt
84 repositories usually hold `main`, `contrib`, and `non-free`. With this
85 we can opt out of any of them.
86
87 `arches` list the actual architectures we want to merge. If we are not
88 using certain architectures, it is easy to exclude them from merging
89 this way.
90
91 It is advised to not touch any variables listed below these, as they
92 are currently setup to provide a correct (valid) apt repository.
93
94
95 Running amprolla
96 ----------------
97
98 After you've setup amprolla, it is needed to perform the initial full
99 download and full merge. First run `amprolla_init.py`, which is going
100 to download the necessary directory structures (as defined through the
101 config file) we will merge afterwards. When the download is done, it is
102 time to perform the full initial merge of these repositories. This will
103 provide us with a complete merged repository and we will then be able
104 to easily perform incremental updates of it.
105
106 After the first merge has been performed, it is advisable to run a
107 script called `populate_aliases.sh` found in the `contrib` directory.
108 Make sure you edit it and set a proper path to your merged directory,
109 and fill out the proper names needed. It will populate the merged
110 directory with symlinks to certain versions such as `1.0`, `stable`, or
111 `testing`.
112
113 To merge Contents files, run `amprolla_merge_contents.py`. This module
114 does not do incremental updates and should not be ran often due to its
115 heavy IO/RAM requirements.
116
117 Incremental updates are performed through `amprolla_update.py`, however,
118 for more stable performance and uptime, the incremental updating is
119 being orchestrated by a shell script called `orchestrate.sh`. This shell
120 script holds the logic to have near-atomic switching of repositories to
121 minimize repo downtime during performed merges. Not doing this could
122 result with users downloading corrupted repository files if they are
123 requested during an ongoing merge.
124
125 In a screen/tmux session, simply execute the `orchestrate.sh` script
126 and it will start looping and doing incremental updates every hour.
127 If you prefer, you can remove this loop and run the shell script through
128 a cron job based on your needs.
129
130 To actually serve the merged directory over HTTP, a basic nginx
131 configuration is provided as `contrib/nginx.conf`, and a lighttpd conf
132 is provided in `contrib/lighttpd.conf`.