--- author: email: mail@petermolnar.net image: https://petermolnar.net/favicon.jpg name: Peter Molnar url: https://petermolnar.net/ copies: - http://web.archive.org/web/20210125222811/https://petermolnar.net/article/xmpp/ lang: en published: '2021-01-25T19:00:00+00:00' summary: 'Fleeing Whatapp due to their policy changes? Tired of the never ending roam between the actual trendy instant messaging app? Here is a simple, elegant solution: run your own server, onboard your friends and family, and live happily ever after.' tags: - server title: Run your artisan instant messaging service for your friends & family --- ## Chat systems should be able to talk across one another, why are they not? You probably have used email before. I have multiple email addresses, among them and . It's perfectly normal that I will be able to send a mail from one another, even though these email addresses are on different domain (example.net vs gmail.com), different servers, and different are the people running them. There's a similar case with SMS: nowadays I expect that I'd be able to send a text to anyone in the world who's number I know, and if I'm using the international number format correctly. +44 (UK) to +36 (Hungary), or +49 (Germany) to +1 (USA) - it should all work. *I have to admit, I don't know if sending a text to mainland China works; I never tried, but I'd believe it does.* When it comes to instant messages, this was never the case, even though it has been possible for a long time - just disabled by most providers. Around 23 years ago I launched ICQ 98, and the people I could interact with were the people on ICQ. If they were on AIM, I'd need to launch AIM. Later MSN, Skype, Gtalk came onto play, all before mobile took off. Somewhere down the road fantastic developers had enough of needing to use many software for the same purpose, and thus multi-protocol instant messengers happened, notably: Trillian[^1], Pidgin[^2], Miranda[^3]. Since then Trillian became yet another instant messaging protocol *(et tu, Brute?!)*, Miranda got heavily neglected. Pidgin, or, more specifically, libpurple, the library behind it, is doing acceptably well, and can be made to connect to "modern" walled garden services, such as Discord, Slack, Signal, or even Whatsapp[^4] ![This is how my Pidgin, my multi-protocol application looked in 2020. Wouldn't it be simpler to have only one for work, and another for personal?](pidgin-2020.png) When smartphones started to become widely available, the messenger space grew exponentially, fuelled by "free" SMS and calls. Now we have more instant messengers, then ever, and they can only, and strictly only be used within the same system, many are mobile "exclusive": Whatsapp, Viber, Telegram, Signal, new ICQ, Google ~~Talk~~ ~~Hangouts~~ ~~Allo~~ ~~Duo~~ ~~Meet~~ Chat *(I think it's called Chat at the moment)*, Threema, Facebook Messenger, QQ, WeChat - the list is endless. There was a brief, interesting period of time, namely when Facebook introduced their chat, Gtalk was still called Gtalk, and Whatsapp was still just starting: they all used the very same underlying standard, called XMPP[^5]. XMPP is not an app; it's more like HTTP and HTML; a wire frame, so people can build their own apps on top of it. It has a feature that sadly none of the three mentioned service provided, called federation: it means that just like email, it should have been possible to talk from your Whatsapp with someone on Gtalk. Since then, all three of them abandoned XMPP, and made different, incompatible with each other systems. This feature then was disabled (EDIT: with the exception of Google Talk in 2006, that had federation until it's demise[^6]); probably to train people to associate an app with the service behind it. Many of us know that the internet can be accessed with many browsers - Edge, Internet Explorer, Safari, Chrome, Firefox. Much less know that nearly ever email service, including Gmail, or Outlook, can also be accessed with different programs, like Apple Mail[^7], or K9[^8] on android. With instant messengers, using a 3rd party client sometimes even leads to instant ban. If you think of it, it's completely insane. ### What can be done to address the fragmented nature of instant messaging? The answer is simple, but the execution is not at all, and it needs effort from all of us. 1. We need to return to smaller service providers. Find a firm where you'll know real people, where you'll know the faces to turn to if you have a problem. There are so many stories about people locked out from their accounts on Facebook without any option to appeal that it's frightening[^9]. These days places like is a good start for such provider. 2. Use a provider which allows your account to talk to other accounts on the same network - it's called federation. (See below) 3. Choose an app you like (see later), not one someone wants to force on you. 4. If you can, run your own service, and do it for friends, and family as well. ### What are the networks capable of talking to each other? At the moment there are two main messaging ecosystems which can do federation and accepts many clients: XMPP and Matrix. They cannot (yet?) talk to one another. There are fundamental philosophical differences in their approach, which can be summarised as: - and XMPP server is nothing, but a relay station. It barely stores any data, it's purpose is to allow the data to flow through it to the clients, and lets the client decide what to do. - a Matrix server, on the other hand, stores as much of this data on the server side as possible, so the clients never have to worry about losing data from their devices. I prefer the approach of XMPP. ## About modern XMPP clients One of the beautiful benefits of using a service that allows, sometimes even encourages, using the application you prefer, is that we can decide based on our needs and likes. There have been many critical voices aimed at XMPP, mainly like "too late to add end-to-end encryption". This is a complex question, because in fact XMPP was one of the first things to offer encryped messaging - PGP based -, but it was very difficult to use. A more modern approach, borrowed from Signal, arrived later, and it's called OMEMO. There are now many client supporting OMEMO encryption[^10], providing basically transparent, secure chat to many. Another feature that many waited a long time was voice and video chat - but this, again, has a similar story to encryption. Jingle[^11] is a very old addition to the XMPP standard, so old, that it doesn't work with Pidgin any more - the relevant Pidgin part was built around a media library that has been dead for over a decade, namely gstreamer-0.1 -, but on mobile clients, this feature was missing for a long time. A notable exception is AstraChat[^12] which offered it for quite a while, but in return, it still doesn't offer e2e encryption. The point is: in 2020, XMPP clients on mobile or desktop are up to all needs, all requirements, and are certainly much easier to use, then they used to be even just a few years ago. ### The clients I'd recommend these days - **Conversations** (or a derivative of it) for Android - - £2.19 - - free - **aTalk** for Android - - - **Siskin** for iOS: - - **Dino** or **Gajim** for desktop: - - *With a heavy heart, I'm not putting Pidgin on that list, because it's basically impossible to get audio/video support for XMPP on it, and while I'll remain a heavy user of it due to needing to connect to many networks, it is not the best choice if someone is looking for a good, modern XMPP client.* There's an ever growing list of clients at for more options. aTalk has an incredible FAQ at which probably answers any and all questions one might have with XMPP from the user side. ## Running a service: prosody behind nginx ### Notes and warnings This is how I've done it; it's not guaranteed it'll work for you. If you get stuck, or want to hire me to set it up for you, reach out to me at (that's both an email and an XMPP address). My server runs FreeBSD[^13], meaning most paths will be `/usr/local/etc` instead of `/etc` as it would be on Debian or CentOS. ### Prerequisites: access to DNS configuration Running a webserver is usually simple, but those who'd ever set up an email server know there are additional magic that needs to be added to the domain records. The case of an XMPP server is not as complex as with email, but neither is so simple, with a mere A record. #### Base A (and AAAA records, if possible) ``` {.dns} example.net. 1800 IN A your.server.ipv4.address example.net. 1800 IN AAAA your.servers.ipv6.address ``` #### CNAMEs for subdomains used by components ``` {.dns} proxy.example.net. 1800 IN CNAME example.net. upload.example.net. 1800 IN CNAME example.net. groupchat.example.net. 1800 IN CNAME example.net. pubsub.example.net. 1800 IN CNAME example.net. ``` #### Service records for the XMPP clients ``` {.dns} _xmpp-client._tcp.example.net. 1800 IN SRV 1 1 5222 example.net. _xmpps-client._tcp.example.net. 1800 IN SRV 1 1 5223 example.net. _xmpp-server._tcp.example.net. 1800 IN SRV 1 1 5269 example.net. ``` #### TXT record for BOSH ``` {.dns} _xmppconnect.example.net. 1800 IN TXT "_xmpp-client-xbosh=https://example.net/http-bind" ``` ### Prerequisites: nginx and SSL (letsencrypt) certificates There are two topics **not covered** in this mini tutorial: - setting up nginx - obtaining SSL certificates from letsencrypt There are many ways of doing both, and excellent tutorials are available. ### nginx configuration to act as reverse-proxy for Prosody Nginx is an ideal reverse proxy. By putting it in front of Prosody, we keep the ability to serve websites, or web interfaces for our XMPP server, like movim[^14]. This is the minimum required configuration that goes inside a `server { }` block, before the `location /` part for nginx to forward everything needed for Prosody: ``` {.nginx} # BOSH location /http-bind { proxy_pass http://127.0.0.1:5280/http-bind; proxy_set_header Host $host; proxy_set_header X-Forwarded-For $remote_addr; proxy_buffering off; tcp_nodelay on; } # websockets location /xmpp-websocket { proxy_pass http://127.0.0.1:5280/xmpp-websocket; proxy_http_version 1.1; proxy_set_header Connection "Upgrade"; proxy_set_header Upgrade $http_upgrade; proxy_set_header Host $host; proxy_set_header X-Forwarded-For $remote_addr; proxy_read_timeout 900s; } # http_upload location /upload { proxy_pass http://127.0.0.1:5280/upload; proxy_set_header Host $host; proxy_set_header X-Forwarded-For $remote_addr; proxy_buffering off; tcp_nodelay on; } ``` ### Letsencrypt post-renewal hook I've added this into `/usr/local/etc/letsencrypt/renewal-hooks/post/10-prosody.sh` so whenever the `certbot` requests new certificates automatically, it'll get deployed for prosody. This needs `VirtualHost` in Prosody to work; if you're not using that, you'll need to change this. ``` {.bash} #!/usr/bin/env bash for d in $(cat /usr/local/etc/prosody/prosody.cfg.lua | grep VirtualHost | sed -r 's/.*"([^"]+)"/\1/'); do prosodyctl --root cert import "${d/.conf/}" /usr/local/etc/letsencrypt/live/ done /usr/sbin/service prosody restart ``` ### Get up to date `prosody-modules` The one in every distribution repository, including FreeBSD is outdated. You need `mercurial` aka `hg` to clone it: ``` {.bash} mkdir -p /opt/prosody-modules cd /opt hg clone https://hg.prosody.im/prosody-modules/ prosody-modules ``` ### Install Prosody As I mentioned, I'm using FreeBSD, so installing Prosody is simple these days: ``` {.bash} pkg install prosody ``` ### (optional) luadbi vs SQLite - if you want to use SQLite as backend In order to have luadbi work with SQLite, it needs to be compiled from ports, because the one from pkg is only built against MySQL. More so, it's built against MySQL 5.7, and so far I failed to built it against MySQL 8.0. ``` {.bash} cd/usr/ports/databases/luadbi make config ``` ![You need to compile luadbi to support SQLite3](luadbi-sqlite3.png) ``` {.bash} make make install ``` ### Configuring Prosody Before moving to the prosody config, these probably have to be run: ``` {.bash} # these are all ran as the root user # these are probably not required, they are here just in case mkdir -p /var/run/prosody chown prosody:prosody /var/run/prosody mkdir -p /var/db/prosody chown prosody:prosody /var/db/prosody # this is for the http_upload module mkdir -p /usr/local/www/prosody chown prosody:prosody /usr/local/www/prosody # import the letsencrypt certificates using the script from the # "Letsencrypt post-renewal hook" # above bash /usr/local/etc/letsencrypt/renewal-hooks/post/10-prosody.sh # symlink the main certificate for the legacy SSL services, just in case something needs it cd /usr/local/etc/prosody/certs ln -s example.net.crt https.crt ln -s example.net.key https.key ``` `prosody.lua.cf` ``` {.lua} plugin_paths = { "/opt/prosody-modules" } admins = { "admin@example.net" } modules_enabled = { "announce"; -- Send announcement to all online users "blocklist"; -- Allow users to block communications with other users "bookmarks"; -- XEP-0048: Bookmarks "bosh"; -- Enable BOSH clients, aka "Jabber over HTTP" "carbons"; -- Keep multiple clients in sync "cloud_notify"; -- XEP-0357: Cloud push notifications "csi"; -- client state indication "csi_simple"; -- Simple Mobile optimizations "csi_battery_saver"; "dialback"; -- s2s dialback support "disco"; -- Service discovery "http"; "http_altconnect"; "http_files"; -- Serve static files from a directory over HTTP "limits"; -- Enable bandwidth limiting for XMPP connections "log_auth"; -- Log failed authentication attempts with their IP address "mam"; -- XEP-0313, Store messages in an archive and allow users to access it "motd"; -- Send a message to users when they log in "offline"; -- XEP-0160 XEP-0091, Offline message storage and delayed delivery support "pep"; -- XEP-0163, Enables users to publish their mood, activity, playing music and more "ping"; -- XEP-0199, Replies to XMPP pings with pongs "posix"; -- POSIX functionality, sends server to background, enables syslog, etc. "presence"; -- "private"; -- Private XML storage (for room bookmarks, etc.) "proxy65"; -- Enables a file transfer proxy service which clients behind NAT can use -- if you are planning to use movim - https://movim.eu/ - you'll need this: -- "pubsub"; -- Implements a XEP-0060 pubsub service. "register"; -- Allow users to register on this server using a client and change passwords "roster"; -- Allow users to have a roster, aka friends list "saslauth"; -- Authentication for clients and servers. Recommended if you want to log in. "server_contact_info"; -- Publish contact information for this service "smacks"; -- XEP-0198: Reliability and fast reconnects for XMPP "throttle_presence"; -- presence throttling in CSI "time"; -- Let others know the time here on this server "tls"; -- Add support for secure TLS on c2s/s2s connections "turncredentials"; -- to pass TURN/STUN service that allows voice/video calls in modern clients "uptime"; -- XEP-0012, Report how long server has been running "vcard4"; -- XEP-0292 "vcard_legacy"; -- XEP-0398 User Avatar to vCard-Based Avatars Conversion "version"; -- Replies to server version requests "watchregistrations"; -- Alert admins of registrations "websocket"; -- XMPP over WebSockets; useful for web/JS clients "welcome"; -- Welcome users who register accounts } modules_disabled = { } allow_registration = false; daemonize = true; pidfile = "/var/run/prosody/prosody.pid"; legacy_ssl_ports = { 5223 } legacy_ssl_ssl = { key = "certs/https.key"; certificate = "certs/https.crt"; } c2s_require_encryption = true; s2s_require_encryption = true; s2s_secure_auth = true; auth_append_host = true; pep_max_items = 10000 -- see https://groups.google.com/g/prosody-users/c/U1LN78jhh_A network_default_read_size = "*a" trusted_proxies = { "127.0.0.1", "::1" } storage = "internal" -- note: you can, and probably should use an SQL backend -- but this was not covered in the tutorial. If you were -- to use MySQL: --storage = "sql" --sql = { -- driver = "MySQL"; -- database = "prosody"; -- host = "localhost"; -- port = 3306; -- username = "prosody"; -- password = "acbd542f-9009-46e4-9c01-a5a75092160f"; --} -- or if it's SQLite: --sql = { -- driver = "SQLite3"; -- database = "prosody.sqlite"; --} -- Logging configuration log = { info = "*syslog"; } -- Certificates certificates = "certs"; -- BOSH http_ports = { 5280 } http_interfaces = { "*" } https_ports = { 5281 } https_interfaces = { "*" } cross_domain_bosh = true consider_bosh_secure = true cross_domain_websocket = true -- Smacks and cloud notify push_notification_with_body = true push_notification_with_sender = true smacks_hibernation_time = 86400 -- Server contact info -- don't forget to either create an admin@example.net email account -- or to replace these with the real admin email contact_info = { abuse = { "mailto:admin@example.net", "xmpp:admin@example.net" }; admin = { "mailto:admin@example.net", "xmpp:admin@example.net" }; security = { "mailto:admin@example.net", "xmpp:admin@example.net" }; support = { "mailto:admin@example.net", "xmpp:admin@example.net" }; }; -- Limits limits = { c2s = { rate = "100kb/s"; burst = "2s"; }; s2sin = { rate = "100kb/s"; burst = "5s"; }; } -- Registration watch -- so you'll get alerted if anyone registers on your server registration_watchers = { "admin@example.net" } -- Proxy -- this is barely ever used since http_upload, but could come useful proxy65_ports = { 5000 } proxy65_address = "YOUR.EXTERNAL.IP.ADDRESS" -- http_upload http_upload_path = "/usr/local/www/prosody" http_upload_file_size_limit = 10485760 -- TURN/STUN -- for voice/video; it needs another, external service, see later turncredentials_secret = "CHANGE-THIS-TO-A-LONG-SECRET-WE-WILL-NEED-IT-IN-THE-TURNSERVER"; turncredentials_ttl = 86400; turncredentials_host = "YOUR.SERVER.FQDN"; turncredentials_port = 3478 -- this is the setting that needs to be turned on to make your server -- accept registrations -- without this setting, you can still create users manually, -- on the server itself. -- allow_registration = true; -- Virtual hosts VirtualHost "example.net" -- if you are running dovecot as an imap server on the same host -- and it already has authentication configured, use can use this: -- authentication = "dovecot" -- dovecot_auth_host = "127.0.0.1" -- dovecot_auth_port = "9993" -- but then you are limited to the users who exist in dovecot. -- Alternatively, you can stick to: authentication = "internal_hashed" -- this is telling the built-in webserver what to really use for URLS http_external_url = "https://example.net/" http_host = "example.net" -- this is the http_upload module to exchange files Component "upload.example.net" "http_upload" -- some things need to be defined per component, these are one of them: http_external_url = "https://example.net/" http_host = "example.net" -- probably superseeded by http_upload, but could come useful for old clients Component "proxy.example.net" "proxy65" -- this is to have multi user chats, aka rooms, aka groupchat Component "groupchat.example.net" "muc" modules_enabled = { "muc_mam", -- For XEP-0313 "vcard_muc" -- For XEP-0153 } -- If you're using movim - https://movim.eu/ -- or if you're planning to use this XMPP server as an IoT hub: -- Component "pubsub.example.net" "pubsub" -- pubsub_max_items = 10000 ``` ### Start the prosody server On most systems `service prosody start` should do. Don't forget to either create the `admin@example.net` user, or change the value in the config to one that exists, and can receive alerts: ``` {.bash} prosodyctl adduser admin@example.net ``` ### Install `coturn` TURNserver (needed for voice/video) Again, depending on your system. On FreeBSD: ``` {.bash} pkg install coturn ``` ### Configure TURN for voice/video This setup is for a server which is on the internet directly, with a public facing IP address. `turnserver.conf` ``` {.ini} listening-port=3478 tls-listening-port=5349 listening-ip=0.0.0.0 min-port=49152 max-port=65535 use-auth-secret static-auth-secret=CHANGE-THIS-TO-A-LONG-SECRET-WE-WILL-NEED-IT-IN-THE-TURNSERVER server-name=example.net realm=example.net cert=/usr/local/etc/prosody/example.net.crt pkey=/usr/local/etc/prosody/example.net.key syslog proc-user=prosody proc-group=prosody ``` ### Open firewall ports I hope you have a firewall if you have an internet facing server. This is a snippet from my `ipfw list` ``` {.bash} ipfw add 03600 allow tcp from any to me 5222 ipfw add 03700 allow tcp from any to me 5223 ipfw add 03800 allow tcp from any to me 5269 ipfw add 04100 allow tcp from any to me 5000 ipfw add 04700 allow tcp from any to me 3478 ipfw add 04800 allow udp from any to me 3478 ipfw add 04900 allow tcp from any to me 5349 ipfw add 05000 allow udp from any to me 5349 ipfw add 05200 allow udp from any to me 49152-65535 ipfw add 05300 allow tcp from any to me 49152-65535 ``` And in my `/etc/rc.conf` (note: it contains the TCP port 22, SSH as well, in case you copy-pasted it, to prevent anyone from locking them out from their server. If you don't need it, remove it): ``` {.bash} firewall_script="/etc/rc.firewall" firewall_type="workstation" firewall_myservices="22/tcp 5222/tcp 5223/tcp 5269/tcp 5000/tcp 3478/tcp 3478/udp 5349/tcp 5349/udp 49152-65535/udp 49152-65535/tcp" firewall_allowservices="any" firewall_logging="YES" firewall_logdeny="YES" firewall_enable="YES" firewall_quiet="YES" ``` ## Closing words Running a service is not a simple task, but all of the above is possible to run on a Raspberry Pi, at home, with no extra costs on your existing internet connection - that is if your service provider allows you to run a server. If you have a dynamic IP address, you'll need to be able to change the DNS entries according to it, and fast. DigitalOcean[^15] offers free DNS service and has an API the allows one to easily alter the needed entries, but it is definitely simpler if you have a fixed IP at home, or if you rent a VPS or a server that has a fixed IP. For more documentation, is a wonderful resource, but it's a bit hard to find what you're looking for on it. If you want to test your XMPP server, offers such a service. If you spot any mistakes, or have any improvement recommendations, do not hesitate, let me know! Good luck, and don't forget to spread the word of your artisan service! [^1]: [^2]: [^3]: [^4]: [^5]: [^6]: [^7]: [^8]: [^9]: [^10]: [^11]: [^12]: [^13]: [^14]: [^15]: