Email - Apmail and ezmlm management

Created by Drew Foulks, last modified by Daniel Gruno on Dec 18, 2022

Note: this document contains outdated information and will have a major update. The major update, however, depends on completion of complex software revisions. Consult with an Infra team member, or open a Jira ticket for INFRA before acting on any information contained here.

Notes for people who use the Apmail account

DO NOT GET FANCY with symlinks or the like. Look at the detailed directory listing for the same type of file FIRST and replicate it. If the similar one is a symlink, then see what it points to and why.
Don't just blindly copy files or directories and expect it to work.

Getting started as an Apmailer

Read all of this file. Twice.
Read all of `man ezmlm-make`, `man ezmlm-sub`, `man ezmlm-unsub`. Twice. You can read these on hermes.
Read the output of ~apmail/bin/makelist-apache.sh -h. At least twice.
Take a look at the issue history for the Mailing lists component of the INFRA project in Jira.
Assign a Jira issue to yourself and handle it through to completion.
Bug other apmail people on #asfinfra when you inevitably screw something up. Remember its not you, its ezmlm.
ANY time you use -++ in an ezmlm-make command. Back up the list configuration files. -++ will overwrite list configuration files using the flags set in the config file.
- trailer files will be lost etc.

Using the Apmail account

The sudoers list maintained by root defines who can edit the Apmail account. They should be Apmail on both hermes and minotaur. The list directories are only readable by Apmail, so most Apmailers use a sudo of their favorite shell to get started. For example,

sudo -H -u apmail tcsh

Many of the shared files (NOT the lists) are managed using Subversion:

https://svn.apache.org/repos/infra/infrastructure/apmail/trunk

The same files under /home/apmail are checked out (read only), so make changes ON YOUR OWN CHECKED-OUT COPY first, commit them, and then log in to Apmail and do a

cd /home/apmail
svn --config-dir=/home/apmail/.subversion2 --no-auth-cache update

Note that the list files are not in svn because running ezmlm-make changes the files directly, introducing far too many conflicts.

Commonly used commands are often scripted, using the scripts in apmail/bin.

All apache.org email addresses

All email aliases for anything @*apache.org are handled by

~apmail/.qmail-*

files on mail.apache.org. The pattern goes like this:

Mail for foo@apache.org is handled by ~apmail/.qmail-foo
Mail for foo@bar.apache.org is handled by ~apmail/.qmail-bar-foo

Yes, this also means that

bar-foo@apache.org

is going to be the same as

foo@bar.apache.org

However spamassassin treats those addresses differently on the MXs: the former is typically held at a 10.0 threshold whereas the latter is set at 5.0. This is worth remembering when you need to hack up special forwarding addresses for Sally (marketing) and Google Apps senders.

Note: we also support .qmail-foo-default and .qmail-foo-owner files for committer foo. These files are set up automatically by the makelist-apache.sh script

You can use

~apmail/bin/qmail-simple-srs.sh

to activate or deactivate owner files.

Dropping a forwarding account should also go through

~apmail/bin/remove-forwarder.sh

once the account has been removed from LDAP. That script will remove the correct files and is properly supported as we expand the number of associated .qmail files we deploy for account holders.

Outgoing email

When an external domain changes its MX records, qmail will ignore the change for a period of time unless dnscache is restarted manually:

# sudo svc -t /var/service/dnscache

nb- this section may be obsolete now that we are not using dnscache.

How to work with mailing lists

ezmlm-sub

Subscribes a specific email address to your mailing list. Type the following at your command prompt, replacing LISTNAME with the domain/listname, e.g. "apache.org/announce" or "httpd.apache.org/dev" and user@domain.tld with the email address you want to add to the list: ezmlm-sub /home/apmail/lists/LISTNAME user@domain.tld
You can subscribe multiple addresses at the same time by putting them all on the same line, separated by spaces. For example: ezmlm-sub /home/apmail/lists/LISTNAME user@domain.tld user2@domain.tld user3@domain.tld
You can also add multiple addresses to the subscriber list by creating a file with the addresses (one to a line, being sure to press [Enter] after the last address) and issuing the following command: ezmlm-sub /home/apmail/lists/LISTNAME < FILENAME >

Important:: The above methods to add subscribers do NOT meet the Confirmed Opt-In requirement of the Terms of Service. You will need to retain proof that all subscribers have been added to your list only after they have confirmed their subscription via email.

ezmlm-unsub

Unsubscribes a specific email address from your mailing list. Type the following at your command prompt, replacing replacing LISTNAME with the domain/listname and user@domain.tld with the email address you want to remove from the list: ezmlm-unsub /home/apmail/lists/LISTNAME user@domain.tld
You can unsubscribe multiple addresses at a time by simply putting them all on the same line, separated by spaces. For example: ezmlm-unsub /home/apmail/lists/LISTNAME user@domain.tld user2@domain.tld user3@domain.tld
You can also remove multiple addresses from the subscriber list by creating a file with the addresses (one to a line, being sure to press [Enter] after the last address) and issuing the following command: ezmlm-unsub /home/apmail/lists/LISTNAME < FILENAME >

ezmlm-list

Shows a list of all users subscribed to a specific mailing list with each address on its own line. Type the following at your command prompt, replacing replacing LISTNAME with the domain/listname: ezmlm-list /home/apmail/lists/LISTNAME

ezmlm-make

CAUTION! This command is used to create new lists and modify the letter flags of existing lists. DO NOT USE this if you have not read this documentation.

NOTE: the use of `-+` directs ezmlm to use the letter switches that are currently active for the list, as modified by the current command line. Thus, -+ makes switches sticky (most useful for changing one switch). By default, only switches specified on the current command line are used as the new switch configuration (if you want to change an option with this mode, specify the entire switch string, found in $domaindir/$listdir/config as the line matching `^F:`). This switch implies -e as it is meaningless except in edit mode (also specifies to use the entire config string).

e.g.: Allow MIME content in messages

ezmlm-make -+ -X solr.apache.org dev

Note that the config file choice (see -c and -C ) is also sticky. ezmlmrc(5) is set up so that most text files (and DIR /headeradd , DIR /headerkeep , and DIR /headerremove ) are not overwritten if they already exist, to preserve manual customizations. If local is specified, ezmlm-make overrides this behavior and rewrites all files.

You can also force ezmlm-make to rewrite all files by using -++ (which does happen during certain maintenance operations, such as when adding a message footer).

for a practical example of -++, see customized list info files .

Configuration flags:

Important options

Flag	Configures
m/M	Moderation Enabled / Moderation disabled
t/T	Enable / Disable message trailer (requires -++ see customized list info files)
u/U	Subscribers only / Anyone can post
x/X	Strip MIME content, drop mail with 'precedence: bulk' header, limit message size / Allow MIME content, 'precedence:bulk', and large messages
y/Y	Send a copy of the message to security@apache.org / do nothing.
z/Z	depends on -m . -mz: Moderate anyone not sending from an @apache.org address. -Mz: Unmoderated, but only @apache.org senders can post.

Common defaults

announce : -mUXYz (moderated only after From: anyone@apache.org)
commits : -MUXYz (limit sender to anyone@apache.org)
security : -MUXyZ (anyone can post, send copy to security@apache.org)
users : -MuXYZ (limit to subscriber or apache.org sender)
dev,private : -muXYZ (limit to subscriber|moderated|apache.org sender)

-muxYZ (limit to subscriber|moderated|apache.org sender after size/mime check)

Managing moderators

Subscribing a moderator

cd lists/<domain.apache.org>/
ezmlm-sub <list name, e.g., dev, private> mod <email to sub as mod>

Unsubscribing a moderator

Remember to check there will be sufficient moderators left before doing this.

cd lists/<domain.apache.org>/
(Make sure that the person is actually a moderator)
for i in */ ; do ezmlm-list $i mod | xargs; done
(look for moderator-to-unsubscribe's email to appear in at least one list)
for i in */ ; do ezmlm-unsub $i mod <email to unsub as mod>; done

Maintaining the apache.org mailing lists

All of the apache.org lists use qmail and ezmlm to run. The lists are located under /home/apmail/lists/{host}/{list-name} on mail.apache.org.

Always

Check "man ezmlm-make" to remind yourself of the commands
Be sure to run any ezmlm commands as user "apmail" and *not* as user root, since that will inevitably leave certain files as owned by root and unreadable/unwriteable by the ezmlm routines.
Be sure to always use the '-c' flag to ezmlm-make so that it picks up ~apmail/.ezmlmrc; this is because it needs some screwy thing with 'inlocal' that will, with luck, go away in the next version of ezmlm.

If you are adding options to an existing list, be sure to use the -+ option:

ezmlm-make -+ -{new_options} /home/apmail/lists/{host}/{listname}

Only use the -++ option if you want to restore ALL customizations to their default .ezmlmrc settings: trying to add trailers, subject prefixes, or resetting the outhost or headeradd files require this flag to be effective.

Working with existing mailing lists

An "ls -al ~ | grep incubator-jackrabbit" is very instructive. Except for simple forwards, aliases and archives, do *not* modify any of these files directly.
ezmlm-list /home/apmail/lists/incubator.apache.org/jackrabbit-dev . Note: change 'list' to 'sub' or 'unsub' and add an id or three at the end of the command, and you are doing something useful.
Changing a moderator list is done in a similar way: ezmlm-list /home/apmail/lists/incubator.apache.org/jackrabbit-dev mod
To check if there are any pending items in the moderation queue: ls /home/apmail/lists/incubator.apache.org/jackrabbit-dev/mod/pending (each file in the directory is a separate email)
To release one of the pending emails in the moderation queue (say, licensing):

cd /home/apmail/lists/apache.org/licensing/mod/pending
setenv SENDER apmail@apache.org # csh syntax -- adjust to your shell
ezmlm-accept /home/apmail/lists/apache.org/licensing <file-name>

To change the settings of a mailing list so that it automatically rejects messages from non-subscribers, without soliciting action from the list moderator. Non-subscribers should get a message saying that the list is subscriber-only: look at a "config" file in the ezmlm list dir. It shows the current settings; and in this case, it was -mu, where posts from non-subscribers are bounced to the moderators. Instead, what is desired is the behavior you'd get with -Mu, which means moderation is turned off but only posts from subscribers are allowed. So, the command line that needed to be run is

ezmlm-make -+ -M /home/apmail/lists/{host}/{listname}

-M turns off moderation entirely.
-u requires poster to be subscribed or on allow list.
-MU anyone can post
-Mu posts from non-subscribers/non-allow are rejected
-mU sends all posts to moderator (e.g., announce lists)
-mu sends posts from non-subscribers/non-allow to moderator

New top-level projects

Before lists can be created for a new TLP, complete all steps in https://svn.apache.org/repos/infra/infrastructure/trunk/docs/misc/tlp-creation.txt

Be sure to move any existing lists using the

/home/apmail/bin/movelist-apache.pl

script rather than creating them from scratch.

Creating a mailing list

Roy's super-easy makelist script svn:/infrastructure/apmail/trunk/bin/makelist-apache.sh is

/home/apmail/bin/makelist-apache.sh

on hermes.

It automates the steps above related to mailing list *creation*. It is self-documenting. Run

~/bin/makelist-apache.sh -h

Besides, you can look through the tcsh history for the apmail user on hermes to see example invocations of the script, and through the tcsh history on minotaur for apmailers, following up on the instructions it provides. The script is supposed to be reasonably foolproof in that you won't totally screw up existing mailing lists, though you can screw up the new one if you make a mistake.

Normally makelist is invoked by trunk:mlreq/queuerun.py, which processes trunk:mlreq/*.json, which is created by whimsy^Winfra.a.o via trunk:projects/infra/www/officers/mlreq.cgi. Running makelist manually should only be needed for non-TLP mailing lists.

This tool is very similar to the one used for user account management.

NOTE: The frontend Mail servers sync up with apmail's home directory on hermes between 55-59 past the hour. Until they have synced up, the mail servers are configured to REJECT inbound mail to any new mailing lists. Please keep this in mind before testing or announcing the availability of any new or moved mailing list.

Updating mod_mbox

Currently this needs to be manually updated by someone with an Ajax account after the first sync to Ajax.

Some instructions from Message-ID <439D11E5.30001@apache.org>, from infrastructure@apache.org 200512 archive:

Set or export HOME temporarily, Scripts rely on it: export HOME=/home/jerenkrantz
Create a new list, based on the raw PUBLIC rsync'ed archives. /home/jerenkrantz/work/httpd-mbox/scripts/create-archive-list \ /x1/mail-archives/raw/ > /home/pquerna/list.txt # Make sure /home/pquerna/list.txt is sane.
Backup the old list: /home/jerenkrantz/archives cp mbox-archives.list /home/pquerna/
The crontab'ed scripts use mbox-archives.list, # to determine where to run various things that create DBM files, # and make mod_mbox fast for doing indexes of lists and stuff. cat /home/pquerna/list.txt > mbox-archives.list
Regenerate the HTML page that is at # http://mail-archives.apache.org/mod_mbox/ /home/jerenkrantz/work/httpd-mbox/scripts/create-site-index

Manually creating a mailing list

Avoid doing this manually, if at all possible, since it is generally better to use the makelist script first even if you want to tweak the settings later. Read that script to see all of the things it does.

A command for creating a mailing list with a digest and index/archive:

ezmlm-make -cdigk /home/apmail/lists/xml.apache.org/axis-dev/home/apmail/.qmail-xml-axis-dev axis-dev xml.apache.org

However, a lot more options are needed for a typical list. All mailing lists must have:

-c to pick up /home/apmail/.ezmlmrc
-a to archive messages sent [concom- and board-private are exceptions]
-g to guard the archive from spambots gathering addresses
-i to index the lists (necessary for digest and archives)
-k to enable the deny list

The other options will depend on how moderation, posting, and subscription are set up. All of them must be understood before you run ezmlm-make, even if that means going back and re-checking the man file several times.

One good way to check configurations is to use egrep on the config files. The config file lists all the options set for a given list on a line that begins with "F:".

For example, to find all lists that were not using -c, I did

cd /home/apmail/lists
egrep '^F:.*C' */*/config

since C is the same as "not c".

See the script named apmail/bin/makelist-apache.sh for an example of how to automate list creation. Note that the right options depend on how the list is moderated, who can post, etc. Note also that you need to run commands on both hermes and minotaur as apmail in order to complete the process of creating or moving new lists.

Subscribe the moderators to the mod list immediately after creation:

ezmlm-sub /home/apmail/lists/{host}/{listname} mod someone@apache.org

If desired (normally), modify the list so that replies go to the list. Edit /home/apmail/lists/{host}/{listname}/headeradd so that it has an entry like:

Reply-To: "List Name" <<#l#>@<#h#>>

If in doubt, check the headeradd file for existing lists. Also, if you create a new -commits list (recommended), you will want to set the Reply-To in that list's headeradd so that the reply goes to the -dev list, rather than the -commits list, e.g.:

Reply-To: dev@<#h#>

Again, when in doubt, check the existing lists for examples.

Renaming a mailing list

Start the following command on both mail.apache.org (hermes) and people.apache.org (minotaur):

~/bin/movelist-apache.pl oldlist@old.apache.org newlist@new.apache.org:

E.g.,

~/bin/movelist-apache.pl foo-dev@incubator.apache.org dev@foo.apache.org:

Then press Enter first on minotaur and then on hermes.

Read the output carefully to be sure that the move succeeded.

If the Reply-To on mail looks wrong, edit it in the newlist/headeradd file.

Finally, send mail to the new list saying the move is done, and then verify that you receive the mail and that it gets archived.

Verify that the archives moved successfully before removing the old archive directories (as described in the output on minotaur).

Edit apmail/bin/archivealiasmonthly locally to reflect the new location, svn commit it and then do a "svn update" in ~apmail/bin on both hermes and minotaur.

After this point, there are a virtually unknowable number of steps, possibly including, but not limited to:

Add redirects to mail-archives.apache.org
Update where cvs commit messages go (search for "-cvs" above)
Update bugzilla (http://issues.apache.org/bugzilla/index.html)
Update gump (http://jakarta.apache.org/gump/)
Update other archives (e.g.:
- http://marc.theaimsgroup.com/
- http://www.gmane.org/

When in doubt, inquire on infrastructure@apache.org .

NOTE: The front-end Mail Servers sync up with apmail's home directory on hermes between 55-59 past the hour. Until they have synced up, the mail servers are configured to REJECT inbound mail to any new mailing lists. Please keep this in mind before testing or announcing the availability of any new or moved mailing list.

Merging two mailing lists into one new one

When merging two mailing lists it is necessary to move the subscribers of at least one of the old lists into the new one before using removelist.sh to remove the old list(s).

For the first list, use the 'renaming a mailing list' section above.

For the second list, merge the subscribers into the new one:

On mail.apache.org (hermes)

cd ~apmail/lists
ezmlm-list `pwd`/$tlp.apache.org/old-list-name/ | \ ezmlm-sub `pwd`/$tlp.apache.org/new-list-name

(in case the split line doesn't make it clear, we are piping the output of the ezmlm-list command into ezmlm-sub)

Duplicates should be taken care of.

Remove the old list archive subscriber:

ezmlm-unsub ~apmail/lists/$tlp.apache.org/new-list-name \ old-list-name-archive@$tlp.apache.org

You can then remove the old list:

~apmail/bin/removelist-apache.sh old-list-name@$tlp.apache.org

This automated process removes the .qmail files, We need to re-create one to point to the new list.

Set up a .qmail-old-list-[dev|user|commits] file with an entry pointing to the new list address.

echo "new-list-name@$tlp.apache.org" > ~apmail/.qmail-old-list-name

e.g:

echo "dev@lucene.apache.org" > ~apmail/.qmail-lucene-solr-dev

Deleting a mailing list

There is a script on hermes:

cd ~apmail/bin
./remove-list.sh nameoflist [-m "bounce message"]

example:

./remove-list.sh log4php-private@incubator.apache.org

Post ml delete cleanups
-----------------------

After removal of mailing lists also ensure that any other references to those addresses are altered/removed as necessary. Such areas to clean up could include:

asf-mailer.conf
[jira|bugzilla|wiki|cwiki] notifications
reviewboard

Exceptions in mailing lists

announce@apache.org
announce@{host}.apache.org
-- uses -z option to ezmlm-make to add in editor
-- |/home/apmail/bin/ezmlm-filter-from-apache
to only accept posts from @apache.org

security@{host}.apache.org
-- uses -y option to ezmlm-make to add in editor
-- |/usr/local/bin/ezmlm-send -r '/home/apmail/lists/apache.org/security'
causing all messages to be forwarded to security@apache.org

bugs@httpd.apache.org (see lists/httpd.apache.org/bugs/LOCALHACKS)
-- editor runs |/home/apmail/bin/automod-bugzilla1.pl
for special pre-processing

Customized list info files (message footers)

ezmlm allows you to customize all of the list response messages by editing the files under {listdir}/text. Unfortunately, ezmlm-make has a bad habit of replacing any customized files with the system defaults (/home/apmail/.ezmlmrc) if it thinks the list needs to be reconfigured (or when the moon is blue). Note that the ezmlm-make -++ option tells it to replace all of these files, including headeradd, with the defaults.

For that reason, we have a parallel tree of mailing list directories, located at "/home/apmail/lists-info" and stored under Subversion, that we use to maintain the history of customized files and restore them if ezmlm-make decides to clean them out. See the README.txt file in that directory for instructions on how to diff the trees and patch whichever side needs to be updated. Please note that this requires careful review of the patches, since folks tend to move the live list files around without also adjusting lists-info.

Make changes to files under lists-info in your local copy, commit it to Subversion, and then update the checked-out copy on mail.apache.org prior to running the diff_lists.pl command.

Files of interest (relative to ${listdir}):

headeradd self-explanatory (used for Reply-To, etc)
outhost what comes after the @ in list-help, List-Subscribe, etc
prefix Subject prefix
text/trailer message trailers (with the -T/-t flag set)

Overview:

sudo -H -u apmail bash

cd ~/lists/$domain/$list

cp config config.bak

cp headeradd headeradd.bak

ezmlm-make -++ -t /full/path/to/list/

diff headeradd and diff config

Private list accidentally created as public

Twiddle the ezmlm flags as necessary. Example for "security" lists:

ezmlm-make -+ -Bd -s -MU -XyZ ~/lists/foo.apache.org/security

(For details, see bin/makelist-apache.sh or .ezmlmrc)

Unsubscribe mail-archive.com and markmail
Move the archives dir on minotaur:~apmail, edit bin/.archives, commit it,
Run bin/archivealias. (To confirm, check ~apmail/mail-arch-aliases.)
On www(eos,aurora), remove the list archives from everywhere they are in:

~apmail/public-arch/subversion.apache.org/dev
/x1/mail-archives/raw/subversion.apache.org/dev
/x1/mail-archives.apache.org/mod_mbox/subversion-dev
(TODO: is this a complete list?)

Unsubscribe from the list anyone who is not allowed to be on it (for example, is not an ASF member and not a member of the relevant PMC).

Changing someone's email subscription

When you receive a request, first check to see that the new email entry for the user actually exists in ldap.

ldapsearch -xLLL uid=<apache user id> | grep mail (look for new email entry)

Or check for the entry in MailAlias.txt.

If all looks good, then inside of ~apmail/bin on hermes, run:

change-mailaddress.sh old-mail-address new-mail-address

Send a subscriber directly to moderation

Add a feature that moderators can use to send subscriber emails to the moderation queue instead of the mailing list. This helps control bad actors by filtering their emails to moderators, whether they are subscribed or not.

The steps to do this are currently manual.

ssh to hermes (you need apmail or root karma to perform the following steps)
Edit /home/apmail/.qmail-$project-$listname, e.g .qmail-ponymail-dev. For top level lists @apache.org, the pattern is .qmail-$listname.

Add the following code to the second line (directly below the sender-demunger 'deny' line), being very careful to not insert extra line breaks. The strings in this file are bourne shell/bash based, and must be valid shell syntax. For most lists, the $projectname will be the TLP name, e.g. ponymail.apache.org. For top level apache.org lists, the $projectname will be 'apache.org'. When in doubt, refer to the format of the 'deny' line.

|/x1/apmail/bin/sender-demunger /usr/bin/ezmlm-issubn -n '/x1/apmail/lists/$projectname.apache.org/$listname' 'sendsubscribertomod' || { /usr/bin/ezmlm-store '/x1/apmail/lists/$projectname.apache.org/$listname' ; exit 99 ;}

Save the file and close.

Create a new directory:

mkdir /home/apmail/lists/$projectname/$listname/sendsubscribertomod

# Note ezmlm-sub may not work unless a subscribers subdirectory is created
mkdir /home/apmail/lists/$projectname/$listname/sendsubscribertomod/subscribers

Add someone to the list of forced moderation:

cd ~apmail/lists/$projectname
ezmlm-sub $listname sendsubscribertomod foo @bar .baz

# Note you can verify the addition with ezmlm-list
ezmlm-list $listname sendsubscribermod
The email addresses in the forced moderation list are used to compare against incoming From: lines. They have nothing to do with the subscribers, or the allow list for the list.

Please note these configs are not 100% sticky - in other words, if another config change were to rewrite the dot file, then these configs would be lost and you would need to re-apply them. But this should be a rare occurrence.

Checking MTA Logs

Q: "an Apache mailing list (or someone using minotaur's SMTP) emailed me, and I didn't receive the email."

A: Check /var/log/qmail/current (or the @* named files). Look for the line with a 250/471/500 reply from the remote server, DTRT. Normally finding the right lines involves first finding the 'delivery' number and then grepping -w for that. The @ numbers at the beginning of each line can be decoded with tai64nlocal(1).

New podling mailing lists

Make sure they are listed in https://svn.apache.org/repos/infra/websites/production/incubator/content/podlings.xml
Once they are listed, grep infrastructure/trunk/dns/zones/normaltlps.txt to see if they need to be added to DNS. If so, add DNS ($podling.a.o because of mail stuff)
New lists should be requested via https://selfserve.apache.org/ and self-serve takes care of the rest.

Migrating / renaming existing lists

Important documents to review (they may be access restricted):

Run the following command on people.apache.org (minotaur) as apmail

~/bin/movelist-apache.pl oldlist@old.apache.org newlist@new.apache.org
E.g.,
~/bin/movelist-apache.pl foo-dev@incubator.apache.org dev@foo.apache.org

Verify archives on people.apache.org (minotaur)

Verify

that /home/apmail/public-arch/teaclave.apache.org/notifications contains the complete archives,

that /home/apmail/public-arch/mesatee.apache.org/notifications is now empty.

and that the permissions and group ownership of the directories are set correctly.

Run the following commands on mail.apache.org (hermes) as apmail:

~/bin/movelist-apache.pl oldlist@old.apache.org newlist@new.apache.org
E.g.,
~/bin/movelist-apache.pl foo-dev@incubator.apache.org dev@foo.apache.org

Svn commit /home/apmail/bin on minotaur.apache.org.

svn diff ~apmail/bin/.archives && read -r _ && \
svn commit -m 'rename notifications@mesatee.apache.org to notifications@teaclave.apache.org' ~apmail/bin/.archives && \
~apmail/bin/archivealias

and svn up /home/apmail/bin.

Svn up /home/apmail/bin on mail.apache.org (hermes) to catch changes made on minotaur

svn up /home/apmail/bin

In a local checkout, modify rcpthosts and virtualdomains for qmail\

svn co https://svn.apache.org/repos/infra/infrastructure/trunk/qmail/control

update the domain names in:
- rcpthosts
- virtualdomains

commit the files.

svn up /var/qmail/control on mail.apache.org (hermes) and reload qmail configuration

svn up /var/qmail/control

Reload qmail on hermes to pick up the control file changes.
(infrastructure/trunk/docs/services/qmail-ezmlm.txt)

svc -h /var/service/qmail-send

If the Reply-To on mail looks wrong, edit it in the newlist/headeradd file.

Send out verification email

Verify that you receive the mail and that it gets archived correctly.

Clean up

Remove the now-empty directory on mail.apache.org (hermes) and on people.apache.org (minotaur).

Directories to remove:

hermes:/home/apmail/lists/$oldlist.apache.org

minotaur:/home/apmail/public-arch/$oldlist.apache.org

Moving mailing lists from public to private

Allow 1 hour to do this.

To start, take as a reference https://issues.apache.org/jira/browse/INFRA-13375 , We are going to move user@reef.apache.org from Private to Public.
Access Required:

root and apmail on minotaur.apache.org
apmail on hermes.apache.org
commit access to .archives

First http://mail-archives.apache.org/mod_mbox/#reef and lists.apache.org confirm that the list is private as it is not publicly available
Check the subscription of the archive

On hermes as apmail user, list the subscribers of the user@reef list to check what archives are subscribed:

-* hermes% pwd, /home/apmail/lists *

hermes% ezmlm-list reef.apache.org/user |grep -i archive-asf-public@cust-asf.ponee.io user-archive@reef.apache.org

List the cust-asf archiver on the dev list as comparision, as 'apmail' user and in the /home/apmail/lists/ directory
1. ^ that tells us the list we want
1. ^ that will subscribe it
1. ^that will unsubscribe the one we don't want
1. users list = people who use the software .
2. dev list = people who write the software, as a general rule.
3. commits list = usually things like git commits and svn commits go there
4. So, for example: dev@reef.apache.org on hermes is reef.apache.org/dev
5. commits@reefa.o then is mostly commit mails
6. private@reef.a.o is for private communications between PMC members
7. Use private lists sparingly

[apmail@hermes lists]$ ezmlm-list reef.apache.org/dev | grep archive-asf-public@cust-asf.ponee.io
ezmlm-sub reef.apache.org/user archive-asf-public@cust-asf.ponee.io
ezmlm-unsub reef.apache.org/user archive-asf-private@cust-asf.ponee.io
For general knowledge: dev is always a public list

On minotaur.apache.org take a quick look around first. ls -al /home/apmail/public-arch/ --> this directory is where all our public lists are meant to reside
ls -al /home/apmail/private-arch is where the private lists are
ls -al /home/apmail/public-arch/reef.apache.org/ show all reef projects public lists. So in this case the "user list" is not here, because it is private.
ls -al /home/apmail/private-arch/reef-user/ is where the user list resides.
Next step is to move those to the right place, so sudo to root user and carefully move the reef-users list to where it should be and rename it accordingly; ie. reef-user becomes user.
Change the permissions to match.
On your local machine, make sure you have an up-to-date copy of .archives file. Edit it to change the reef-user entry from private to public and commit it.
Then on both minotaur and hermes as the apmail user do an svn up in /home/apmail/bin "(supply your own credentials when prompted)"
On minotaur (as apmail user) in /home/apmail/bin directory run ./archivealias
Open a ticket with ponee.io who host list.apache.org for us: we need to email them and ask that all reef-user mails be made public

root@ponee.io is our contact address
cc: root@apache.org

Our *current official* archives are http://mail-archives.apache.org/mod_mbox
Our secondary is lists.apache.org, so we must ensure both work
Testing:
1. [apmail@hermes lists]$ cat subversion.apache.org/users/config | grep ^F
2. [apmail@hermes lists]$ cat reef.apache.org/user/config | grep ^F

On hermes as the apmail user

That lists the config switches for two user(s) lists, the only difference here is the m vs M, which is important.

m = send non -subscriber posts to a moderator
M = reject non subscriber posts

We want m, and we currently have M
on hermes as the apmail user: ezmlm-make -+ -m /home/apmail/lists/reef.apache.org/user
[apmail@hermes lists]$ cat reef.apache.org/user/config | grep ^F

F:-aBcdeFgHiJklmnOpqrSTuVWXYZ
lower case m

Now we are good to go
For more details, read: the ~apmail/README file about these switches
Also :- http://untroubled.org/ezmlm/man/man1/ezmlm-make.1.html

Shutting down / removing a mailing list

On hermes su to apmail and cd ~/bin/
Run the following, replacing <lists> and <project> as required: ./remove-list.sh -m 'The project is retired. See https://attic.apache.org' <listname>@<project>.apache.org
Follow the instructions for .archives and archivealias provided by the remove-list script
1. The script output says changes "should be picked up by minotaur.apache.org within an hour or so." – the cron that does this doesn't exist anymore.
2. Manually deploy changes: commit `.archive` changes from a local copy and run `svn up` in the /home/apmail/bin/ directory as the `apmail` user on both hermes and minotaur
Remove private@ address from ~apmail/.qmail-pmcs on hermes
Delete ~apmail/lists/<project> directory once empty

More details/background can be found in /home/apmail/README

Re-indexing a list or sub-list

If a subscriber index was created by an earlier version of ezmlm, it may refuse to update due to database versioning mismatches.
To address this, you can re-index the list using the following command ( as apmail user!! ):

Re-indexing a list

# re-index a standard list (dev@httpd.a.o here):
/x1/apmail/bin/ezmlm-fix-index.py /x1/apmail/lists/httpd.apache.org/dev

# re-index a sub-list (dev@httpd digest list in this example):
/x1/apmail/bin/ezmlm-fix-index.py /x1/apmail/lists/httpd.apache.org/dev digest

Whitelisting a specific address

Using the ezmlm-sub command you can whitelist a system account on a list that is configured to accept messages from subscribers only. Using allow ensures that this address will not receive messages sent to the list, just that the sending address is exempt from the subscribers-only rule.

to do so, run the following commands on hermes.apache.org:

Whitelisting an Address

# On hermes as apmail
cd $listdir
ezmlm-sub . allow address@to-whitelist.com

# To verifytop
ezmlm-list . allow

Note that some automated services generate dynamic 'envelope sender' email addresses, in which case 'allow'ing the sender will work at most once.

There is a 'sender-demunger' script which can potentially be used to convert the dynamic address into a static one - see INFRA-18843 and INFRA-19360 for an example. This requires editing the sender-demunger script with a known pattern for the dynamic sender at https://svn.apache.org/repos/infra/infrastructure/apmail/trunk/bin/sender-demunger which lives at hermes:/home/apmail/bin/sender-demunger.

Selfserve Mailing List Creation Queue

https://selfserve.apache.org allows for self-service PMC creation of project mailing lists. The code which queues the list creation json object is here: https://github.com/apache/infrastructure-p6/blob/production/modules/selfserve_portal/files/www/site/cgi-bin/queue.cgi

The queue files are created in tools-vm-he-de:/usr/local/etc/selfserve/queue and can be edited until the list is created.

The actual list creation occurs on hermes via the ~apmail cron job /home/apmail/bin/selfserve-make-lists.sh which calls selfserve-make-lists.py located in SVN: https://svn.apache.org/repos/infra/infrastructure/apmail/trunk/bin/selfserve-make-lists.py. List creation runs every 4 hours.

email users@infra.apache.org for any queries or suggestions.