June 28, 2005
qmail sendmail postfix - 三种MTA的比较
关于sendmail/qmail/postfix孰优孰劣，以及部署邮件系统的时候该选哪一个的讨论已经重复了千百次了。但事实往往并不是A好B坏，或B好A坏，必须根据场合和应用的要求来定。但虽然如此，大多数人还是需要一个相对公平的评价，以引导邮件系统的部署。
自己一直很慎重于回答这类问题，以免引发不必要的争论甚至矛盾，但还是必须面对这个问题做一定的分析和比较的，否则很多朋友经常会问“到底用哪个好？”，却拿不出完整的答案。
首先看看三个MTA的历史...
MTAs的发展历史
Sendmail
毫无疑问，sendmail是最古老的MTA之一。它比qmail和postfix要古老得多。最早它诞生的时候，Internet还没有被标准化，当时主机之间使用的是UUCP技术来交换邮件。
它被设计得比较灵活，便于配置和运行于各种类型的机器。
Qmail
qmail是新生一代的MTA代表，它以速度快、体积小、易配置安装等特性而著称。作者D. J. Bernstein(djb)是一个数学教授，富有传奇色彩。djb于1995年开发qmail，1996年发布0.70版，并使用了多种当时比较先进的技术，包括Maildir，与sendmail单个binary不同的模块化设计，权限分离，以及使用了大量由djb编写的配套工具，如daemontools，ucsip-tcp等。
qmail迅速成为了Internet上最有名的MTA，使用者众。
Postfix
Postfix作者是Wietse Venema，一名著名的安全专家。最早postfix起源于1996年，当时venema 在美国IBM研究中心负责研究更安全的邮件系统，当时称为Vmailer。后因为商标问题于1998年11月正式更名为Postfix
Postfix以替代sendmail为目的，并提供了一个更安全、更高性能的灵活的体系。它同样也采用模块化设计，使用了大量优秀的技术，以达到安全的目的。由于作者的设计理念独到，经过7，8年时间，Postfix现今已发展成为功能非常丰富，扩展性和安全性强的优秀MTA。
概括的比较
以下的分析主要基于我在CASA上发的一个小文章，对sendmail/qmail/postfix做了一个概括性的比较。sendmail
sendmai功能非常强大，很多先进功能在sendmail上都最先有实现。sendmail里的Milter技术是一个非常好的框架，目前postfix及qmail仍然没有官方发布的方案比milter要好。
但sendmail也有典型的历史问题，只有一个binary程序，需要sid权限，m4配置文件复杂难懂。这些是是阻碍sendmail更好发展的一些客观问题。客观来说，调教得好的sendmail，其性能也是相当不俗的，据一个国外的Unix杂志称，在solaris+内存文件系统+带电池的raid系统下，sendmail能达到惊人的287封/秒的注入速度！
目前sendmail比较适合那些老用户，因为他们习惯了sendmail的应用环境和配置。
qmail
qmail体积非常小巧，source的gz包大概只有260多K，是三大MTA中最小的！模块化设计，避免了sid问题，基本功能齐全。配置相对sendmail而言，简单了很多，而且用户非常广泛。而且补丁和插件非常多，例如著名的vpopmail，netqmail，以及qmail-ldap等。
但qmail有几个问题，一是djb已经5，6年没有继续开发了，补丁的良莠不齐及版本依赖是非常麻烦的事，这对初学者极为不利。二是功能扩充需要补丁来完成，扩展能力不足。
总体上qmail依然是个非常不错的选择。对于希望了解mta原理，或希望修改mta代码的爱好者，qmail是值得推荐的。对于需要建立中小型邮件系统的用户也同样适合。而对于需要丰富功能却不想面对补丁困难，或者需要建立大型的系统，qmail不太合适，需要更丰富的经验和技术。
postfix
postfix如今已经独树一帜，流水线、模块化的设计，兼顾了效率和功能。灵活的配置和扩展，使得配置postfix变得富有趣味。其主要的特点是速度快、稳定，而且配置/功能非常强大，并和sendmail类似，提供了与外部程序对接的API/protocol。尤其是配置部分，可以说是一扫qmail和sendmail的各自缺点。
但postfix管理及配置的入门依然需要一定的工夫，必须仔细阅读官方文档。postfix另一个优势是至今依然保持活跃的开发工作，而且稳步发展，适合高流量大负载的系统，扩充能力较强。
大规模应用例子
国内若干个大型email ISP（如163.net/tom.com/163.com及sohu等）过去都使用qmail，后来全部更换成postfix。
新浪使用qmail，yahoo使用qmail。但这些已经不是普通的qmail了。
技术层面的分析
这里仅探讨一些典型的技术特点，从这些特点可以看出每个MTA设计的异同，主要讨论的焦点是qmail和postfix。磁盘I/O
从队列文件的读写来看，qmail处理每一封邮件时，都至少需要建立3个文件，mess, intd, info等。而Postfix使用的是单队列文件设计，因此磁盘I/O的开销要比qmail小得多，如果仅仅从这个方面考虑，postfix的队列是qmail的2-4倍那么快。
从我过去的一个qmail vs postfix对比测试中，也可以发现这个问题。
数据同步
如果从MTA对待操作系统的文件是否安全写入磁盘的策略来看，qmail和postfix也是不同的。Postfix使用的是随机写，并且需要写入完成并安全同步到磁盘后才算完成。而qmail的写入则是即刻执行的，因此它将等待数据安全写入磁盘后才返回。对于高流量的系统而言，这将导致性能问题。
此外，Postfix的队列对于FreeBSD的softupdate是安全的，而qmail则是不安全的，qmail作者明确警告用户不要使用softupdate，除非是有磁盘后写电池。
扩充能力
sendmail有着非常好的扩充能力，支持众多的特性，功能可谓豪华。包括频率控制到集群支持应有尽有。而milter API则更加使sendmail的灵活性发挥至极，通过milter，用户可以对邮件几乎所有的参数进行控制！但是在存储方面，由于只支持mbox，会有一定的问题。
qmail在系统容量扩展上有着独到的设计，配合qmail-ldap补丁，可以充分利用qmqp及分布存储的优势。现今已有各式各样的qmail扩展方案，最著名的是qmail-ldap。但qmail缺乏类似milter的设计，功能扩展需要各种补丁，而补丁的设计水平参差不齐，配置能力有限。实施起来相对是最复杂的。
Postfix同样有着非常好的容量扩充能力，利用LMTP或transport的/alias的方法，可以分布式的存储邮件，扩充容量。同时postfix的功能扩展也非常强，通过灵活的配置即可实现复杂的功能，这是其最突出的优点之一，是qmail望尘莫及的。此外，类似sendmail的milter，postfix拥有content_filter和policy 两个与外部程序/应用对接的接口，但不如milter那样功能集中和灵活，也没有完整实现qmail的qmqp及类似qmail-ldap的机制。
可配置性
sendmail 使用m4语法，单一的主配置文件（sendmail.cf）是三个mta中最难使用的，但是如果熟悉使用的话却能实现复杂的功能。
qmail使用的是大量小配置文本，格式最简单，每个配置一个文件，存放在/var/qmail/control目录里。
postfix也使用单一的主配置文件（main.cf），同时还有对应master主服务进程的配置文件master.cf，但使用的是简明易懂的key = value 格式。
总体而言，qmail的配置文件较易管理（格式最简单）但配置文件多（10个以上），而postfix的格式简单只有2个配置文件，并配备强大的postconf工具，sendmail的配置文件最复杂。
数据库支持
sendmail通过一些插件/补丁，可以支持mysql/pgsql/oracle等，ldap及小型的dbm/cdb等数据存储格式。
qmail默认只支持cdb，需通过补丁才可支持ldap，mysql，pgsql及oracle等。
postfix可以支持的数据库应该是最多的，默认就包括了mysql/pgsql/ldap及dbm/cdb和cidr/nis*/btree等一堆。还支持特殊的tcp_table(仅在snapshot里支持)
稳定性/负载能力
sendmail, qmail, postfix都比较稳定。在高负载下，配置不佳或没有打足够补丁的qmail容易被DOS攻击打跨，而postfix在遇到超过配置的限制时会降低处理能力，但系统依然有一定资源可用。
作者介绍
sendmail - Eric Allman Unix专家、学者
qmail - DJB 数学教授，科学家
Postfix - wietse venema 安全专家 学者
Recommentaion - 建议
我建议在使用Postfix MTA，无论是小型系统，还是大中型系统，能带来最高的性价比。一些有用的link
在足够好的硬件条件下Postfix比qmail更快的原因分析benchmark，无聊还是骗局？
qmail/postfix/sendmail 比较
Sendmail 历史
有关mta benchmark
Sendmail性能调整
qmail可靠性FAQ
Posted by hzqbbc at 12:15 PM | Comments (3)
June 27, 2005
Mbox vs Maildir - 两者原理和区别
在开放源代码的世界里，电子邮件服务器最主流的目前有三种，分别是sendmail, qmail, postfix。而存储格式最流行的有两种，Mbox和Maildir，它们都是开放的存储格式，因此兼容性比较好。
而mbox和maildir的历史，却不为大多数用户所了解。今天和一个朋友聊到mbox和maildir哪个好，顺便搜集一下mbox和maildir的资料，谈谈这两者的原理和区别，以及应该如何做选择。
Mbox的历史较Maildir悠久，sendmail支持mbox，qmail和postfix都支持mbox，其主要特点就是“所有邮件都存放到一个文件里”。每个邮件之间以特定的标记分割。
而Maildir则相反，每一封邮件保存成一个文件，每个文件名称一般有一定的规律，例如会包含时间戳、pid及inode节点号等。
除了mbox和maildir格式外，还有一个叫mbx的格式，它是对mbox的改进版本。主要用在UW-IMAP server里。它最大的特点是有一个针对mbox文件的索引，能改善读/写性能。但依然需要file lock。
随着qmail/postfix的普及，以及mbox的一些问题暴露，maildir得到了越来越多的应用。其中mbox的最主要问题是文件缩定（file lock），其次是大多数update操作的效能问题。
以下是引自courier-mta.org上的mbox和maildir的测试对比文章，介绍了mbox和maildir：
mbox mail storage format
This is the traditional way to store mail on UNIX-based mail servers.
Individual messages are simply concatenated together, and saved in a
single file. A special marker is placed where one message ends and the
next message begins. Only one process can access the mbox file in
read/write mode. Concurrent access requires a locking mechanism. Anytime
someone needs to update the mbox file, everyone else must wait for the
update to complete.maildir mail storage format
Maildirs were originally implemented in the Qmail mail server, supposedly
to address the inadequacies of mbox files. Individual messages are saved
in separate files, one file per message. There is a defined method for
naming each file. There's a defined procedure for adding new messages to
the maildir. No locking is required. Multiple processes can use maildirs
at the same time.mbx mail storage format
This is a slightly modified version of the original mbox format that's offered by the UW-IMAP server. mbx mailboxes still require locking. The
main difference from the mbox format is that each message in the file is
preceded by a record that carries some message-specific metadata. As
such, certain operations that used to require the entire mbox file to be
rewritten can now be implemented by updating the fixed-size header
record.This benchmark focuses mainly on the mbox and maildirs formats. In March of 2003 an unrelated party conducted a similar benchmark for mbx
formats. See http://www.decisionsoft.com/pdw/mailbench.html
for more details.
Mbox vs Maildir之优缺点比较
这里给出一个基本的特性对比，读者很容易就清楚根据自己的应用到底应该选什么存储格式：可靠性
优选是Maildir，因为mbox只有一个文件，一旦出问题之后，所有邮件都将损毁。
更新速度
这里主要指的是删除/增加邮件的能力，无疑Maildir完胜Mbox
搜索速度
这点Mbox因为是单文件，因此搜索的能力要强于maildir
并发访问能力
对于繁忙的邮件系统，多个进程同时访问同一封邮件是可能的事情，Mbox需要flock()的支持，而且如果某一个进程操作时间长，则其他所有进程都堵塞了。Maildir没有这个问题。在NFS等网络文件系统上，Maildir相对安全，Mbox不能用于此类型环境
扩充能力
现在的邮箱已经不是5年前甚至10年前的1MB,2MB而是100,200甚至1G/2G，Mbox应付那么大的容量已力不从心，无疑Maildir是比较适合的。
文件系统依赖
Maildir较依赖文件系统，尤其是依赖对目录的索引能力，用ReiserFS会比较快，对于超大型的maildir，读写性能将受到考验。相对而言Mbox则不存在这个问题。
综合结论
推荐使用Maildir格式，安全可靠，绝大部分操作都远快于Mbox。而且现今支持Maildir的软件越来越多，qmail/Postfix 都支持。一些有用的链接
原理标准
http://cr.yp.to/proto/maildir.html
http://www.qmail.org/qmail-manual-html/man5/mbox.html
Benchmark/比较
http://www.courier-mta.org/mbox-vs-maildir/
http://www.decisionsoft.com/pdw/mailbench.html
转换工具
http://www.qmail.org/mbox2maildir
http://batleth.sapienti-sat.org/projects/mb2md/
Posted by hzqbbc at 10:45 PM | Comments (2)
May 12, 2004
Postfix Ecartis HOWTO - 集成ecartis + Postfix
English version: 0.02alpha
Chinese caution: 本文是过去用英文写的，有兴趣的朋友可以翻译成中文，谢谢。
Introduce
There are a lot of maillist softwares compatible with the Postfix, but we need a simple configure and powerful one, Ecartis will meet our need.Ecartis is a certain kind of mailing list manager software, designing for small binary and fast operation, with modular archtecture. It support sendmail/Postfix Exim4 and qmail, very easy to configure, but it's bit regreat that ecartis lacks documentation :-(
Requirement
Before you begin your work, make sure the following softwares or configuration are done:Postfix (1.x/2.x, 2.x are better)
gcc and gcc-devel package
GNU make and other tools
root account
Unix like OS, linux is prefer
Make and build
Getting ecartis
Ecartis developed since 1998, but it's not 1.0 release ready. The latest version is 1.0.0-snap20040426, use wget or ncftp to get the tar ball from http://web.archive.org/web/20041010224951/ftp://ftp.ecartis.org/.
Complie ecartis
Unpack the tar ball with GNU tar, like this:
$ tar xfz ecartis-1.0.0-snap20040426.tar.gz
Then get into the src directory, there is no autoconf/automake style configure file ship with ecartis, only a Makefile.dist, copy the Makefile.dist to Makefile:
$ cp Makefile.dist Makefile
Make sure that gcc and make can work properly before you begin to complie, type make to complie:
$ make
Installation
If nothing exception occur, ecartis and modules complie finished, binary will in the build direcotry. Install ecartis when you want to use it:$ make install
The default location of ecartis installation will at the top of source directory, if you want to use ecartis in another location, copy them to the different path.
At the top of source directory, assume you want to place ecartis and modules in /home/ecartis, do as the following steps:
$ su root # useradd -d /home/ecartis -s /bin/false ecartis # cp ecartis /home/ecartis/ # cp ecartis.cfg.dist /home/ecartis/ecartis.cfg # cp ecartis.aliases.dist /home/ecartis/ecartis.aliases # cp -rf modules lists queue /home/ecartis/
Configuration - create a testlist
You should remember that use full path to run ecartis or things will be corupt! Please refer to ecartis documentation if you've some doubt.
# /home/ecartis/ecartis -newlist lists
This will create a list name "lists" for you, ecartis will prompt for list admin/owner, enter email address response for it. Then ecartis will create "lists" directory in the /home/ecartis/lists/lists and user/config file, at last ecartis display aliases content:
# Aliases for 'lists' mailing list. lists: "|/home/ecartis/ecartis -s lists" lists-request: "|/home/ecartis/ecartis -r lists" lists-repost: "|/home/ecartis/ecartis -a lists" lists-admins: "|/home/ecartis/ecartis -admins lists" lists-moderators: "|/home/ecartis/ecartis -moderators lists" lists-bounce: "|/home/ecartis/ecartis -bounce lists"
Configuration - Postfix
Use vi or emacs or other editor, open main.cf, paste the ecartis aliases content into the default aliases file, which commonly located at /etc/postfix/aliases, but i highly recommend that use a seperate aliases file, it's more clean and easy to manage:) my configuration shown as follow:
alias_maps = hash:/etc/postfix/aliases hash:/home/ecartis/ecartis.aliases mydestination = $myhostname, $mydomain mydomain = postfix.org.cn myhostname = lists.postfix.org.cn
Save main.cf and ecartis.aliases, then run postalias to rebuild ecartis.aliases, last reload postfix:
# postfix reload
Test lists
It's the time to test ecartis now. Send an email to [listname]-request@domain.tld as title or body contain "subscribe" to test subscribe function, in my case, send email to lists-request@lists.postfix.org.cn. Look at the maillog to get detail information.Almost immediately you will receive reply from ecartis, if nothing unexpected, you've been subscribed. The email will contains:
>>subscribe Subscribed.
Virtual Hosting
Ecartis can easily support virtual hosting, with postfix we need some tricks to get ecartis to work.
By some means virtual hosting equal to virtual domain, the Postfix MTA support virtual domain via alias, but virtual_alias_maps(2.0) or virtual_maps(1.0) could not support sendmail style aliases, then it means we can't simpliy put some lines like:
lists-request@lists.postfix.org.cn "| /home/ecartis/ecartis -r lists"
into those maps/files, how could we implement virtual domain? The trick is according to sendmail style aliases, the processing flow show as below:
mail to lists@domain.tld -> lookup virtual_alias_maps -> found lists@domain.tld point to "lists-domain.tld" -> lookup aliases maps -> found lists-domain.tld point to "|/home/ecartis/ecartis -s listname" -> OK
Assume we want to support thisdomain.tld and thatdomain.tld at the same machine, follow the steps:
1) DNS setup, point both the thisdomain.tld and thatdomain.tld's mx to the machine running ecartis
2) We only make thisdomain.tld as an example, add the following lines into the Postfix virtual_alias_maps file:
lists@thisdomain.tld lists-thisdomain.tld lists-request@thisdomain.tld lists-request-thisdomain.tld lists-repost@thisdomain.tld lists-repost@thisdomain.tld lists-admins@thisdomain.tld lists-admins-thisdomain.tld lists-moderators@thisdomain.tld lists-moderators-thisdomain.tld lists-bounce@thisdomain.tld lists-bounce-thisdomain.tld
Then run postmap to rebuild it
3)add the following lines into sendmail style aliases file:
lists-thisdomain.tld: "|/home/ecartis/ecartis -s lists-thisdomain.tld" lists-request-thisdomain.tld: "|/home/ecartis/ecartis -r lists-thisdomain.tld" lists-repost-thisdomain.tld: "|/home/ecartis/ecartis -a lists-thisdomain.tld" lists-admins-thisdomain.tld: "|/home/ecartis/ecartis -admins lists-thisdomain.tld" lists-moderators-thisdomain.tld: "|/home/ecartis/ecartis -moderators lists-thisdomain.tld" lists-bounce-thisdomain.tld: "|/home/ecartis/ecartis -bounce lists-thisdomain.tld"
Then run postalias to rebuild it
4)run this command:
#/home/ecartis/ecartis -newlist lists-thisdomain.tld
and enter the listowner address, then ecartis will create config and users file for the list "lists-thisdomain.tld" in /home/ecartis/lists direcotry.
5)configure /home/ecartis/lists/lists-thisdomain.tld/config to meet your need, refer to ecartis documentation or varible difinition for details.
If everything works ok, then we done it :-)
Trouble Shooting
When the first time you install and configure ecartis, there will be a lot of problem, see this section carefully.Q:ecartis can't work!
A:Turn on debug mode, in /path/to/ecartis/ecartis.cfg, uncomment "debug = 0" and set it large than 0, for example 10(the highest level), ecartis will log anything it do and see in ecartis.log, look at it for detail error.
Q:ecartis always send email as unkonw sender domain
A:check your system hostname, if it's not fqdn, ecartis will append the domain after the hostname, and make it as sender domain, if this name can't resolve, then recipient's MTA will bounce/reject the mail.
Q:How to config ecartis to confirm subscrition?
A: set "subscribe-mode = confirm" and save the config file, that's ok!
Reference
Docs of ecartis
ecartis.hlp in source
ecartis source file
ecartis mailing list
Postfix documentation
Posted by hzqbbc at 05:25 PM | Comments (0)
March 14, 2004
Postfix smtpd的性能瓶颈分析
smtpd performance ...(zz) ( 2004年3月14日9:25 )
List: postfix-users
Subject: Re: smtpd performance
From: Victor.Duchovni () morganstanley ! com
Date: 2004-03-11 2:23:25
Message-ID:
[Download message RAW]
On Wed, 10 Mar 2004, Cesar Bonadio wrote:
> The perl program delivers only one message per time so increasing the
> smtpd number
> did not help.
>
message submission latency is the sum of the "smtpd" and "cleanup"
latencies. The main source of latency in "smtpd" is DNS and table lookups
against LDAP, MySQL, or other "expensive" tables. The main source of
latency in "cleanup" is table lookups, body checks and disk I/O latency.
Time the response delay to each of the protocol interactions connect
(response is 220 banner), "HELO", "MAIL FROM:", "RCPT TO", "DATA", and
".", where is the delay?
--
Viktor.
Posted by hzqbbc at 05:57 AM | Comments (0)
November 14, 2003
Maildrop的若干常见问题集锦（FAQ）
为什么maildrop的quota不起作用？
请确认在编译maildrop时，增加了--enable-maildirquota 选项。此外，也要确定是否给用户增加了Quota的限制，检查用户$HOME/Maildir下的maildirsize文件，如果第一行没有类似：
10240000S,1024C
的信息的话，那说明Quota = unlimit（无限制）。
收件人只有一个时maildrop投递正常，但多个收件人时为什么只有第一个收到？
原因是：Postfix默认会一次性的给多个收件人发邮件，而maildrop每次只处理一个收件人，所以只有第一个收件人能收到邮件，其他的则会被maildrop不留痕迹的丢弃。
解决方法：如果maildrop配置成一个叫maildrop的transport，那么在Postfix的配置文件main.cf里增加一个配置：
maildrop_destination_recipient_limit = 1
这样迫使Postfix每次只处理一个收件人，使maildrop能正常工作。
如何使maildrop支持用户快超出Quota时自动报警？
要达到这个效果，必须使用maildrop 1.5.1以上版本，该版本之后的maildrop支持-w参数，设定警报的上限百分比，例如90%就是-w 90 。在Postfix的master.cf里配置内容可以这样写：
maildrop unix - n n - 50 pipe
flags=DRhu user=vuser argv=maildrop -w 90 -d ${user}@${nexthop}
${recipient} ${user} ${extension} ${nexthop}
同时，还必须在/etc/下放置quotawarnmsg文件，具体可参考maildrop说明文档。
Posted by hzqbbc at 12:33 PM | Comments (0)
January 25, 2003
qmail与Postfix的性能对比测试
一直以来，人们普遍的观点都认为qmail很快，比sendmail快得多，甚至有人还吹嘘说qmail比postfix也快，为了来一个客观一点的评价，对qmail/Postfix进行测试看来是必要的了，至少可以给邮件系统的选择提供一个评价的基准。
在测试postfix同样配置的另外一个机器上装了由hleil提供的rpm包，并咨询了一下他关于qmail的问题，在此表示感谢。
1.环境
2台配置完全一致的PIII 933*2 + 512M + U160 SCSI 磁盘，qmail-1.03-7。另外系统做了优化，调整了fs.file-max=65535并放开一般限制，将/var目录单独分开。
先使用smtp-sink测试qmail，使用系统帐号，效果是26封/秒，觉得效果不理想。
hleil表示，我做的smtp 测试只测试了smtpd的性能而已，队列并没受到真正考验，因为smtp的速度才是bottleneck。
2.简单结果：
用系统帐号，smtp-sink 配置成发1000封信，10个并发，耗费了56秒。换成100并发，测试失败，加大了qmail的并发数后完成顺利，结果也大概是26封/秒。
然后用postal软件，50个并发连接，每个连接发一封信，qmail配置到400个并发上限。结果就大概是1150-1200封/分钟的样子，怎么提高都不行了。看了一下top，发现kjounald频繁的出现，估计是写操作耗费了很多资源，并根据postfix测试结果推断可能是multilog耗费了很多资源的原因。于是去掉multilog记录的功能，qmail不使用log系统，再测试：
[root@ns2 postal-0.61]# ./do.sh
time,messages,data(K),errors,connections,SSL connections
14:52,2029,2767,0,2078,0
14:53,2056,2684,0,2056,0
14:54,2061,2785,0,2062,0
14:55,2022,2686,0,2021,0
14:56,1998,2684,0,1998,0
14:57,2006,2666,0,2007,0
14:58,1971,2644,0,1971,0
14:59,1971,2584,0,1970,0
15:00,1942,2596,0,1943,0
15:01,1984,2655,0,1984,0
15:02,1957,2567,0,1957,0
很明显速度提升1倍左右，但部分测试出现could not creat qq(队列）的问题，看不出具体的原因。估计是qmail没足够资源去创建队列，或者速度太快队列出了问题？
然后再将/etc/passwd的信息转换成cdb提高查询速度：
qmail-pw2u < /etc/passwd > /var/qmail/users/assign
qmail-newu
然后再测试，结果如下：
[root@ns2 postal-0.61]# ./do.sh
time,messages,data(K),errors,connections,SSL connections
16:42,2118,2858,0,2167,0
16:43,2095,2788,0,2096,0
16:44,2114,2790,0,2114,0
16:45,2245,3002,0,2244,0
16:46,2233,2971,0,2233,0
16:47,2165,2891,0,2166,0
16:48,1977,2637,0,1976,0
16:49,1999,2645,0,2000,0
注入速度似乎提高了5%-10%。
hleil对上述测试结果认为：这个测试是测试smtp的注入，并没办法真正考验队列，如上所发现的那样，队列应该没有受到特别的考验，即便是fail to create queue也可能是其他原因。必须是mx-mx真实环境下才有意义，而且一个smtp过程比现在的复杂。
注意：原文我记得不是太清晰，只是转述hleil说的内容。
当时我认为这样的测试起码还是有点效果的，首先就是测试了邮件dos情况。如果出现dos的时候，不管mta用何方法，只要支持住就可以了。如果qmail真的如我测试这样容易出现队列crash的话，是比较危险的。
但似乎这个问题不是那么严重，有那么多qmail用户，如果真的那么多DOS攻击，那么多qmail出问题，一定有解决办法的。
其次就是log的问题，去掉log后速度提高很多，可以说明必须将log的开销尽量降低，但一个产品系统必须有log。这个是必须解决的问题。
另外，由于qmail的队列设计，每封处理的邮件至少要在队列中建立3个文件，因此write的i/o操作频繁很多（top的时候就看到kjounald占用的cpu资源明显高于postfix的测试环境），可能会造成一定bottleneck（尤其是log不是async写的时候更明显，通过简单的调整为非同步写的话就能提高很多速度，至少注入快了很多）
从2003年这个测试结果说明了如下的问题：
1.这个结果比较的只是smtp注入速度
2.这个结果主要是体现了若干MTA的I/O消耗及高流量下的并发处理能力
3.从结果可推断，postfix的fsync()操作最少，而qmail的fsync()较多，从体系结构及设计来说，qmail需要的是一个能立刻将数据同步(sync)到磁盘的文件系统，如果是使用了softupdates或async则在系统崩溃或掉电时，造成丢信。
也就是说，在bottleneck上，同样的文件系统及磁盘子系统，postfix使用更少资源，如果cpu足够快，能够忽略处理队列耗费cpu的时间的话，那么postfix的队列应该比qmail快而稳定。
最后，测试过程中qmail的队列出现过crash，而postfix从未发生过。
参考资料：
一个专门做测试的网站
Serverwatch
Postfix vs qmail
Posted by hzqbbc at 08:22 AM | Comments (1)
January 18, 2003
D. J. Bernstein 和Wietse Venema有关qmail的争论
两人高手过招，一个mta所需要做的工作不多，也不至于什么都干吧??限制smtp log大小是log的事.我觉得。当然,反过来看，mta如果能自己限制，也不错。类似qmail的配套工具一样。
以下是wietse的邮件，关于djb的写的slander回应.
http://groups.google.com/groups?q=artificial+limit+postfix+log+group:mailing.postfix.users&hl=zh-CN&lr=&ie=UTF-8&inlang=zh-CN&selm=9t130n%2412ha%241%40FreeBSD.csie.NCTU.edu.tw&rnum=1
DJB对venema的说法
http://cr.yp.to/qmail/venema.html
另外一个maillist上的邮件，包含了一个简单的patch
http://lists.q-linux.com/pipermail/plug-misc/2001-November/000963.html
Posted by hzqbbc at 07:18 PM | Comments (0)
在足够好的硬件条件下Postfix比qmail更快的原因分析
经过实际的测试我也发现Postfix比qmail快（在较平等的条件下比较）。究其原因，主要是因为磁盘I/O 的差异，Postfix的磁盘I/O原则上比qmail少耗用资源，仅1/3左右，所以速度原则上应该快3倍。
以下是wietse的解释。
以下是postfix作者的原话，就偶自己的小知识，对硬盘写操作越多性能越低。it's true..
===================================================================
寄件者：Wietse Venema (wietse@porcupine.org)
主旨：Re: performance tuning
View this article only
新闻群组：mailing.postfix.users
日期：2001-02-15 17:34:07 PST
jet:
> I'm trying to dump e-mails into postfix via smtp, but can only achieve
> speeds of about 15-20 mails/second (locally).
>
> I've tried spawning multiple outgoing-mail scripts, and tweaked with them,
> but always end up in the 15-20 mail/second range.
This is pretty much single disk Postfix performance. Most other
mailers are much, much, worse than this.
The bottle neck is your disk.
When I first tested Postfix against qmail, Postfix was 3x faster
on local and LAN benchmarks because Postfix uses one queue file
where qmail uses three. The create/delete operations are much,
much, more time consuming than reading and writing the content.
In order to make Postfix fast you need to avoid queue file
create/delete operations. Multiple recipients per queue file are
a big win. With one recipient per queue file Postfix is starved
because the disk is too slow.
Postfix does mostly random writes, and it actually has to wait
until a file is safe on disk. It's not allowed to wait until dirty
blocks are synced automatically.
In order to process more mails per unit time, you need faster disks.
Well, that is not possible.
You can, however, spread the load over multiple spindles. Don't
use RAID5 because it still has single-disk performance for applications
that produce random writes like Postfix does.
Another possibility is to use non-volatile RAM of the type that
used to be sold for SUN file servers. It speeds up disk access of
random writes by an order of magnitude, because writes can be sorted
for speed without loss of reliability. Next to solid-state disks
it is the best thing you can do.
Wietse
--
>My test machine is a:
> AMD Athlon Thunderbird 800 mHz
> 128 megs memory
not enough memory for sending 50K msgs/hour. You'll need 512 or more
to get 400+ SMTP processes into memory.
You will also have to increase maxfiles and maxfilesperprocess with sysctl.
I have a client with a listar announcement list machine hitting 40
msgs/hour on a P500 + 512 megs RAM and just one IDE disk. I think 50
is reachable, but 40 is more than enough for him.
> UDMA66 IDE drive
Better to have two IDE master drives on two separate IDE channels,
one for system + logging and the other for mailq, and maybe even a
third for mailbox storage, but that means master/slave disk on IDE
which is slower.
Of course, IDE is slower than SCSI, and SCSI cards with 64 or 128
megs on-board buffer would help the disk channel congestion, which
Wietse says is the limit to an MTA.
好象wietse说不要用raid5哦。。因为也是random write，这样的效能低。确实也是。因为如果是连续的有规律的写，速度会块得多的。。
Posted by hzqbbc at 01:50 PM | Comments (0)
Postfix+ldap的性能测试及分析
ChangeLog
·2005-04-16 进行内容整理 ·2005-04-10 搜集到blog的Postfix专栏 ·2003-01-18 第一版 发表于http://www.hzqbbc.com/forum/
1.测试平台
PIII 933 × 2 + 512M + SCSI 18G
Postfix 2.0.0.1 + openldap 2.0.11(默认安装) + virtual 投递
2.使用HASH格式作为查询表（lookukp table）时,SMTP 注入速度大概都有5000多封信/分钟。最高能达到87.5封/s。
这个时候采用的方式为只测试信头,100个并发连接。将postfix的默认进程限制提升到500，即：default_process_limit = 500，这也加大了并发速度。其中对syslogd采用了异步写优化(在设置的目录前加-号) 也能大幅度提高速度。
mail.* -/var/log/maillog
3.将用户信息保存在ldap里，SMTP注入速度基本都有2500-2800/分钟,再简化了lookup的方式，transport_maps采用固定文本的方式,只有virtual_mailbox_maps才用ldap，这样速度提高到3200-3500封/分钟左右。测试recipients地址固定为10个
4.在ldap里存放50万条用户记录。进行500,1000,2000地址的并发测试，结果发现SMTP注入速度和使用的用户地址多少关系不大，以下是2000地址的结果:
Postal result: time,msg/m,KB,error,connection,SSL 10:26,3779,1516,2,1930,0 10:27,3889,1556,0,1953,0 10:28,3987,1592,0,1997,0 10:29,4174,1667,0,2067,0 10:30,4308,1720,0,2166,0 10:31,4039,1613,0,1986,0 10:32,2660,1061,0,1338,0 10:33,2436,972,0,1216,0
SMTP注入速度比较快，这里信件内容只包含了信头，采用50个并发连接，每个连接只发2-3封信。感觉效果还是不错的。不过10:32之后速度开始有点问题了，我检查了/var/spool/postfix目录，发现incoming的数量大幅提高，估计是cleanup不能那么快处理incoming的信件，使队列堆积起来；但也可能是ldap反应速度开始变慢。
小结：性能优化技巧
·关闭ldap的log使之=0·将maillog等需要频繁进行write/sync的日志设置为async write，方法见上文
·调整postfix的default_process_limit > 150
·调整/var/spool/postfix目录的属性为async，使用chattr命令
5. 50个并发2000地址，邮件只包含信头，每个连接3封信的结果：
11:17,3613,1447,0,1849,0 11:18,3876,1547,0,1930,0 11:19,4088,1634,0,2080,0 11:20,3920,1566,0,1977,0 11:21,4057,1621,0,2021,0 11:22,3946,1573,0,1960,0 11:23,3316,1324,0,1695,0 11:24,2576,1030,0,1295,0 11:25,2889,1155,0,1439,0 ^^^^^^^^^^^^^^^^^^^^^^* 11:26,3984,1596,0,1990,0 11:27,3986,1588,0,1993,0 11:28,4164,1662,0,2064,0 11:29,3973,1591,0,1986,0 11:30,3991,1592,0,1982,0 11:31,3521,1406,0,1758,0 11:32,2590,1035,0,1297,0 ^^^^^^^^^^^^^^^^^^^^^^* 11:33,2655,1058,0,1322,0 11:34,2745,1097,0,1348,0
注意^^*所标记的时间段，SMTP注入性能大幅度下降。检查/var/spool/postfix目录，发现qmgr已达到处理上限了，该上限由qmgr_message_active_limit 参数设置。故此incoming开始增长很多，Postfix开始放慢处理速度。
以上所有测试都是在default_destination_concurrency_limit = 10的条件下完成的，所以在virtual进行本地mailbox投递时，并发数很少，而active目录增长得很快，没有足够多的virtual进行投递，结果是无法快速地将邮件写到硬盘。
6.提升default_destination_concurrency_limit：
default_destination_concurrency_limit = 1000
此时出现另外一个问题：active队列大概只维持在200-300百封信的样子，但defer/defered目录开始有所增长，并且SMTP注入速度下降，大概只有1850封信/分钟的速度。此时virtual的并发数目达到了400个！
可以推断，这是因为virtual投递速度和数量大幅度增加，致使磁盘写(write)操作过于频繁，使磁盘I/O性能吃紧，导致性能下降。
解决办法：使用分布式存储，利用mailswitch这一类技术，将最终的邮件投递由存储机器上的软件完成。
将这个default_destination_concurrency_limit改成100，速度提升了一些，但很多出现了很多lookup failure：
time,messages,data(K),errors,connections,SSL connections
Server error:451 <user10798@bigmail.hzqbbc.com>: Temporary lookup failure
.
Server error:451 <user10700@bigmail.hzqbbc.com>: Temporary lookup failure
.
11:57,2773,1124,2,1459,0
11:58,2302,920,0,1156,0
11:59,2492,996,0,1264,0
12:00,2204,879,0,1087,0
Server error:451 <user10380@bigmail.hzqbbc.com>: Temporary lookup failure
.
Server error:451 <user11876@bigmail.hzqbbc.com>: Temporary lookup failure
.
12:01,2311,924,2,1146,0
Server error:451 <user10559@bigmail.hzqbbc.com>: Temporary lookup failure
.
12:02,2487,993,1,1265,0
12:03,2284,912,0,1112,0
12:04,1515,602,81,808,0
^^ 81个错误
12:05,1969,787,11,1012,0
12:06,2139,858,0,1073,0
12:07,2174,868,0,1095,0
12:08,2214,884,0,1093,0
12:09,2252,900,0,1118,0
Server error:451 <user11171@bigmail.hzqbbc.com>: Temporary lookup failure
并发降至50：
default_destination_concurrency_limit = 50
测试的log：
Server error:451 <user10062@bigmail.hzqbbc.com>: Temporary lookup failure . 12:21,2645,1064,2,1379,0 12:22,3033,1211,0,1503,0 12:23,2996,1206,0,1472,0 Server error:451 <user10413@bigmail.hzqbbc.com>: Temporary lookup failure . 12:24,2911,1152,1,1455,0 12:25,2716,1093,0,1359,0 12:26,2844,1131,0,1434,0 12:27,2748,1096,0,1360,0 12:28,2738,1107,0,1396,0 12:29,2652,1051,0,1352,0 12:30,2759,1108,0,1355,0 Server error:451 <user11773@bigmail.hzqbbc.com>: Temporary lookup failure . Server error:451 <user11684@bigmail.hzqbbc.com>: Temporary lookup failure . Server error:451 <user10859@bigmail.hzqbbc.com>: Temporary lookup failure . Server error:451 <user11567@bigmail.hzqbbc.com>: Temporary lookup failure . Server error:451 <user11039@bigmail.hzqbbc.com>: Temporary lookup failure . Server error:451 <user11336@bigmail.hzqbbc.com>: Temporary lookup failure . Server error:451 <user10781@bigmail.hzqbbc.com>: Temporary lookup failure . 12:31,2472,987,13,1253,0
Postal测试时打开100个并发,每个邮件最大10k,每个连接1封,发5000封，产生了非常多的failure，不过速度还不错。
接下来Postal 使用50并发，每并发1个邮件，打开了openLDAP cache支持：
time,messages,data(K),errors,connections,SSL connections
23:03,3540,19196,46,3685,0
23:04,4246,23136,0,4247,0
23:05,4182,22873,19,4194,0
^^19个错误
23:06,3736,19922,74,3808,0
^^ 70多个错误!
23:33,3683,3359,0,3733,0
23:34,3935,3632,0,3935,0
23:35,4041,3664,0,4041,0
23:36,3990,3636,0,3989,0
23:37,4164,3798,0,4165,0
23:38,3872,3512,0,3872,0
23:39,3210,2945,0,3209,0
23:40,2624,2400,0,2625,0
Server error:451 <user10717@bigmail.hzqbbc.com>: Temporary lookup failure
.
Server error:451 <user10614@bigmail.hzqbbc.com>: Temporary lookup failure
.
Server error:451 <user10634@bigmail.hzqbbc.com>: Temporary lookup failure
.
23:41,2502,2286,3,2505,0
23:42,2760,2508,0,2759,0
Server error:451 <user11449@bigmail.hzqbbc.com>: Temporary lookup failure
.
Server error:451 <user11660@bigmail.hzqbbc.com>: Temporary lookup failure
.
23:43,2528,2305,2,2531,0
23:44,2078,1972,0,2078,0
Server error:451 <user10298@bigmail.hzqbbc.com>: Temporary lookup failure
.
Server error:451 <user10706@bigmail.hzqbbc.com>: Temporary lookup failure
.
Server error:451 <user11515@bigmail.hzqbbc.com>: Temporary lookup failure
.
Server error:451 <user10499@bigmail.hzqbbc.com>: Temporary lookup failure
.
23:45,1520,1387,4,1524,0
23:46,1715,1536,0,1715,0
23:47,1475,1362,0,1475,0
23:48,1508,1371,0,1507,0
可见active目录的上限已经达到了，所以SMTP注入速度开始下降，邮件已经开始处理不过来了。
调整Postal 参数，使用50并发，每个并发1个msg，尽最大能力发送，打开openLDAP cache模式，qmgr_messages_active_limit=50000，测试持续了近一个小时，日志如下：
[root@ns1 postal-0.61]# ./do.sh time,messages,data(K),errors,connections,SSL connections Server error:451 <user11017@bigmail.hzqbbc.com>: Temporary lookup failure . Server error:451 <user10714@bigmail.hzqbbc.com>: Temporary lookup failure . Server error:451 <user10989@bigmail.hzqbbc.com>: Temporary lookup failure . 23:52,3663,3353,3,3714,0 23:53,3784,3454,0,3786,0 23:54,3747,3409,0,3747,0 23:55,3861,3498,0,3861,0 23:56,3922,3577,0,3922,0 23:57,3822,3491,0,3822,0 23:58,3748,3429,0,3747,0 23:59,3608,3264,0,3609,0 00:00,3320,3027,0,3319,0 00:01,3410,3117,0,3411,0 00:02,3317,3042,0,3316,0 00:03,3580,3286,0,3581,0 00:04,3507,3205,0,3507,0 00:05,3475,3114,0,3475,0 00:06,3259,2995,0,3258,0 00:07,3186,2912,0,3186,0 00:08,3552,3216,0,3553,0 00:09,2887,2635,0,2887,0 00:10,2517,2292,0,2517,0 00:11,2721,2505,0,2721,0 00:12,2574,2317,0,2574,0 00:13,2587,2357,0,2587,0 00:14,2573,2342,0,2573,0 00:15,2704,2463,0,2704,0 00:16,2647,2405,0,2647,0 00:17,2585,2383,0,2585,0 00:18,2421,2235,0,2421,0 00:19,2573,2349,0,2572,0 00:20,2378,2170,0,2378,0 Server error:451 <user11351@bigmail.hzqbbc.com>: Temporary lookup failure . Server error:451 <user11458@bigmail.hzqbbc.com>: Temporary lookup failure . ...... 期间Temporary lookup failure有12个 00:21,2330,2144,15,2346,0 00:22,2610,2376,0,2610,0 00:23,2551,2317,0,2550,0 00:24,2586,2380,0,2587,0 00:25,2530,2282,0,2528,0 00:26,2558,2306,0,2560,0 00:27,2444,2232,0,2444,0 00:28,2509,2250,0,2508,0 00:29,2523,2293,0,2524,0 00:30,2445,2210,0,2445,0 00:31,2555,2345,0,2554,0 00:32,2455,2225,0,2456,0 00:33,2477,2266,0,2476,0 Server error:451 <user10490@bigmail.hzqbbc.com>: Temporary lookup failure . Server error:451 <user10822@bigmail.hzqbbc.com>: Temporary lookup failure . 00:34,2236,2038,2,2238,0 00:35,2245,2061,0,2245,0 00:36,2536,2311,0,2537,0 00:37,2210,2004,0,2210,0 00:38,2385,2150,0,2385,0 00:39,2463,2242,0,2463,0 00:40,2496,2291,0,2495,0 00:41,2471,2261,0,2471,0 00:42,2422,2208,0,2422,0 00:43,2370,2159,0,2371,0 00:44,2387,2197,0,2386,0 00:45,2203,2028,0,2203,0 00:46,2494,2301,0,2494,0 00:47,2313,2133,0,2314,0 00:48,2255,2016,0,2255,0 00:49,2428,2236,0,2428,0 00:50,2327,2113,0,2326,0 00:51,2436,2244,0,2437,0 00:52,2334,2127,0,2334,0 00:53,2399,2202,0,2398,0 00:54,2271,2078,0,2272,0 00:55,2255,2046,0,2255,0 00:56,2397,2205,0,2397,0 00:57,2392,2183,0,2392,0 00:58,2364,2161,0,2364,0 00:59,2241,2031,0,2240,0 01:00,2354,2137,0,2355,0 01:01,2223,2000,0,2223,0 01:02,2326,2118,0,2325,0 01:03,2343,2155,0,2344,0 01:04,2388,2209,0,2388,0
队列里积压了13万封信，70多分钟内注入了22万封信，正确投递并写入用户$HOME目录的有95000多封，由于人为限制了投递的并发数，因此这个结果也在意料之内了。
[root@ns2 postfix]# postsuper -d ALL postsuper: Deleted: 137410 messages [root@ns2 postfix]# du -sk /home/domains/bigmail.hzqbbc.com/ 392124 /home/domains/bigmail.hzqbbc.com [root@ns2 postfix]# find /home/domains/bigmail.hzqbbc.com/ | wc -l 95349
6. 新一轮测试
Postal 使用50/100个并发，1-3封邮件/每连接，将Postfix升级到Snapshot 2.0.2，并打开proxymap daemon，做如下的测试：
[root@ns1-bjcnc postal-0.61]# ./do.sh time,messages,data(K),errors,connections,SSL connections Server error:451 <user11097@bigmail.hzqbbc.com>: Temporary lookup failure . Server error:451 <user10170@bigmail.hzqbbc.com>: Temporary lookup failure . Server error:451 <user11333@bigmail.hzqbbc.com>: Temporary lookup failure . 14:02,3743,3366,3,1880,0 14:03,4244,3809,0,2129,0 14:04,4345,3915,0,2204,0 14:05,4187,3790,0,2114,0
[root@ns1-bjcnc postal-0.61]# ./do.sh time,messages,data(K),errors,connections,SSL connections 14:10,4380,3973,5,2289,0 14:11,4481,4065,0,2215,0 14:12,4541,4083,0,2260,0 . 14:13,2741,2455,14,1389,0 14:14,3927,3549,0,1949,0 14:15,3611,3235,0,1807,0 14:16,3634,3304,0,1796,0 14:17,3889,3475,0,1933,0
[root@ns1-bjcnc postal-0.61]# ./do.sh time,messages,data(K),errors,connections,SSL connections 14:22,1838,1644,0,961,0 Server error:451 <user11857@bigmail.hzqbbc.com>: Temporary lookup failure . Server error:451 <user11434@bigmail.hzqbbc.com>: Temporary lookup failure . 14:23,1910,1721,2,959,0 Server error:451 <user10268@bigmail.hzqbbc.com>: Temporary lookup failure . Server error:451 <user11688@bigmail.hzqbbc.com>: Temporary lookup failure . 14:24,1989,1769,2,1007,0 Server error:451 <user11264@bigmail.hzqbbc.com>: Temporary lookup failure . Server error:451 <user11066@bigmail.hzqbbc.com>: Temporary lookup failure . 14:25,1828,1652,2,915,0
[root@ns1-bjcnc postal-0.61]# ./do.sh Can't open config file "conver". Doing no expansion. time,messages,data(K),errors,connections,SSL connections Server error:451 <user11182@bigmail.hzqbbc.com>: Temporary lookup failure . 14:27,4110,3721,1,2105,0 14:28,289,221,0,149,0
[root@ns1-bjcnc postal-0.61]# cat do.sh
#!/bin/sh MAXMSGSIZE=1 PROC=50 MSGPERCONN=3 MSGPERMIN=5000 SSLPERCENT=0 SMTP_HOST=210.82.193.91 ./postal -m $MAXMSGSIZE -p $PROC -c $MSGPERCONN -r $MSGPERMIN -a \ -b netscape $SMTP_HOST myuser conver
[root@ns1-bjcnc postal-0.61]# ./do.sh time,messages,data(K),errors,connections,SSL connections . 14:38,4213,3793,10,2176,0 14:39,4308,3869,0,2174,0 14:40,4296,3835,0,2160,0 14:41,4261,3850,0,2136,0 14:42,4342,3930,0,2168,0 14:43,4258,3843,0,2139,0 14:44,4172,3769,0,2067,0 14:45,4111,3684,0,2052,0 14:46,3920,3577,0,1961,0 14:47,3987,3590,0,1987,0 14:48,3729,3365,0,1851,0 14:49,3702,3318,0,1842,0 14:50,3745,3371,0,1879,0 14:51,3961,3558,0,1969,0 14:52,3753,3415,0,1876,0 14:53,3498,3139,0,1781,0 14:54,3337,3001,0,1661,0 14:55,2671,2398,0,1341,0 14:56,2722,2453,0,1365,0 14:57,2648,2388,0,1342,0 14:58,2466,2199,7,1227,0 14:59,2601,2344,0,1299,0
通过上述多次测试表明，在打开proxymap daemon后，SMTP注入速度明显提高，结果是提升了10%-15%。
接下来将virtual投递并发限制
[root@ns1-bjcnc postal-0.61]# ./do.sh
time,messages,data(K),errors,connections,SSL connections
20:44,4289,3902,0,2166,0
20:45,4329,3874,0,2169,0
Server error:451
.
期间产生12个Temporary lookup failure
20:46,3993,3594,13,1983,0
20:47,4422,4018,0,2216,0
20:48,4639,4163,0,2323,0
以上部分是final_destination 限制在20个并发virtual
========================================
接着改成10个，速度提升。说明bottleneck仍然在disk i/o（投递多了就明显影响i/o）ldap仍然有潜力可挖掘。
[root@ns1-bjcnc postal-0.61]# ./do.sh
Can't open config file "conver". Doing no expansion.
time,messages,data(K),errors,connections,SSL connections
20:53,5172,4690,0,2636,0
20:54,5074,4530,0,2529,0
20:55,4988,4529,0,2494,0
20:56,4856,4376,0,2456,0
20:57,5114,4602,0,2530,0
20:58,4720,4251,0,2345,0
20:59,4490,4038,0,2230,0
21:00,4716,4234,0,2363,0
21:01,4355,3931,0,2188,0
21:02,4488,4046,0,2241,0
21:03,4495,4062,0,2271,0
21:04,4680,4224,0,2363,0
21:05,4295,3882,0,2141,0
21:06,3822,3436,0,1921,0
21:07,2899,2597,0,1425,0
21:08,2863,2587,0,1447,0
完整结果如下：
[root@ns1-bjcnc postal-0.61]# ./do.sh
Can't open config file "conver". Doing no expansion.
time,messages,data(K),errors,connections,SSL connections
20:53,5172,4690,0,2636,0
20:54,5074,4530,0,2529,0
20:55,4988,4529,0,2494,0
20:56,4856,4376,0,2456,0
20:57,5114,4602,0,2530,0
20:58,4720,4251,0,2345,0
20:59,4490,4038,0,2230,0
21:00,4716,4234,0,2363,0
21:01,4355,3931,0,2188,0
21:02,4488,4046,0,2241,0
21:03,4495,4062,0,2271,0
21:04,4680,4224,0,2363,0
21:05,4295,3882,0,2141,0
21:06,3822,3436,0,1921,0
21:07,2899,2597,0,1425,0
21:08,2863,2587,0,1447,0
21:09,2887,2600,0,1473,0
21:10,2772,2494,0,1372,0
21:11,2106,1896,0,1033,0
21:12,1188,1079,0,584,0
21:13,1493,1355,0,742,0
21:14,1696,1540,0,828,0
21:15,1860,1677,0,920,0
21:16,1581,1428,0,784,0
21:17,2022,1837,0,1018,0
21:18,1477,1317,0,735,0
21:19,1511,1375,0,760,0
21:20,1877,1691,0,950,0
21:21,1459,1314,0,725,0
21:22,1494,1321,0,740,0
21:23,1526,1386,0,756,0
21:24,1309,1188,0,643,0
21:25,1446,1288,0,751,0
21:26,1396,1252,0,722,0
21:27,1431,1260,0,713,0
21:28,1312,1199,0,663,0
21:29,1290,1149,0,658,0
21:30,1190,1078,0,608,0
21:31,949,864,0,464,0
21:32,969,875,0,484,0
21:33,1210,1095,0,625,0
21:34,1384,1255,0,681,0
21:35,1332,1207,0,645,0
21:36,1018,928,0,503,0
21:37,882,795,0,442,0
21:38,1222,1095,0,613,0
21:39,1171,1051,0,581,0
21:40,1222,1123,0,609,0
21:41,1223,1088,0,606,0
21:42,1220,1086,0,607,0
21:43,1328,1208,0,665,0
21:44,1409,1263,0,719,0
21:45,1488,1349,0,723,0
21:46,1237,1115,0,628,0
21:47,1160,1035,0,586,0
21:48,1270,1146,0,626,0
21:49,1400,1261,0,708,0
到了21:10开始，actvie已接近上限，处理速度开始来不及了。此外，postal里限制了一分钟只发5000封信，所以本次测试的峰值51xx信/分未必可信，应能提高更多。
提高注入速度的新测试
笔者根据在IBM developerWorks上得到的灵感，将Postfix queue的所在的磁盘分区挂接参数(mount)改为noatime，这样实际减少了磁盘的写入操作（noatime表示对于访问操作，不更新文件的atime记录，这可减少磁盘的write操作），提高了5%的性能。通过elvtune调整了磁盘write操作的延时，可明显提高速度。
注意使用linux的iostat及相关的I/O监视工具。。以下是调整后的数值：
time,messages,data(K),errors,connections,SSL connections 23:57,5325,4804,0,2703,0 23:58,5480,4905,0,2702,0 23:59,5160,4624,0,2573,0 00:00,5211,4689,0,2609,0 00:01,4918,4441,0,2453,0 00:02,4872,4377,0,2395,0 00:03,5039,4564,0,2492,0 00:04,4694,4223,0,2308,0
效果有了明显进展。而再进行多次测试发现，有些时段可达到峰值63xx封/分，估计这因为是Postfix的Multiplex技术及磁盘I/O的cache所带来的收益，multiplex可复用已打开的smtp通道，减少了进程的创建带来的开销。
[root@ns1-bjcnc postal-0.61]# ./do.sh time,messages,data(K),errors,connections,SSL connections 22:23,6330,5721,0,3213,0 22:24,5592,5038,0,2783,0 22:25,5069,4555,0,2509,0 22:26,4624,4203,0,2333,0 22:27,4778,4311,0,2438,0 22:28,4396,3943,0,2215,0
[root@ns1-bjcnc postal-0.61]# ./do.sh Can't open config file "conver". Doing no expansion. time,messages,data(K),errors,connections,SSL connections 22:32,5121,4615,0,2611,0 22:33,4696,4263,0,2352,0 22:34,4532,4094,0,2277,0 22:35,4622,4138,0,2297,0 22:36,5006,4515,0,2493,0 22:37,4487,4052,0,2284,0 22:38,4595,4121,0,2300,0 22:39,4669,4221,0,2300,0 22:40,4381,3961,0,2149,0 22:41,4217,3781,0,2100,0
[root@ns1-bjcnc postal-0.61]# ./do.sh time,messages,data(K),errors,connections,SSL connections 22:44,5148,4642,0,2629,0 22:45,4868,4390,0,2414,0 22:46,4890,4392,0,2451,0 22:47,4803,4306,0,2419,0 22:48,5005,4552,0,2493,0 22:49,4435,3976,0,2221,0 22:50,4515,4055,0,2246,0 22:51,4666,4187,0,2336,0 22:52,4556,4105,0,2283,0 22:53,4472,3979,0,2239,0 22:54,4136,3749,0,2057,0 22:55,4649,4172,0,2355,0 22:56,4304,3895,0,2142,0 22:57,4055,3607,0,2008,0 22:58,2692,2425,0,1335,0 22:59,2690,2402,0,1346,0 23:00,2796,2533,0,1379,0 23:01,2738,2448,0,1396,0 23:02,2522,2287,0,1252,0 23:03,2574,2337,0,1286,0
[root@ns1-bjcnc postal-0.61]# ./do.sh time,messages,data(K),errors,connections,SSL connections 23:08,6084,5525,0,3068,0 23:09,5834,5254,0,2912,0 23:10,4683,4205,0,2303,0 23:11,4859,4376,0,2453,0 23:12,4265,3814,0,2142,0 23:13,4595,4179,0,2282,0 23:14,4466,4002,0,2217,0 23:15,4438,3982,0,2201,0 23:16,4463,4033,0,2242,0 23:17,4592,4123,0,2327,0 23:18,4234,3802,0,2097,0 23:19,4545,4049,0,2265,0 23:20,3853,3437,0,1925,0 23:21,2708,2417,0,1349,0 23:22,2730,2452,0,1363,0
[root@ns1-bjcnc postal-0.61]# ./do.sh Can't open config file "conver". Doing no expansion. time,messages,data(K),errors,connections,SSL connections 23:42,5327,4852,0,2721,0 23:43,5201,4653,0,2631,0 23:44,5377,4815,0,2705,0 23:45,4899,4431,0,2398,0 23:46,4863,4378,0,2421,0 23:47,4885,4382,0,2460,0 23:48,5012,4523,0,2496,0 23:49,5038,4501,0,2561,0 23:50,4898,4427,0,2431,0 23:51,4888,4400,0,2494,0 23:52,4856,4350,0,2398,0 Server error:451 <user10945@bigmail.hzqbbc.com>: Temporary lookup failure . Server error:451 <user11208@bigmail.hzqbbc.com>: Temporary lookup failure . Server error:451 <user10766@bigmail.hzqbbc.com>: Temporary lookup failure . Server error:451 <user11461@bigmail.hzqbbc.com>: Temporary lookup failure . 23:53,4584,4115,4,2306,0 23:54,4693,4243,0,2336,0
接下来，将保存对列的磁盘sda3进行调整:
(queue) r_d w_d from 4096/8192 -> 4096/2048
并清理队列，重新测试，注入速度快了不少。
没调整磁盘R/W延迟之前的结果（r/w -> 8192/4096) ：
time,messages,data(K),errors,connections,SSL connections 00:08,5280,4790,0,2704,0 00:09,5135,4570,0,2546,0 00:10,5280,4810,0,2635,0 00:11,5082,4572,0,2547,0 00:12,5195,4648,0,2596,0 00:13,5053,4591,0,2549,0 00:14,5089,4568,0,2583,0 00:15,4937,4438,0,2485,0 00:16,4727,4312,0,2340,0 00:17,4396,3958,0,2224,0 00:18,4860,4353,0,2413,0 00:19,4451,3994,0,2227,0 00:20,4582,4132,0,2300,0 00:21,2809,2486,0,1439,0
调整为r/w -> 4096/2048，结果连续出现了6次6xxx/分:
[root@ns1-bjcnc postal-0.61]# ./do.sh time,messages,data(K),errors,connections,SSL connections 00:23,6247,5597,0,3174,0 00:24,6581,5891,0,3280,0 00:25,6544,5898,0,3236,0 00:26,6457,5737,0,3235,0 00:27,6283,5690,0,3137,0
最后结论
LDAP的速度不容质疑，在硬件足够好的前提下，系统的瓶颈一般在磁盘的I/O上，LDAP一般较磁盘I/O要快。而如何调整Postfix的注入并发数、系统总进程数、投递的并发数等，需要大量的测试，找到在具体平台、具体硬件下所能达到的最高效能，继而定出合理的数值。作为一个繁忙的邮件系统，磁盘的I/O能力，尤其是队列所在的磁盘I/O能力要求非常高。在不增加硬件投入的前提下，善于利用iostat, vmstat及相关I/O分析工具，可以找到磁盘的限制在哪里，利用elvtune调整磁盘的读/写时延(delay)来减少这种限制所带来的性能劣化。
如果有足够的经济实力，可以使用带NVRAM的磁盘系统，使用Ultra 320的磁盘，甚至组成RAID0+1的系统来构成队列。这样可有效的提高速度。有报道称，国外有公司使用Sun的存储设备，在Solaris下实现了sendmail 287封邮件/秒 的处理速度。换言之相当于17220封/分的速度。这得益于它们使用了Sun的高级存储设备，配备了NVRAM的Cache，非常高速的硬盘，相当好的性能调整和优化。
通过这个国外的实例可以说明一点，Sendmail其实性能也可以相当好，关键是看系统的设计和部署人员的能力和经验。
而本文所实现的结果，也反映了Postfix在低端硬件（U160 SCSI磁盘，PIII 933的普通CPU）下效能相当不俗。如果按正常的4000封/分钟的处理速度，那么一天相当于处理576万封普通大小的电子邮件。
原文地址：http://www.hzqbbc.com/forum/read.php?forumid=2&filename=f_73
Posted by hzqbbc at 10:56 AM | Comments (0)
January 01, 2003
配置per user transport实现用户分布存储
高级Postfix配置
大容量邮件系统的关键之一就在于存储，而存储无外乎集中存储和分布存储。
O 集中存储主要是通过光纤磁盘阵列，SAN或iSCSI等统一存储技术，将邮件数据集中或较为集中的存储在一个地方，应用系统共享使用这些数据，所带来的问题主要是如何解决同时请求一个文件的缩定问题
O 分布存储则将用户数据分布在多个存储设备/机器上，主要的手段是通过一定的算法，将邮件分发到存储的设备或主机上。原理上有点象路由器，邮件被路由到了最终的目的地。所带来的问题是管理比较麻烦，但较少共享缩定的问题。
如何利用Postfix实现分布的用户数据存储呢？这里通过per user transport来实现。
在postfix没有release 2.0.0之前这个per user transport的功能只出现在snapshot里，现在好了可以在正式版里使用了。本文所描述的方法适合于：postfix stable > 2.0.0.0或snapshot version > 1.1.11-200205xx
1.描述
什么是per user transport？有什么用？简单来说就是可以为每个用户设置单独的transport，类似原来1.1.x里的domain transport一样，只不过现在的key可以是user+extension@domain.tld了。好处是什么？好处就是可针对不同用户设置特定的transport，例如有些用户使用默认的local transport，而有些则可以使用病毒过滤或maildrop功能，甚至将用户送到nexthop继续进行处理等。可能的应用：病毒过滤、关键字过滤、垃圾邮件过滤、邮件的物理分布或转发等。
自己的感觉：这个功能就类似qmail里的qmqp（中央队列服务器<-->qmtpd/qmqpd机器）或者类似qmail-ldap里的cluster功能。而且要强大得多！！qmail的qmqp/qmtp协议设计得很好，思路也很不错，不过具体的implemenation就不够好。使用postfix可享受最大的灵活性。
如果postfix能考虑完整的实现qmail的qmqp/qmtp等不错的特性，并吸取qmail的优点话，那么qmail的价值则更加大打折扣了。
2.配置思路
# # +------+ # local | | # handling | MTA1 | # layer | | # +------+ # ^ foo@lvs.hzqbbc.com # | # +--------+ +------+ # +------+ | | | | # inbound |mail | ----->|switch |-----> | MTA2 | bar@lvs.hzqbbc.com # email +------+ | | | | # +--------+ +------+ # # 邮件交换 local handling layer
将不同用户的邮件switch到不同的机器，达到分布的目的。配置方法见下文。
3.配置方法：
在mail switch机器上：
postconf -n
alias_database = hash:/etc/postfix/aliases alias_maps = hash:/etc/postfix/aliases command_directory = /usr/sbin config_directory = /etc/postfix daemon_directory = /usr/lib/postfix debug_peer_level = 2 inet_interfaces = all local_recipient_maps = unix:passwd.byname $alias_maps $virtual_mailbox_maps mail_owner = postfix mailq_path = /usr/bin/mailq.postfix manpage_directory = /usr/share/man mydestination = $myhostname,$mydomain,$transport_maps mydomain = LVS.hzqbbc.com myhostname = LVS.hzqbbc.com mynetworks = 192.168.0.0/16, 127.0.0.0/8 myorigin = $mydomain newaliases_path = /usr/bin/newaliases.postfix queue_directory = /var/spool/postfix readme_directory = /etc/postfix/README_FILES sample_directory = /etc/postfix/samples sendmail_path = /usr/sbin/sendmail.postfix setgid_group = postdrop smtpd_banner = $myhostname ESMTP $mail_name Mail-Switch ($mail_version) smtpd_recipient_restrictions = check_relay_domains,reject transport_maps = hash:/mnt/disk/vmail/maps/transport unknown_local_recipient_reject_code = 450
在MTA1和MTA2（这里我定义的机器名字是srv1, srv2)上配置完全一样，除了机器名不同，因此只列出一个srv2的example：
postconf -n
alias_database = hash:/etc/postfix/aliases alias_maps = hash:/etc/postfix/aliases command_directory = /usr/sbin config_directory = /etc/postfix daemon_directory = /usr/lib/postfix debug_peer_level = 2 inet_interfaces = all local_recipient_maps = unix:passwd.byname $alias_maps $virtual_mailbox_maps local_transport = virtual mail_owner = postfix mailq_path = /usr/bin/mailq.postfix manpage_directory = /usr/share/man mydestination = $myhostname,$mydomain, $virtual_mailbox_domains mydomain = LVS.hzqbbc.com myhostname = SRV2.LVS.hzqbbc.com mynetworks = 192.168.0.0/16, 127.0.0.0/8 myorigin = $mydomain newaliases_path = /usr/bin/newaliases.postfix queue_directory = /var/spool/postfix readme_directory = /etc/postfix/README_FILES sample_directory = /etc/postfix/samples sendmail_path = /usr/sbin/sendmail.postfix setgid_group = postdrop smtpd_banner = $myhostname ESMTP $mail_name ($mail_version) transport_maps = hash:/mnt/disk/vmail/maps/vuser_transport unknown_local_recipient_reject_code = 450 virtual_gid_maps = static:250 virtual_mailbox_base = /mnt/disk/vmail virtual_mailbox_domains = lvs.hzqbbc.com virtual_mailbox_maps = hash:/mnt/disk/vmail/maps/mailbox virtual_uid_maps = static:250
之后就是配置了。
/mnt/disk/vmail/maps下有这些文件：
cat mailbox
LVS.hzqbbc.com anything hzq@LVS.hzqbbc.com hzq/Maildir/ ben@LVS.hzqbbc.com ben/Maildir/ foo@LVS.hzqbbc.com foo/Maildir/ bar@LVS.hzqbbc.com bar/Maildir/
cat transport
foo@LVS.hzqbbc.com smtp:[SRV1.LVS.hzqbbc.com] bar@LVS.hzqbbc.com smtp:[SRV2.LVS.hzqbbc.com]
cat vuser_transport
foo@LVS.hzqbbc.com virtual: bar@LVS.hzqbbc.com virtual:
这样，对外部用户只有一个postfix mail-switch/hub。投递给该switch的邮件会转到合适的后端机器（srv1 or srv2）并进行本地投递，整个系统扩容时也特别简单，增加1-n个switch的机器，并增加相应mx记录即可。负载可以比较平均，而不需要使用LVS设备
后端扩充用户及存储也不难，不断增加srvn(n>2)就可以了。这样的体系缺点是效率不够高，耗费机器比较多，但TOC相当低，不需要太多的开发，直接利用postfix的功能并进行合理扩充即可。
Posted by hzqbbc at 09:27 PM | Comments (0)
December 18, 2002
产生mail loop的原因及解决方案
Mail loop是一个很常见，但有时却又让人很难发现故障所在的问题。如何去发现和解决呢？这里以我自己遇到的mail loop问题为例子，来分析一下个中原因。
日志里关于mail loop的内容如下：
Dec 18 18:35:20 smtp postfix/cleanup[9526]: 4846417D5E:
message-id=<20021218223122.C5B6C680A@smtp.hzqbbc.com>
Dec 18 18:35:20 smtp postfix/qmgr[9342]: 4846417D5E:
from=<hzqbbc@hzqbbc.com>, size=1431, nrcpt=4 (queue active)
Dec 18 18:35:20 smtp postfix/local[9585]: C4F5117D5C:
to=<postmaster@192.168.0.27>, relay=local, delay=33,
status=sent (forwarded as 4846417D5E)
Dec 18 18:35:20 smtp postfix/local[9585]: 4846417D5E:
to=<postmaster@192.168.0.27>, relay=local, delay=22,
status=bounced (mail forwarding loop for postmaster@192.168.0.27)
这里ldap的entry是：
dn: mail=postmaster@overtheland.com,virtualDomain=overtheland.com,o=myldap
objectClass: top
objectClass: MailAccount
objectClass: MailAlias
accountActive: TRUE
mailacceptAction: RELAY
mail: postmaster@overtheland.com
homeDirectory: overtheland.com/postmaster
mailbox: overtheland.com/postmaster/Maildir/
quota: 20971520S,1000C
maildrop: postmaster@smtp.overtheland.com
maildrop: root@overtheland.com
maildrop: postmaster@192.168.0.27
maildrop: hzqbbc@redhut.net
相应的ldap设置是：
=================================================================
ldapstaraliasfixup_query_filter = (&(maildrop=%s)(accountActive=TRUE))
ldapstaraliasfixup_result_attribute = mail
~~~~~~~~~~~~
ldapforwardings_query_filter =(&(mail=%s)(accountActive=TRUE))
ldapforwardings_result_attribute = maildrop
~~~~~~~~~~~~~
这里为什么有问题呢？很简单的解释：我发信给root@192.168.0.27。按照alias部分的ldap
设定，信笺应该投递给postmaster@overtheland.com。。确实正常的投递了。不过问题也就
来了。因为还设置了forward部分，而它expect的是maildrop属性的值，这里maildrop有多个值
其中包括了root@192.168.0.27，这样forward时除了给那些非root@192.168.0.27的地址发信外
还会给root@192.168.0.27自己发信。。。这个就产生loop了（看log可看到）
之后。由于root@overtheland.com并不存在机器上（只是一个postmaster@overtheland.com）
的一个alias）。而这时pipe将信笺给了maildrop，maildrop查ldap时并没有办法查得出
root@overtheland.com就是postmaster@overtheland.com（这个是maildrop的ldap部分缺陷，
有待改进）。因此返回user unknow。这个时候自然得给发信者。。。
之后就是很混乱的loop及bounce 信笺了。所以注意一点。forward及alias必须分开来
设置（用不同属性）否则就麻烦了。
而有关类似的mail loop问题的讨论，在Postfix邮件列表上非常多，这里仅举一例：
寄件者：Dirk Datzert (Dirk.Datzert@rasselstein-hoesch.de)
主旨：postfix and ldap
View this article only
新闻群组：mailing.postfix.users
日期：2001-04-10 14:54:04 PST
Hi all,
I have a postfix question with ldap using:
How can I configure aliases_maps and virtual_maps for the following
issue:
a user has a local 'mail'/'mailalternateaddress' with 'uid' and an
'mailforwardingaddress'
I could configure for local delivery:
alias_maps: ldap:aliases
aliases_query_filter = (│(mail=%s)(mailalternateaddress=%s))
aliases_result_attribute = uid
If a user has no local address I can forward:
virtual_maps: ldap:forwarders
forwarders_query_filter = (│(mail=%s)(mailalternateaddress=%s))
forwarders_result_attribute = mailforwardingaddress
Now my question:
How can I configure postfix to do both for one user: delivery local and
forward ?
Thanx,
Dirk
-
To unsubscribe, send mail to majordomo@postfix.org with content
(not subject): unsubscribe postfix-users
第 4 条留言
寄件者：LaMont Jones (lamont@hp.com)
主旨：Re: postfix and ldap
View this article only
新闻群组：mailing.postfix.users
日期：2001-04-10 16:04:07 PST
> aliases_query_filter = (│(mail=%s)(mailalternateaddress=%s))
> aliases_result_attribute = uid
> virtual_maps: ldap:forwarders
> forwarders_query_filter = (│(mail=%s)(mailalternateaddress=%s))
> forwarders_result_attribute = mailforwardingaddress
> Now my question:
>
> How can I configure postfix to do both for one user: delivery local and
> forward ?
result_attribute is a _LIST of attributes, so if you can get the changes
down to one map (merge alias and virtual), then you can just say:
_result_attribute = uid mailforwardingaddress
Untested, your mileage may vary, but we have similar things working in
production.
lamont
Wietse关于mail loop的解释
m 如何防止mail loop？看看wietse如何说的 ( 2003年1月2日21:28 )http://groups.google.com/groups?hl=zh-CN&lr=&ie=UTF-8&inlang=zh-CN&threadm=a3f25j%24urn%241%40FreeBSD.csie.NCTU.edu.tw&rnum=6&prev=/groups%3Fq%3Dmail%2Bloop%2Blist%2Bgroup:mailing.postfix.users%26hl%3Dzh-CN%26lr%3D%26ie%3DUTF-8%26inlang%3Dzh-CN%26selm%3Da3f25j%2524urn%25241%2540FreeBSD.csie.NCTU.edu.tw%26rnum%3D6
You must list myhostname under mydestination:
mydestination = $myhostname localhost.$mydomain ...
Otherwise you will have a mail loop.
Wietse
寄件者：Wietse Venema (wietse@porcupine.org)
主旨：Re: mail loops back to myself
View this article only
新闻群组：mailing.postfix.users
日期：2003-01-18 17:29:08 PST
Is clanetsys.homeunix.net listed in mydestination? See the
main.cf file comments for guidance.
Is clanetsys.homeunix.net listed in virtual_alias_domains?
See the virtual(5) manual page for guidance.
Is clanetsys.homeunix.net listed in virtual_mailbox_domains?
See the VIRTUAL_README file for guidance.
If the domain is not listed in any of the above, then mail
will loop.
Wietse
Posted by hzqbbc at 05:58 PM | Comments (0)
December 09, 2002
Postfix更换Logo了
一直以来，Postfix的logo都没有，现在，终于有一个了！不过不知道是谁设计的呢？感觉看上去还以为是mailman呢，但不管如何，有了就是有了！
请看这里：
具体地址在：
Posted by hzqbbc at 09:58 PM | Comments (1)
December 05, 2002
限制特定用户收发邮件的高级访问控制方法
一般邮件应用稍微有一电规模的企业，大多都会涉及到这样的应用需求：
某销售A只能收到来自163.com及sina.com的邮件 某技术员B只能给内部用户发邮件，不允许向外网发邮件，但可以收到外网邮件 某主管可以收发内、外网邮件 ......
如何实现呢？Postfix 1.x就已经可以通过配置实现这些功能了。
以下是邮件列表里Noel Jones 提出的解决方案：
Hello!
I have the same problem:
1. I need to permit some internal users (not all) to send mail to any
external user (Internet)
2. I need to permite any external user to send mail to some my internal
users (not all)
In fact, the problem is: just "some" of my internal users have permission to
receive mail from and send mail to the Internet.
How can I solve this?
寄件者：Noel Jones (njones@megan.vbhcs.org)
主旨：Re: RES: User Restriction.....
View this article only
新闻群组：mailing.postfix.users
日期：2002-12-04 15:18:04 PST
At 05:39 PM 12/4/02 -0200, raissa.medeiros@caixa.gov.br wrote:
>1. I need to permit some internal users (not all) to send mail to any
>external user (Internet)
>2. I need to permite any external user to send mail to some my internal
>users (not all)
>
>In fact, the problem is: just "some" of my internal users have permission
>to receive mail from and send mail to the Internet.
>
>How can I solve this?
>
>Thanks,
>Raissa
Restricting also who can receive mail from the internet changes the example
I posted a few minutes ago.
In the example below, the same list of restricted_users is used for
controlling both who can send and who can receive internet mail. If you
don't require the local_plus feature, just leave that part out.
- in main.cf:
- use restriction classes to make restricted_users file more readable.
smtpd_restriction_classes = local_only, local_plus
local_only =
reject_unauth_destination
permit_mynetworks
reject
local_plus =
check_recipient_access hash:/etc/postfix/local_plus
check_sender_access hash:/etc/postfix/local_plus
reject_unauth_destination
permit_mynetworks
reject
- this is the default setting, required for this setup.
smtpd_delay_reject = yes
- we'll do this in sender restrictions to avoid open relay problems.
smtpd_sender_restrictions =
check_sender_access hash:/etc/postfix/restricted_users
check_recipient_access hash:/etc/postfix/restricted_users
and in /etc/postfix/restricted_users
# /etc/postfix/restricted_users
# this file contains a list of users only allowed to send and receive local
mail
# postmap this file after changes
# local users not listed here have no restrictions
user1@miodemi.com local_only
user2@miodemi.com local_plus
and in /etc/postfix/local_plus:
# /etc/postfix/local_plus
# this file contains allowed destinations and senders
# for users restricted to local_plus
# postmap this file after changes
miproveedor.com OK
Remember to "postmap local_plus" and "postmap restricted_users" after
making changes to them.
Remember to run "postfix reload" after changing main.cf
我自己常用的限制方法
有几个方法：
1) smtpd_restriction_classes = local_only
设置一个限制类别叫local_only，然后参考access(5)的格式做一个访问控制：
local_only = check_recipient_access hash:/etc/postfix/maps/my_rcpt
文件my_rcpt内容：
163.com RELAY 21cn.com RELAY hzqbbc.com RELAY
然后，设置：
smtpd_sender_restrictions = check_sender_access hash:/etc/postfix/maps/mysender
mysender内容：
hzqbbc@local.hzqbbc.com local_only
这样凡是Mail from:
按这个方法，还可以设置更多的类别，例如remote_only以及不限制的帐号等。但这些都只对from:限制。而且不管是否SASL后的。所以有一定缺陷。不过，已经达到目的了。
2)使用snapshot版的policy策略
根据某个hash表或配置文件，判断对应的sender和recipient是否匹配，匹配就返回OK或者DUNNO或者RELAY等（可能RELAY已经过时，这个是postfix 1.1.x的）如果不匹配就返回错误代码
按postfix所带的smtpd-policy.pl模式，修改一下就可以使用了。详细参考POLICY_README等。
3)使用APF for postfix
详细地址：http://apf.org.cn/addon/
其实和2）方法是基本一致的。同样利用postfix内置的policy功能。
Last Update: 2005-06-06
Posted by hzqbbc at 07:34 PM | Comments (0)
December 02, 2002
维护Postfix的一些脚本、程序集锦
作为一名Postfix的管理员，维护Postfix的正常运行，并随时排除故障，这是最基本的任务。本文提供了一些我自己写及平时搜集到的脚本和程序，以便实现自动化的维护和操作。包括自动清理日志，分析日志，自动增加用户等等。
分析拒收邮件的日志分析程序
作者：Wietse Venema
Message 6 in thread
寄件者：Wietse Venema (wietse@porcupine.org)
主旨：Re: Freebsd Postifx Daily messages
View this article only
新闻群组：mailing.postfix.users
日期：2002-12-01 10:48:47 PST
Here's my own /etc/periodic/daily/460.status-mail-rejects script,
and yes it needs updating when logging formats change.
This script will not show rejected mail when the client never
returned after a soft (450) error code from the server. I use 450
for non-FQDN HELO hostnames, so that I can whitelist mis-configured
sites that aren't spammers.
Wietse
#!/bin/sh
#
echo ""; echo "Scanning maillog for rejections:"
zcat -fc /var/log/maillog.0* /var/log/maillog | egrep reject: | \
cut -d : -f 5- | sort | uniq -c | sort -nr | \
grep -v '^ *[1-2] *[^ ]* [^ ]* from [^ ]*: 450 '
echo "";
echo "Scanning maillog for warnings:"
zcat -fc /var/log/maillog.0* /var/log/maillog | egrep warning: | \
cut -d : -f 5- | sort | uniq -c
print "";
echo "Scanning maillog for trouble:"
zcat -fc /var/log/maillog.0* /var/log/maillog | egrep '(fatal|panic):'
exit
Post a follow-up to this message
一个简单的分析log的脚本
Author: Michael Tokarev
#!/bin/sh
# Parses postfix style logs
if [ $# -ne 2 ]; then
echo "Usage: `basename $0` logfile search-criteria"
exit 1
fi
TMPLOG="/tmp/`basename $0`.work.$$"
MATCHES="/tmp/`basename $0`.matches.$$"
LOGFILE="$1"
if [ "${LOGFILE}" != "-" -a ! -f ${LOGFILE} ]; then
echo "File not found (${LOGFILE})"
exit 2
fi
if echo ${LOGFILE} │ egrep '\.bz2$' >/dev/null 2>&1; then
CAT="bunzip2 -c ${LOGFILE}"
elif echo ${LOGFILE} │ egrep '\.(Z│gz)$' >/dev/null 2>&1; then
CAT="gunzip -c ${LOGFILE}"
elif [ "${LOGFILE}" = "-" ]; then
cat > ${TMPLOG}.orig
CAT="cat ${TMPLOG}.orig"
else
CAT="cat ${LOGFILE}"
fi
shift
${CAT} │ egrep -i "$1" │ \
awk '$9 ~ /^[A-Z0-9][A-Z0-9]*:$/ {print $1 " *" $2 ".*" $9}' > ${MATCHES}
[ -s ${MATCHES} ] && ${CAT} │ egrep -i -f ${MATCHES}
[ -f ${TMPLOG}.orig ] && rm ${TMPLOG}.orig
[ -f ${TMPLOG} ] && rm ${TMPLOG}
[ -f ${MATCHES} ] && rm ${MATCHES}
exit 0
我写的一个简单邮件IN/OUT分析脚本
Author: hzqbbc
是参考了Wietse的reject 分析脚本后，得到启发写的，希望对用Postfix的朋友有点帮助。
#!/bin/bash
# mail-statistic.sh :
# Simple script for analysis Postfix maillog
# Version: 0.03
# Author: hzqbbc@hzqbbc.com
#
RAN=`echo $RANDOM`
echo "Mail log analysis ID is $RAN "
echo "Checking ...................."
echo ""
echo ""
# make a direcotry to contain temp log file
mkdir $RAN
cp /var/log/maillog $RAN/maillog
zcat /var/log/maillog.*.gz >> $RAN/maillog
# statistic recipient
echo "Part one : - Top 10 Recipient domain "
fgrep to= $RAN/maillog | cut -d\< -f2 | cut -d\> -f1 | \
cut -d@ -f2 | sort | uniq -c | sort -nr |head
echo ""
echo ""
# statistic sender
echo "Part two: - Top 10 Sender domain "
fgrep from= $RAN/maillog | cut -d\< -f2 | cut -d\> -f1 | \
cut -d@ -f2 | sort | uniq -c | sort -nr | head
echo ""
echo ""
# statistic destination mail total count
echo "Part three: Total lines and sizes of log, And total mail sent"
echo "Log_lines = `cat $RAN/maillog | wc -l | awk '{ print $1 }'`"
echo "Log_sizes = `du -s $RAN/maillog | awk '{ print $1}'`"
echo "Mail_sent = `cat $RAN/maillog | fgrep status=sent | wc -l | \
awk '{ print $1 }'`"
rm -rf $RAN
Posted by hzqbbc at 10:07 AM | Comments (0)
November 28, 2002
关于Postfix虚拟域的相关释疑
在Postfix 1.1.x中，virtual_maps及virtual_mailbox_maps容易搞混，而在未来的Postfix 2.x中，virtual_*相关的设置也会有所变动。因此有必要搞清楚两者之间的区别和联系，以及虚拟域在Postfix中的实现和相关含义。
寄件者："James T. Richardson, Jr." (james.richardson@wickander.com)
主旨：Postifix & Virtual Domains
View this article only
新闻群组：mailing.postfix.users
日期：2002-11-26 15:12:06 PST
After setting up Postfix + MySQL I'm having a hard time getting "virtual
aliases" working.
I'm wanting to use Virtual Mailbox Domains (not Postfix-Style, nor
Sendmail-Style).
Mail is delivered to both my local domain and to my virtual domains. At
least for my users that have Maildir's defined.
Relevant sectionof the config:
---
myhostname = mail-gateway.mydomain.com
myorigin = $mydomain
mydestination = $myhostname, localhost.$mydomain $mydomain
local_recipient_maps = $alias_maps unix:passwd.byname
alias_maps = hash:/etc/postfix/aliases
alias_database = $alias_maps
transport_maps = mysql:/etc/postfix/transport.cf
virtual_mailbox_maps = mysql:/etc/postfix/mysql_virt.cf
hash:/etc/postfix/virtual
virtual_uid_maps = mysql:/etc/postfix/uids.cf
virtual_gid_maps = mysql:/etc/postfix/gids.cf
virtual_mailbox_base = /var/mail/vhosts
---
The "mysql_virt.cf" file looks like:
user=postfix
password=sekret
dbname=maildb
table=users
select_field_maildir
where_field=address
hosts=localhost
Following instructions by Victor Duchovni
(http://groups.google.com/groups?hl=en&lr=&ie=UTF-8&oe=UTF-8&selm=ajh5mv
$h93$1@FreeBSD.csie.NCTU.edu.tw&rnum=1):
> I call such domains *virtual-mailbox* domains. To define a virtual-mailbox
> domain, just a "domain whatever" entry to one of the tables listed in
> $virtual_mailbox_maps. For now it is also necessary to add the domain to
> the transport table in order to arrange for mail to the domain to be
> handled by a suitable virtual delivery agent (presumably but not
> necessarily the Postfix "virtual" delivery agent).
I created a simple hash (for now) files of "domain whatever" (and ran
postmap on it) and stuck it on the end of virtual_mailbox_maps.
Now, I'm not sure where to create these aliases or where to tell Postfix
about them. I tried putting an alias to postmaster@domain underneath the
"domain whatever" line in the virtual file, but that didn't work. I got
a
To=
(recipient postmaster@domain: uid not found in virtual_uid_maps)
Surely my aliases don't need assciated UID/GID's? do they?
---
James T. Richardson, Jr.
james.richardson@wickander.com
Post a follow-up to this message
Message 2 in thread
寄件者：Russell Mosemann (mose@ns.cune.edu)
主旨：Re: Postifix & Virtual Domains
View this article only
新闻群组：mailing.postfix.users
日期：2002-11-26 15:33:51 PST
On Tue, 26 Nov 2002, James T. Richardson, Jr. wrote:
> After setting up Postfix + MySQL I'm having a hard time getting "virtual
> aliases" working.
>
> I'm wanting to use Virtual Mailbox Domains (not Postfix-Style, nor
> Sendmail-Style).
>
> Mail is delivered to both my local domain and to my virtual domains. At
> least for my users that have Maildir's defined.
>
>
> Relevant sectionof the config:
> ---
> myhostname = mail-gateway.mydomain.com
> myorigin = $mydomain
> mydestination = $myhostname, localhost.$mydomain $mydomain
> local_recipient_maps = $alias_maps unix:passwd.byname
> alias_maps = hash:/etc/postfix/aliases
> alias_database = $alias_maps
> transport_maps = mysql:/etc/postfix/transport.cf
> virtual_mailbox_maps = mysql:/etc/postfix/mysql_virt.cf
> hash:/etc/postfix/virtual
> virtual_uid_maps = mysql:/etc/postfix/uids.cf
> virtual_gid_maps = mysql:/etc/postfix/gids.cf
> virtual_mailbox_base = /var/mail/vhosts
> ---
>
> The "mysql_virt.cf" file looks like:
> user=postfix
> password=sekret
> dbname=maildb
> table=users
> select_field_maildir
> where_field=address
> hosts=localhost
>
> Following instructions by Victor Duchovni
> (http://groups.google.com/groups?hl=en&lr=&ie=UTF-8&oe=UTF-8&selm=ajh5mv
> $h93$1@FreeBSD.csie.NCTU.edu.tw&rnum=1):
>
> > I call such domains *virtual-mailbox* domains. To define a virtual-mailbox
> > domain, just a "domain whatever" entry to one of the tables listed in
> > $virtual_mailbox_maps. For now it is also necessary to add the domain to
> > the transport table in order to arrange for mail to the domain to be
> > handled by a suitable virtual delivery agent (presumably but not
> > necessarily the Postfix "virtual" delivery agent).
>
> I created a simple hash (for now) files of "domain whatever" (and ran
> postmap on it) and stuck it on the end of virtual_mailbox_maps.
>
> Now, I'm not sure where to create these aliases or where to tell Postfix
> about them. I tried putting an alias to postmaster@domain underneath the
> "domain whatever" line in the virtual file, but that didn't work. I got
> a
> To=
> (recipient postmaster@domain: uid not found in virtual_uid_maps)
>
> Surely my aliases don't need assciated UID/GID's? do they?
I don't think you read Victor's message very carefully. A virtual domain
is different from a virtual mailbox domain. virtual_maps are used not
only for redirecting email from one address to one or more other
addresses, it is also used for virtual domains. virtual_mailbox_maps are
used for virtual mailbox domains and for mailboxes on the local computer
that do not have an associated Unix account. You are mixing the two, and
that is the cause of your problems.
----
Russell Mosemann, Ph.D. * Computing Services * Concordia University, Nebraska
"A penny saved is a penny earned." "Penny wise, pound foolish."
Post a follow-up to this message
Message 3 in thread
寄件者：Victor.Duchovni@morganstanley.com (Victor.Duchovni@morganstanley.com)
主旨：Re: Postifix & Virtual Domains
View this article only
新闻群组：mailing.postfix.users
日期：2002-11-26 19:12:51 PST
On Tue, 26 Nov 2002, James T. Richardson, Jr. wrote:
> Following instructions by Victor Duchovni
> (http://groups.google.com/groups?hl=en&lr=&ie=UTF-8&oe=UTF-8&selm=ajh5mv
> $h93$1@FreeBSD.csie.NCTU.edu.tw&rnum=1):
>
> > I call such domains *virtual-mailbox* domains. To define a virtual-mailbox
> > domain, just a "domain whatever" entry to one of the tables listed in
> > $virtual_mailbox_maps. For now it is also necessary to add the domain to
> > the transport table in order to arrange for mail to the domain to be
> > handled by a suitable virtual delivery agent (presumably but not
> > necessarily the Postfix "virtual" delivery agent).
>
> I created a simple hash (for now) files of "domain whatever" (and ran
> postmap on it) and stuck it on the end of virtual_mailbox_maps.
>
> Now, I'm not sure where to create these aliases or where to tell Postfix
> about them. I tried putting an alias to postmaster@domain underneath the
> "domain whatever" line in the virtual file, but that didn't work. I got
> a
> To=
> (recipient postmaster@domain: uid not found in virtual_uid_maps)
>
You actually searched for and found some documentation and mostly
followed it correctly! You deserve a break.
Your issue is that you are trying to specify address rewriting rules
("aliases" of sorts) for virtual mailbox domains in the
virtual_mailbox_maps tables. These tables define the pathname of the
virtual user's mailbox, not an alternate address.
For virtual mailbox users the only available address rewriting mechanism
is $virtual_maps. Don't put the "domain anything" keys for virtual
mailbox domainsthere, but do put address rewriting rules there.
So
virtual_mailbox_maps = hash:/etc/postfix/vmboxdomains, mysql:mumble
virtual_maps = mysql:fumble
/etc/postfix/vmboxdomains will have all the "domain anything" keys. List
it first, as these lookups will be very fast in comparison to mysql.
mysql:mumble will map virtual users to their mailbox path.
mysql:fumble (or hash:fumble) will map "aliased" virtual users to the
correct email address(es).
--
Viktor.
Posted by hzqbbc at 01:13 AM | Comments (0)
November 13, 2002
有关于mjt所写的tinycdb的技术内幕讨论
TinyCDB是一个对djb提出的CDB全新的实现，体积非常小，速度非常非常快。有俄罗斯人mjt开发。但它有一个致命的缺点，那就是每次更新/删除/添加都必须重建数据库（也因为不需要维护太多东西，所以才快）。
原文：
About TinyCDB
Message 1 in thread
寄件者：Michael Tokarev (mjt@tls.msk.ru)
主旨：[ot] tinyCDB is available in packaged form
View this article only
新闻群组：mailing.postfix.users
日期：2002-10-12 16:28:04 PST
After recent comparisions and mention of CDB map type, I received
several emails from several people about my tinycdb package. I've
made cosmetic changes in packaging scripts and uploaded binaries for
RedHat-like (rpm) and Debian Linux systems. It is available at
ftp://ftp.corpit.ru/pub/tinycdb/ :
21985 tinycdb-0.72.tar.gz source tarball incl. build scripts
26572 tinycdb-0.72-1.i386.rpm RedHat i386 rpm (rpm -tb)
24064 tinycdb_0.72_i386.deb Debian i386 .deb (native build)
This includes both the -lcdb library and simple cdb command-line
utility to work with .cdb files.
CDB map for postfix is available at ftp://ftp.corpit.ru/pub/postfix/,
see README.CDB in this directory.
/mjt
Message 2 in thread
寄件者：Wietse Venema (wietse@porcupine.org)
主旨：Re: [ot] tinyCDB is available in packaged form
View this article only
新闻群组：mailing.postfix.users
日期：2002-10-12 17:22:01 PST
Why does mkmap_cdb_open() lock the .cdb file?
Wietse
Michael Tokarev:
> After recent comparisions and mention of CDB map type, I received
> several emails from several people about my tinycdb package. I've
> made cosmetic changes in packaging scripts and uploaded binaries for
> RedHat-like (rpm) and Debian Linux systems. It is available at
> ftp://ftp.corpit.ru/pub/tinycdb/ :
>
> 21985 tinycdb-0.72.tar.gz source tarball incl. build scripts
> 26572 tinycdb-0.72-1.i386.rpm RedHat i386 rpm (rpm -tb)
> 24064 tinycdb_0.72_i386.deb Debian i386 .deb (native build)
>
> This includes both the -lcdb library and simple cdb command-line
> utility to work with .cdb files.
>
> CDB map for postfix is available at ftp://ftp.corpit.ru/pub/postfix/,
> see README.CDB in this directory.
>
> /mjt
>
> -
> To unsubscribe, send mail to majordomo@postfix.org with content
> (not subject): unsubscribe postfix-users
>
>
-
To unsubscribe, send mail to majordomo@postfix.org with content
(not subject): unsubscribe postfix-users
Message 3 in thread
寄件者：Michael Tokarev (mjt@tls.msk.ru)
主旨：Re: [ot] tinyCDB is available in packaged form
View this article only
新闻群组：mailing.postfix.users
日期：2002-10-12 18:32:02 PST
Wietse Venema wrote:
> Why does mkmap_cdb_open() lock the .cdb file?
Hmm, this is an interesting question - now, after more than a
year since that discussions, I actually see what did you mean
when asked similar question.
There are two things to protect by a lock: creating a map file
updating of a map - this last one in case of cdb consists of
two operations, namely creating and renaming. I.e., new (temp)
file may be protected during it's creation/writing so no two
processes will write to it at once, or the whole operation
may be protected, so no two proceses will (create+rename) at
once.
Or, in the other words, lock on .cdb file should mean "wait,
I'm working on this map", while a lock on .tmp file should
mean "wait, I'm working with this file".
There is a minor and somewhat unclean difference, but it
exists.
I don't know which is better - in fact, there is no real
difference in a result, it seems. Why? Because if the
only program who'll update a map will be postfix, there is
really no difference once all postfix's components are
consistent. But if there will be other programs, things
will be more "interesting": there is no "standard" choice
for a name of a temp file, one program may use .cdb.tmp
(as my mkmap_cdb), another may use .tmp, yet another may
use random-unique-name etc. No other tool I know of uses
any locking at all when creating cdb (there are only 2:
DJB's cdbmake and tinycdb's cdb -c).
Well, if some other program will actually do soome locking,
it's a good idea to choose the same locking scheme, and the
right one (from the CDB's ideology) seems to be to lock .tmp
file ("I'm working on this file").
But implementation is more difficult in this case (when locking
.tmp) - it's more tricky to avoid a race here and not to start
updating .cdb file instead of .tmp -- if memory serves me right;
it was long ago when I thought about this. This was the only
reason why I choosed to lock .cdb instead of .tmp when first
wrote the map. Anyway, it shouldn't be impossible ;)
BTW, the whole cdb thing, just like maildir format, seems to
be designed to avoid any and all locking altogether. For this,
simplest solution is to choose a random (unique) name of temp
file and forget about concurrency issues. This solution has
one implication (already noted by you) - it's possible to have
leftover files after crashes in this case.
/mjt
-
To unsubscribe, send mail to majordomo@postfix.org with content
(not subject): unsubscribe postfix-users
Message 4 in thread
寄件者：Matthias Andree (ma@dt.e-technik.uni-dortmund.de)
主旨：Re: [ot] tinyCDB is available in packaged form
View this article only
新闻群组：mailing.postfix.users
日期：2002-10-14 09:00:06 PST
Michael Tokarev
> There are two things to protect by a lock: creating a map file
> updating of a map - this last one in case of cdb consists of
> two operations, namely creating and renaming. I.e., new (temp)
> file may be protected during it's creation/writing so no two
> processes will write to it at once, or the whole operation
> may be protected, so no two proceses will (create+rename) at
> once. ...
> BTW, the whole cdb thing, just like maildir format, seems to
> be designed to avoid any and all locking altogether.
Not really, but that's why you pass DJB's applications the name of a
temp file. Whether you let the user choose one or use mkstemp, is a
matter of convenience and portability.
> For this, simplest solution is to choose a random (unique) name of
> temp file and forget about concurrency issues. This solution has one
> implication (already noted by you) - it's possible to have leftover
> files after crashes in this case.
Do you think it would be possible to use mkstemp or something for the
new map file and rename(2) that into place after it is complete, and
dropping all locking code?
I'd not really worry about "two processes rename to the same .cdb file"
at all: let the last one to rename win. Usually, these .cdb files are
under administrator control and will actually be built from some input
by a Makefile and make or something.
You could just document the template you passed to mkstemp() and tell
users they can safely delete these files. Should this happen while your
makecdb is running at the same time, rename() will fail without touching
the destination file.
No need to be paranoid here.
--
Matthias Andree
-
To unsubscribe, send mail to majordomo@postfix.org with content
(not subject): unsubscribe postfix-users
Post a follow-up to this message
Message 5 in thread
寄件者：Victor.Duchovni@morganstanley.com (Victor.Duchovni@morganstanley.com)
主旨：Re: [ot] tinyCDB is available in packaged form
View this article only
新闻群组：mailing.postfix.users
日期：2002-10-14 10:12:01 PST
On Mon, 14 Oct 2002, Matthias Andree wrote:
> I'd not really worry about "two processes rename to the same .cdb file"
> at all: let the last one to rename win. Usually, these .cdb files are
> under administrator control and will actually be built from some input
> by a Makefile and make or something.
>
I don't think this is right. When I run two concurrent copies of "postmap
-i" I expect both sets of key/value pairs to be added to the map.
Does the cdb "mkmap" code support incremental inserts or only full map
rebuilds?
--
Viktor.
-
To unsubscribe, send mail to majordomo@postfix.org with content
(not subject): unsubscribe postfix-users
Post a follow-up to this message
Message 6 in thread
寄件者：Michael Tokarev (mjt@tls.msk.ru)
主旨：Re: [ot] tinyCDB is available in packaged form
View this article only
新闻群组：mailing.postfix.users
日期：2002-10-14 12:28:07 PST
Victor.Duchovni@morganstanley.com wrote:
[]
> I don't think this is right. When I run two concurrent copies of "postmap
> -i" I expect both sets of key/value pairs to be added to the map.
>
> Does the cdb "mkmap" code support incremental inserts or only full map
> rebuilds?
No - this is a principal limitation of a cdb, it's a Constant database.
I.e. any and all manipulation with content requires complete rebuild.
Well, not strictly this - updates are possible by rebuilding the whole
hash table (inplace) or by recreating a file using old file as a source,
but that's not worth the effort to program the whole thing IMHO. For
this very reason, cdb isn't sutable for pop-before-smtp and the like
dynamic maps.
Mkmap code in my cdb patch does not support incremental updates, again,
for the same reason (it will be complete rebuild anyway).
/mjt
-
To unsubscribe, send mail to majordomo@postfix.org with content
(not subject): unsubscribe postfix-users
Post a follow-up to this message
Message 7 in thread
寄件者：Bennett Todd (bet@rahul.net)
主旨：Re: [ot] tinyCDB is available in packaged form
View this article only
新闻群组：mailing.postfix.users
日期：2002-10-15 12:24:05 PST
--n8g4imXOkfNTN/H1
Content-Type: text/plain; charset=us-ascii
Content-Disposition: inline
2002-10-14-15:24:50 Michael Tokarev:
> No - this is a principal limitation of a cdb, it's a Constant
> database. I.e. any and all manipulation with content requires
> complete rebuild. [...] For this very reason, cdb isn't sutable
> for pop-before-smtp and the like dynamic maps.
Not to dispute any of the other stuff you said, but I'm really not
sure about this bit.
I'd have to do some experimenting, but I really suspect
pop-before-smtp would work very well indeed with cdb.
It might be worth taking the trouble to have a version of cdb -c
inlined into the daemon, perhaps make a dynamically loadable perl
module if I were doing this on my pop-before-smtp daemon. That'd
save the need for fork-n-exec for each db update; if you ain't on
Linux, fork-n-exec is overpriced:-).
But with an in-memory cdb create, the update would be
open-tmp-file/write/close/rename, and for reasonable size maps ---
many folks don't have that many concurrent poppers from outside
mynetworks --- I'm sure this would be cheaper than
open-db/lock/read/write/flush/unlock/close. In fact, I wouldn't be
surprised if the break-even point for an optimized cdb-writer
weren't actually pretty darned large. cdb is awfully quick.
Hmm. Now I'm gonna have to find time to play with CDB_File.
-Bennett
--n8g4imXOkfNTN/H1
Content-Type: application/pgp-signature
Content-Disposition: inline
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.0.7 (GNU/Linux)
iD8DBQE9rF82HZWg9mCTffwRAuGhAJ9×4iqOrAmlmsXdy0QpPHFFFFaDCgCgjumd
Dpbm/WiCJre7+sXb9TCRJ/g=
=AMc7
-----END PGP SIGNATURE-----
n8g4imXOkfNTN/H1
-
To unsubscribe, send mail to majordomo@postfix.org with content
(not subject): unsubscribe postfix-users
Post a follow-up to this message
Message 8 in thread
寄件者：Michael Tokarev (mjt@tls.msk.ru)
主旨：Re: [ot] tinyCDB is available in packaged form
View this article only
新闻群组：mailing.postfix.users
日期：2002-10-15 12:46:02 PST
Bennett Todd wrote:
> 2002-10-14-15:24:50 Michael Tokarev:
>
>>No - this is a principal limitation of a cdb, it's a Constant
>>database. I.e. any and all manipulation with content requires
>>complete rebuild. [...] For this very reason, cdb isn't sutable
>>for pop-before-smtp and the like dynamic maps.
>
>
> Not to dispute any of the other stuff you said, but I'm really not
> sure about this bit.
>
> I'd have to do some experimenting, but I really suspect
> pop-before-smtp would work very well indeed with cdb.
>
> It might be worth taking the trouble to have a version of cdb -c
> inlined into the daemon, perhaps make a dynamically loadable perl
> module if I were doing this on my pop-before-smtp daemon. That'd
> save the need for fork-n-exec for each db update; if you ain't on
> Linux, fork-n-exec is overpriced:-).
>
> But with an in-memory cdb create, the update would be
> open-tmp-file/write/close/rename, and for reasonable size maps ---
> many folks don't have that many concurrent poppers from outside
> mynetworks --- I'm sure this would be cheaper than
> open-db/lock/read/write/flush/unlock/close. In fact, I wouldn't be
> surprised if the break-even point for an optimized cdb-writer
> weren't actually pretty darned large. cdb is awfully quick.
One word: scalability. With a dynamically updateable map, a
time needed to insert one record does not depend on the number
of records already present in a map (well, almost). Cdb
requires complete rewrite. It will work for small sites.
Also, create/rename requires filesystem metadata updates,
while read/write does not - thus, depending on a filesystem,
situation with cdb will be worse. But it has one advantage:
it will work well over NFS, so it will be possible to use
PBS when pop daemon isn't on the same machine as smtp.
In fact, the best way to implement PBS is to keep all records
in memory (8 bytes for each (IP and time) isn't that much)
and use tcp map in postfix (and this too will work good
on a large site). Crashes/reboots aren't that important
here, it's almost ok to lose data since it isn't keept
for a long time anyway (just "rePOP" will be required).
With such a method, far more efficient search algorithm
may be implemented (i.e. sorted array of IP addresses,
maybe hashed).
> Hmm. Now I'm gonna have to find time to play with CDB_File.
I do NOT recommend this, at least w/o fixing things first.
Cdbmake code has at least one (huge) memory leak and thus
isn't sutable for a long-living daemon. Also, cdbmake code
requires much more memory than tinycdb, and is less
efficient (this all is true for cdb-0.75, I not looked
if new version was released).
What I want to is to find a time to write perl and especially
nss module for cdb - this is on a todo list for more than a
year now... ;)
/mjt
-
To unsubscribe, send mail to majordomo@postfix.org with content
(not subject): unsubscribe postfix-users
寄件者：Bennett Todd (bet@rahul.net)
主旨：Re: [ot] tinyCDB is available in packaged form
View this article only
新闻群组：mailing.postfix.users
日期：2002-10-16 09:40:03 PST
--kXdP64Ggrk/fb43R
Content-Type: text/plain; charset=us-ascii
Content-Disposition: inline
Content-Transfer-Encoding: quoted-printable
2002-10-15-15:43:50 Michael Tokarev:
> Bennett Todd wrote:
> >I'd have to do some experimenting, but I really suspect
> >pop-before-smtp would work very well indeed with cdb.
>=20
> One word: scalability. With a dynamically updateable map, a
> time needed to insert one record does not depend on the number
> of records already present in a map (well, almost). Cdb
> requires complete rewrite. It will work for small sites.
Define "small".
I seriously believe that a large majority of pop-before-smtp users
rarely have more than a few records in their database; and that
sites with as many as a hundred such records much of the time are
liable to be a teensy minority, if not a non-existent limit case.
Where's the crossover point? I.e. where does the cost of the CDB
generation grow to exceed the cost of doing the DB manipulation?
This undoubtedly varies from platform to platform. I'll try and
produce an answer for at least one platform:-).
> Also, create/rename requires filesystem metadata updates,
> while read/write does not - thus, depending on a filesystem,
> situation with cdb will be worse.
I'll try it with ext3, and with tmpfs. But while read/write may not
require metadata updates, it does require locking. I don't know
what sort of difference that will make, if any.
> In fact, the best way to implement PBS is to keep all records
> in memory (8 bytes for each (IP and time) isn't that much)
> and use tcp map in postfix (and this too will work good
> on a large site).
Sounds like we're converging on DRAC:-).
> >Hmm. Now I'm gonna have to find time to play with CDB_File.
>=20
> I do NOT recommend this, at least w/o fixing things first.
> Cdbmake code has at least one (huge) memory leak and thus
> isn't sutable for a long-living daemon.
观看文件全部内容 （仍有 22 行）
Post a follow-up to this message
Message 10 in thread
寄件者：Matthias Andree (ma@dt.e-technik.uni-dortmund.de)
主旨：Re: [ot] tinyCDB is available in packaged form
View this article only
新闻群组：mailing.postfix.users
日期：2002-10-21 06:25:57 PST
Bennett Todd
> I'd have to do some experimenting, but I really suspect
> pop-before-smtp would work very well indeed with cdb.
pop-before-smtp is inherently insecure, no matter HOW you implement
it. The problem is that you give relay permission to a host that may
well be "replaced" by some other (1st hosts disconnects, 2nd is assigned
the previous IP from 1st host from the dialup pool), without your SMTP
server noticing the disconnect. SMTP AUTH does not suffer from these
problems and does not need all these expire cronjobs and tossing data
from POP server to SMTP server and the like, worrying about how you
update a cdb... A cdb does not cut incremental updates. It would cope
well with a passwd like file though.
--
Matthias Andree
-
To unsubscribe, send mail to majordomo@postfix.org with content
(not subject): unsubscribe postfix-users
寄件者：Bennett Todd (bet@rahul.net)
主旨：Re: [ot] tinyCDB is available in packaged form
View this article only
新闻群组：mailing.postfix.users
日期：2002-10-21 07:44:59 PST
--gTtJ75FAzB1T2CN6
Content-Type: text/plain; charset=us-ascii
Content-Disposition: inline
2002-10-21-08:59:40 Matthias Andree:
> pop-before-smtp is inherently insecure, no matter HOW you implement
> it.
Lightly so, yes.
> The problem is that you give relay permission to a host that may
> well be "replaced" by some other (1st hosts disconnects, 2nd is assigned
> the previous IP from 1st host from the dialup pool), without your SMTP
> server noticing the disconnect.
That's why there's a timeout. A default of 1/2 hour seems to be
fine. Yes, there's a window during which someone else who got the IP
addr of a recently-pop-authed machine could relay. No, it doesn't
appear to be a problem in practice; I've never heard of such a setup
actually being exploited through this timing window.
> SMTP AUTH does not suffer from these problems [...]
If SMTP AUTH didn't require SASL, or if there were an implementation
of SASL besides Cyrus, I might be more interested. Is it possible to
implement enough SASL for SMTP AUTH without bringing in mountains of
excruciatingly painful goo, like e.g. GSSAPI with it's ASN.1 and so
forth?
> [...] and does not need all these expire cronjobs and tossing data
> from POP server to SMTP server and the like, [...]
I don't have any expire cronjobs, I just have a single persistent
daemon that manages the data file. Works great for a single server
that's both the pop/imap mailbox server and the smtp relay server.
For server farms, something more like DRAC or whoson would probably
be in order.
> worrying about how you update a cdb... A cdb does not cut
> incremental updates.
Sure it does, it does so beautifully. cdb file rebuild is so fast
that doing a full rebuild for each update is practical in many
settings --- as DNS hosting sites have discovered when using
tinydns.
-Bennett
--gTtJ75FAzB1T2CN6
Content-Type: application/pgp-signature
Content-Disposition: inline
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.0.7 (GNU/Linux)
iD8DBQE9tA7UHZWg9mCTffwRAhs7AKDEw1uALoCxjHAhlSJA0xic+2hfTACgm6a2
0BJdYgRUKo90nwQlSFdkK24=
=XNg1
-----END PGP SIGNATURE-----
gTtJ75FAzB1T2CN6
-
To unsubscribe, send mail to majordomo@postfix.org with content
(not subject): unsubscribe postfix-users
Post a follow-up to this message
Message 12 in thread
寄件者："Schmehl, Paul L" (pauls@utdallas.edu)
主旨：RE: [ot] tinyCDB is available in packaged form
View this article only
新闻群组：mailing.postfix.users
日期：2002-10-21 08:30:04 PST
> -----Original Message-----
> From: Bennett Todd [mailto:bet@rahul.net]
> Sent: Monday, October 21, 2002 9:28 AM
> To: Matthias Andree
> Cc: postfix-users@postfix.org
> Subject: Re: [ot] tinyCDB is available in packaged form
>
>
> 2002-10-21-08:59:40 Matthias Andree:
> > pop-before-smtp is inherently insecure, no matter HOW you implement
> > it.
>
> Lightly so, yes.
>
> > The problem is that you give relay permission to a host that may well
> > be "replaced" by some other (1st hosts disconnects, 2nd is assigned
> > the previous IP from 1st host from the dialup pool), without your SMTP
> > server noticing the disconnect.
>
> That's why there's a timeout. A default of 1/2 hour seems to
> be fine. Yes, there's a window during which someone else who
> got the IP addr of a recently-pop-authed machine could relay.
> No, it doesn't appear to be a problem in practice; I've never
> heard of such a setup actually being exploited through this
> timing window.
There's quite an assumption behind this claim of insecurity. The
assumption is:
1) User one connects using pop before SMTP (or DRAC), reads mail and
disconnects before the timeout.
2) User two knows he a) has user one's IP address and knows b) the
IP/ hostname of the mail server and knows c) the server is open for
relay and intends d) to send spam through the server.
This assumption requires some incredible leaps in logic. Is it
possible? Of course. Just as breaking a 4096 AES key is possible. Is
it likely? Not in this lifetime. Even if I were a spammer, and even if
I knew that a particular server used pop before SMTP or DRAC and even
if I knew someone who used that server was online and using the server
and even if I knew precisely when they disconnected, I would still
have to obtain the exact same IP address to exploit the server.
I have real problems thinking of a scenario where this is even remotely
possible.
Paul Schmehl (pauls@utdallas.edu)
TCS Department Coordinator
The University of Texas at Dallas
AVIEN Founding Member
http://www.utdallas.edu/~pauls/
-
To unsubscribe, send mail to majordomo@postfix.org with content
(not subject): unsubscribe postfix-users
Post a follow-up to this message
Message 13 in thread
寄件者：Simon White (simon@mtds.com)
主旨：Re: [ot] tinyCDB is available in packaged form
View this article only
新闻群组：mailing.postfix.users
日期：2002-10-21 08:32:05 PST
21-Oct-02 at 10:19, Schmehl, Paul L (pauls@utdallas.edu) wrote :
> 1) User one connects using pop before SMTP (or DRAC), reads mail and
> disconnects before the timeout.
> 2) User two knows he a) has user one's IP address and knows b) the
> IP/ hostname of the mail server and knows c) the server is open for
> relay and intends d) to send spam through the server.
>
> This assumption requires some incredible leaps in logic. Is it
> possible? Of course. Just as breaking a 4096 AES key is possible. Is
> it likely? Not in this lifetime. Even if I were a spammer, and even if
> I knew that a particular server used pop before SMTP or DRAC and even
> if I knew someone who used that server was online and using the server
> and even if I knew precisely when they disconnected, I would still
> have to obtain the exact same IP address to exploit the server.
>
> I have real problems thinking of a scenario where this is even remotely
> possible.
Unless you have ISDN users with a fixed IP address : but then you'd have
to sniff their dialup username and password into the bargain.
--
[Simon White. vim/mutt. simon@mtds.com. GIMPS:41.94% see www.mersenne.org]
/"\ ASCII Ribbon Campaign
\ / Respect for open standards
X No HTML/RTF in email
/ \ No M$ Word docs in email
-
To unsubscribe, send mail to majordomo@postfix.org with content
(not subject): unsubscribe postfix-users
Post a follow-up to this message
Message 14 in thread
寄件者：Michael Tokarev (mjt@tls.msk.ru)
主旨：Re: [ot] tinyCDB is available in packaged form
View this article only
新闻群组：mailing.postfix.users
日期：2002-10-21 09:44:04 PST
Matthias Andree wrote:
[]
> pop-before-smtp is inherently insecure, no matter HOW you implement
> it. The problem is that you give relay permission to a host that may
> well be "replaced" by some other (1st hosts disconnects, 2nd is assigned
> the previous IP from 1st host from the dialup pool), without your SMTP
> server noticing the disconnect.
Mattias, this is not an issue at all with current world. Any given
(dialup) host has several minutes (hours? maybe days?) to find out
which mailservers was used by a host that has this IP previously,
from about 2^32 other hosts on the 'net. This is not possible in
that timeframe, and there are way far simpler ways exists to find
other open relays. Granted, if such PBS records will be permanent,
at some time a spammer eventually will find a relay, but that's a)
not the case (records aren't permanent), and b) chances are very
low anyway. Having a timeout of about an hour makes zero chances
for spammers to reuse the "relay". Again, in some cases, PBS is
somewhat insecure - think of common well-known public mail
systems like e.g. mail.com (a dialup spammer may try to send out
spam via mail.com just after dialing in to see if some previous
user used mail.com from this IP address recently), but again,
such cases are rare, and public mail systems should use other
protection methods as well (i.e. limiting number of recipients
in a time period, have smaller timeout for PBS etc).
/mjt
-
To unsubscribe, send mail to majordomo@postfix.org with content
(not subject): unsubscribe postfix-users
Post a follow-up to this message
Message 15 in thread
寄件者："Devin L. Ganger" (devin@thecabal.org)
主旨：Re: [ot] tinyCDB is available in packaged form
View this article only
新闻群组：mailing.postfix.users
日期：2002-10-21 13:20:05 PST
On Mon, Oct 21, 2002 at 10:19:04AM -0500, Schmehl, Paul L wrote:
> There's quite an assumption behind this claim of insecurity. The
> assumption is:
>
> 1) User one connects using pop before SMTP (or DRAC), reads mail and
> disconnects before the timeout.
True.
> 2) User two knows he a) has user one's IP address and knows b) the
> IP/ hostname of the mail server and knows c) the server is open for
> relay and intends d) to send spam through the server.
False. All user two has to do is attempt to relay. I wouldn't be at
all surprised to find that as more and more ISPs start protecting
their servers from relay, we're going to see more and more examples
of this from whackamole accounts.
--
Devin L. Ganger
Co-Admin, The cabalSASL Project ( http://sasl.thecabal.org/ )
A man, a miss, a car -- a curve,
He kissed the miss and missed the curve -- Burma Shave (1948)
-
To unsubscribe, send mail to majordomo@postfix.org with content
(not subject): unsubscribe postfix-users
Post a follow-up to this message
Message 16 in thread
寄件者："Schmehl, Paul L" (pauls@utdallas.edu)
主旨：RE: [ot] tinyCDB is available in packaged form
View this article only
新闻群组：mailing.postfix.users
日期：2002-10-21 14:16:04 PST
Why do you say this? If I attempt to relay through a server that uses
pbfs, it will fail - unless I am using the correct IP. So I not only
have to have the mail server's address, but I also must "own" the IP
address that's authorized to relay - or do you know something no one
else does about pbfs?
You can try relaying all day long, but unless you have the IP, you have
nothing. If you have the IP, then you must also be trying to relay
through that mail server. No other server will do. You make it sound
as though it's trivial to do. I don't see that at all.
Perhaps you could elaborate further?
Paul Schmehl (pauls@utdallas.edu)
TCS Department Coordinator
The University of Texas at Dallas
AVIEN Founding Member
http://www.utdallas.edu/~pauls/
> -----Original Message-----
> From: Devin L. Ganger [mailto:devin@thecabal.org]
> Sent: Monday, October 21, 2002 3:11 PM
> To: postfix-users@postfix.org
> Subject: Re: [ot] tinyCDB is available in packaged form
>
>
> On Mon, Oct 21, 2002 at 10:19:04AM -0500, Schmehl, Paul L wrote:
>
> > There's quite an assumption behind this claim of insecurity. The
> > assumption is:
> >
> > 1) User one connects using pop before SMTP (or DRAC), reads mail and
> > disconnects before the timeout.
>
> True.
>
> > 2) User two knows he a) has user one's IP address and knows b) the
> > IP/ hostname of the mail server and knows c) the server is open for
> > relay and intends d) to send spam through the server.
>
> False. All user two has to do is attempt to relay. I
> wouldn't be at all surprised to find that as more and more
> ISPs start protecting their servers from relay, we're going
> to see more and more examples of this from whackamole accounts.
-
To unsubscribe, send mail to majordomo@postfix.org with content
(not subject): unsubscribe postfix-users
Post a follow-up to this message
Message 17 in thread
寄件者：Simon White (simon@mtds.com)
主旨：Re: [ot] tinyCDB is available in packaged form
View this article only
新闻群组：mailing.postfix.users
日期：2002-10-21 15:46:02 PST
21-Oct-02 at 13:10, Devin L. Ganger (devin@thecabal.org) wrote :
> On Mon, Oct 21, 2002 at 10:19:04AM -0500, Schmehl, Paul L wrote:
>
> > There's quite an assumption behind this claim of insecurity. The
> > assumption is:
> >
> > 1) User one connects using pop before SMTP (or DRAC), reads mail and
> > disconnects before the timeout.
>
> True.
>
> > 2) User two knows he a) has user one's IP address and knows b) the
> > IP/ hostname of the mail server and knows c) the server is open for
> > relay and intends d) to send spam through the server.
>
> False. All user two has to do is attempt to relay. I wouldn't be at
> all surprised to find that as more and more ISPs start protecting
> their servers from relay, we're going to see more and more examples
> of this from whackamole accounts.
>
Indeed, if a spammer already has an inkling of a dialup subnet range
with the same domain name, it is quite likely that he will already have
been able to send spam through an open proxy which is purely a poor SME
connection sharing thing with some Exchange administrator who wouldn't
even pass an MCSE... although Microsoft have made some progress in this
direction there's a cost to upgrade unlike Open Source which offers free
upgrades in return for a bit of clue and your time.
Then he can regularly attempt to find a connection with an IP in the
POP-before-SMTP database but this is of little use to him: he might more
easily forge something and get past a poor server rather than look for
30 minute windows in which to inject as much spam as possible. With a
solid mailserver setup as any self-respecting mail admin should have,
this is less /and more!/ of a problem.
Sadly, there are more people in this world interested in finding mugs,
and they do find them, so spam is here to stay. As much as you'd have
many avenues to explore before feeling obliged to hack enough just to
get a maximum of one half hour of time, an admin proud of his Postfix
server which could (and has done, on my site, once...) deliver something
like 15,000 messages in a short period of time, on a reasonably fat pipe
to the Internet, might think that he is indeed prime spam territory. Of
course this is all relative because here fat is 2mbps!
Anyway anecdotes aside we're into philosophical ground really. The truly
paranoid may never run POP-before-SMTP but a good admin can feel solid
with that kind of system installed. Heck, it's probably reasonably
secure.
--
[Simon White. vim/mutt. simon@mtds.com. GIMPS:42.88% see www.mersenne.org]
Recognizing disagreements in belief requires having enough agreements in
belief to translate or understand the words and deeds of my opponent.
-- Anthony O'Hear (combining, somewhat, several modern philosophers).
-
To unsubscribe, send mail to majordomo@postfix.org with content
(not subject): unsubscribe postfix-users
Post a follow-up to this message
Message 18 in thread
寄件者：Victor.Duchovni@morganstanley.com (Victor.Duchovni@morganstanley.com)
主旨：Re: [ot] tinyCDB is available in packaged form
View this article only
新闻群组：mailing.postfix.users
日期：2002-10-21 16:12:05 PST
On Mon, 21 Oct 2002, Simon White wrote:
> Indeed, if a spammer already has an inkling of a dialup subnet range
> with the same domain name, it is quite likely that he will already have
> been able to send spam through an open proxy which is purely a poor SME
> connection sharing thing with some Exchange administrator who wouldn't
> even pass an MCSE... although Microsoft have made some progress in this
> direction there's a cost to upgrade unlike Open Source which offers free
> upgrades in return for a bit of clue and your time.
>
Pop before SMTP is designed to allow a user using ISP A to relay with the
mailhub of ISP B. ISP A does not need POP before SMTP to relay for their
own users, that's what permit_mynetworks is for.
The probability claims are plausible, one needs to know which third-party
dialup pool the user used to relay mail via their ISP.
Even if one can entice the user to send the attacker a message and hang
up, one then still needs to dial-in into the same IP in the dialup-pool.
If has a user account for the dialup pool, presumably one automatically
gets to relay through the mail relays of the pool provider.
--
Viktor.
-
To unsubscribe, send mail to majordomo@postfix.org with content
(not subject): unsubscribe postfix-users
Post a follow-up to this message
Message 19 in thread
寄件者："Devin L. Ganger" (devin@thecabal.org)
主旨：Re: [ot] tinyCDB is available in packaged form
View this article only
新闻群组：mailing.postfix.users
日期：2002-10-21 18:18:03 PST
On Mon, Oct 21, 2002 at 10:42:32PM +0000, Simon White wrote:
> Anyway anecdotes aside we're into philosophical ground really. The truly
> paranoid may never run POP-before-SMTP but a good admin can feel solid
> with that kind of system installed. Heck, it's probably reasonably
> secure.
Reasonably, yes. However, security is not a magic formula. You have
to understand the strengths and weaknesses of each of the various
pieces you use so that you use them intelligently.
I didn't want statements that could have appeared to place pbfs as some
sort of magic sovereign specific to go without also pointing out that
it does, under certain conditions (likely to be found in smaller ISPs
with limited address space) have drawbacks you need to be aware of.
--
Devin L. Ganger
Co-Admin, The cabalSASL Project ( http://sasl.thecabal.org/ )
A man, a miss, a car -- a curve,
He kissed the miss and missed the curve -- Burma Shave (1948)
-
To unsubscribe, send mail to majordomo@postfix.org with content
(not subject): unsubscribe postfix-users
Post a follow-up to this message
Message 20 in thread
寄件者："Devin L. Ganger" (devin@thecabal.org)
主旨：Re: [ot] tinyCDB is available in packaged form
View this article only
新闻群组：mailing.postfix.users
日期：2002-10-21 18:18:03 PST
On Mon, Oct 21, 2002 at 07:05:21PM -0400,
Victor.Duchovni@morganstanley.com wrote:
> Pop before SMTP is designed to allow a user using ISP A to relay with the
> mailhub of ISP B. ISP A does not need POP before SMTP to relay for their
> own users, that's what permit_mynetworks is for.
Yes, but there are ISPs that are using it for precisely that purpose,
despite the design goals because they think it's somehow more secure.
If you are one of them, be aware of the consequences.
--
Devin L. Ganger
Co-Admin, The cabalSASL Project ( http://sasl.thecabal.org/ )
A man, a miss, a car -- a curve,
He kissed the miss and missed the curve -- Burma Shave (1948)
-
To unsubscribe, send mail to majordomo@postfix.org with content
(not subject): unsubscribe postfix-users
Post a follow-up to this message
寄件者：Victor.Duchovni@morganstanley.com (Victor.Duchovni@morganstanley.com)
主旨：Re: [ot] tinyCDB is available in packaged form
View this article only
新闻群组：mailing.postfix.users
日期：2002-10-21 19:59:40 PST
On Mon, 21 Oct 2002, Devin L. Ganger wrote:
> On Mon, Oct 21, 2002 at 07:05:21PM -0400,
> Victor.Duchovni@morganstanley.com wrote:
>
> > Pop before SMTP is designed to allow a user using ISP A to relay with the
> > mailhub of ISP B. ISP A does not need POP before SMTP to relay for their
> > own users, that's what permit_mynetworks is for.
>
> Yes, but there are ISPs that are using it for precisely that purpose,
> despite the design goals because they think it's somehow more secure.
>
> If you are one of them, be aware of the consequences.
>
Yes, but presumably they require passwords for establishing PPP sessions
with their own dialup pool. So if the user is dialing into their dialup
pool, POP before SMTP is at least as strong as (though not much stonger
than) simply letting the user relay, and if the user dials up into someone
else's dialup pool the previous analysis holds, the spammer does not know
whose relay to try.
Either way POP before SMTP is safe.
--
Viktor.
-
To unsubscribe, send mail to majordomo@postfix.org with content
(not subject): unsubscribe postfix-users
Post a follow-up to this message
Message 22 in thread
寄件者：lst_hoe (lst_hoe@kwsoft.de)
主旨：Re: [ot] tinyCDB is available in packaged form
View this article only
新闻群组：mailing.postfix.users
日期：2002-10-22 02:08:06 PST
At 22:45 21.10.2002 -0400, Victor.Duchovni@morganstanley.com wrote:
>On Mon, 21 Oct 2002, Devin L. Ganger wrote:
>
>> On Mon, Oct 21, 2002 at 07:05:21PM -0400,
>> Victor.Duchovni@morganstanley.com wrote:
>>
>> > Pop before SMTP is designed to allow a user using ISP A to relay with the
>> > mailhub of ISP B. ISP A does not need POP before SMTP to relay for their
>> > own users, that's what permit_mynetworks is for.
>>
>> Yes, but there are ISPs that are using it for precisely that purpose,
>> despite the design goals because they think it's somehow more secure.
>>
>> If you are one of them, be aware of the consequences.
>>
>
>Yes, but presumably they require passwords for establishing PPP sessions
>with their own dialup pool. So if the user is dialing into their dialup
>pool, POP before SMTP is at least as strong as (though not much stonger
>than) simply letting the user relay, and if the user dials up into someone
>else's dialup pool the previous analysis holds, the spammer does not know
>whose relay to try.
>
>Either way POP before SMTP is safe.
The only real hole i see is with (bigger) NATed Networks. If one user
fetches his mail by POP3 all others are able to relay. With a little bit of
glue they will know this and getting the provider by personal talk should
not be a problem.
But this is a clear fact of unsecure (Company-) Network, although this is
common with DSL LANs today.
Regards
--
Andreas H?dle
Kühn & Weyh Software GmbH
WWW.KWSOFT.DE
-
To unsubscribe, send mail to majordomo@postfix.org with content
(not subject): unsubscribe postfix-users
Post a follow-up to this message
Message 23 in thread
寄件者：Simon White (simon@mtds.com)
主旨：Re: [ot] tinyCDB is available in packaged form
View this article only
新闻群组：mailing.postfix.users
日期：2002-10-22 02:28:39 PST
21-Oct-02 at 19:05, Victor.Duchovni@morganstanley.com (Victor.Duchovni@morganstanley.com) wrote :
> On Mon, 21 Oct 2002, Simon White wrote:
>
> > Indeed, if a spammer already has an inkling of a dialup subnet range
> > with the same domain name, it is quite likely that he will already have
> > been able to send spam through an open proxy which is purely a poor SME
> > connection sharing thing with some Exchange administrator who wouldn't
> > even pass an MCSE... although Microsoft have made some progress in this
> > direction there's a cost to upgrade unlike Open Source which offers free
> > upgrades in return for a bit of clue and your time.
> >
I am hinting, here, at the probability that a large dialup pool IP
address range is probably more interesting to scan for open relays /
proxies than for a posisble POP-before-SMTP exploit.
> Pop before SMTP is designed to allow a user using ISP A to relay with the
> mailhub of ISP B. ISP A does not need POP before SMTP to relay for their
> own users, that's what permit_mynetworks is for.
I was thinking of IP spoofing attacks in the first place; if you have to
call the ISP to get access to the right IP, you're probably in
permit_mynetworks anyway. But what if you are NATted on a large private
IP dialup subnet - with one public address which is in the
POP-before-SMTP database?
> The probability claims are plausible, one needs to know which third-party
> dialup pool the user used to relay mail via their ISP.
Exactly. This is what is reasonably hard.
> Even if one can entice the user to send the attacker a message and hang
> up, one then still needs to dial-in into the same IP in the dialup-pool.
Unless the system allows you to spoof the connections, or if you are
behind a NAT IP that is also the NAT IP of many others.
> If has a user account for the dialup pool, presumably one automatically
> gets to relay through the mail relays of the pool provider.
Precisely. That's why mynetworks should never make it into the pbsmtp
database.
--
[Simon White. vim/mutt. simon@mtds.com. GIMPS:44.31% see www.mersenne.org]
The only reason I'm burning my candle at both ends, is because I haven't
figured out how to light the middle yet.
[Linux user #170823 http://counter.li.org. Home cooked signature rotator.]
-
To unsubscribe, send mail to majordomo@postfix.org with content
(not subject): unsubscribe postfix-users
Post a follow-up to this message
Message 24 in thread
寄件者：Ralf Hildebrandt (Ralf.Hildebrandt@charite.de)
主旨：Re: [ot] tinyCDB is available in packaged form
View this article only
新闻群组：mailing.postfix.users
日期：2002-10-23 07:40:19 PST
On Wed, Oct 23, 2002 at 03:58:05PM +0200, Matthias Andree wrote:
> 30 minutes is way too much. 3 might be an option. SMTP AUTH is
> implemented in most clients with GUI, so there is no compelling reason
> to use RELAY-after-POP3 instead of SMTP AUTH. Cyrus-SASL might be one.
I'd go or 10, because that's the default in Mozilla/Netscape
--
Ralf Hildebrandt Ralf.Hildebrandt@charite.de
Postfix Tips: http://www.arschkrebs.de/postfix/ Tel. +49 (0)30-450 570-155
Microsoft: A Proven Danger to National Security
http://www.infowarrior.org/articles/msdanger.pdf
-
To unsubscribe, send mail to majordomo@postfix.org with content
(not subject): unsubscribe postfix-users
Post a follow-up to this message
Message 25 in thread
寄件者：Matthias Andree (ma@dt.e-technik.uni-dortmund.de)
主旨：Re: [ot] tinyCDB is available in packaged form
View this article only
新闻群组：mailing.postfix.users
日期：2002-10-23 08:20:11 PST
Bennett Todd
> 2002-10-21-08:59:40 Matthias Andree:
>> pop-before-smtp is inherently insecure, no matter HOW you implement
>> it.
>
> Lightly so, yes.
Is there "lightly pregnant" as well?
> That's why there's a timeout. A default of 1/2 hour seems to be
> fine. Yes, there's a window during which someone else who got the IP
> addr of a recently-pop-authed machine could relay. No, it doesn't
> appear to be a problem in practice; I've never heard of such a setup
> actually being exploited through this timing window.
> If SMTP AUTH didn't require SASL, or if there were an implementation
> of SASL besides Cyrus, I might be more interested. Is it possible to
> implement enough SASL for SMTP AUTH without bringing in mountains of
> excruciatingly painful goo, like e.g. GSSAPI with it's ASN.1 and so
> forth?
I must amend my statement: if there is a reliable way to detect when the
client has gone offline (session termination notification,
expiry of relay permission via radius), then these points don't hold;
OTOH, if you have radius, you can omit the "POP3-before-" part.
30 minutes is way too much. 3 might be an option. SMTP AUTH is
implemented in most clients with GUI, so there is no compelling reason
to use RELAY-after-POP3 instead of SMTP AUTH. Cyrus-SASL might be one.
I pinged Brian Stafford who has a partial SASL library (LGPL) as part of
his libesmtp to see if he'd work together with Wietse to make a more
secure replacement for Cyrus-SASL, for Postfix authentication
purposes. I've not seen code yet though.
> I don't have any expire cronjobs, I just have a single persistent
> daemon that manages the data file. Works great for a single server
> that's both the pop/imap mailbox server and the smtp relay server.
OK.
> Sure it does, it does so beautifully. cdb file rebuild is so fast
> that doing a full rebuild for each update is practical in many
> settings --- as DNS hosting sites have discovered when using
> tinydns.
Well, it's ok for you, but if you have thousands (because you have a
long timeout) of clients, then it won't scale well and BerkeleyDB will
be faster.
寄件者：Bennett Todd (bet@rahul.net)
主旨：Re: [ot] tinyCDB is available in packaged form
View: Complete Thread (31 articles)
Original Format
新闻群组：mailing.postfix.users
日期：2002-10-23 13:38:03 PST
--DBIVS5p969aUjpLe
Content-Type: multipart/mixed; boundary="uAKRQypu60I7Lcqm"
Content-Disposition: inline
--uAKRQypu60I7Lcqm
Content-Type: text/plain; charset=us-ascii
Content-Disposition: inline
Content-Transfer-Encoding: quoted-printable
2002-10-16-12:18:58 Bennett Todd:
> 2002-10-15-15:43:50 Michael Tokarev:
> > Bennett Todd wrote:
> > >I'd have to do some experimenting, but I really suspect
> > >pop-before-smtp would work very well indeed with cdb.
> >=20
> > One word: scalability. [...] It will work for small sites.
>=20
> Define "small".
> [...]
> Where's the crossover point? I.e. where does the cost of the CDB
> generation grow to exceed the cost of doing the DB manipulation?
Ok, in the interests of fairness, I've gotta come back and confess
that my intuition was way, way off here, and Michael Tokarev's was
right on.
While cdb can work for small sites, where the load of doing things
inefficiently doesn't make any difference anyway --- just what he
suggested --- there's no "crossover point"; no matter how small
the database, DB_Hash is faster than CDB_File for "updates" in all
circumstances.
On ext3fs:
CDBDB
107341797
1006621894
10005811372
10000133772
and on tmpfs:
CDBDB
10103406240060
10032839198009
10003513121343
1000034892536
The first column of these tables is the number of entries in the
table, where an entry is a key that looks like an IP address in
ASCII, four dotted quads randomly generated in [0..255], while the
value is "ok".
The numbers under the columns labelled CDB and DB are the number
of add/remove pair transactions completed in 30 seconds; for CDB
there's one full rebuild for each such transaction, while for DB
each such add/remove pair is surrounded by flocking, with a sync
right before the LOCK_UN. This mimics the traffic patterns seen with
pop-before-smtp, with some simplifying assumptions:-).
I used the attached program to generate that output, killing it
when the bad news was painfully obvious; rather than hassling with
emptying /tmp to overmount, I just edited the script to use /mnt for
the tmpfile for the tmpfs run.
Oh, and I ran these under Red Hat 7.3, kernel 2.4.19, on a Vaio
Picturebook C1VPK, TM5600 Crusoe CPU, 128MB RAM.
-Bennett
--uAKRQypu60I7Lcqm
Content-Type: text/plain; charset=us-ascii
Content-Disposition: attachment; filename=cdbbench
#!/usr/bin/perl -w
use strict;
=head1 NAME
cdbbench --- benchmark CDB, specifically to attempt to determine its appropriateness for pop-before-smtp
=head1 SYNOPSIS
cdbbench
=head1 DESCRIPTION
cdbbench performs alternating runs, while stepping up the chosen
setsize. The setsize is intended to reflect the number of
currently-active users in the pop-before-smtp database; it's the
number of records in the database.
Each record is a random IP-address-like key (four dotted decimals,
each in [0-255]), with the value "ok".
For each setsize, starting with 10 and growing by factors of 10 (10,
100, 1000, 10000, etc). For each setsize, cdbbench generates a load
of 10 times that many random keys. These are then cycled, first
through a fresh new cdb, then through a fresh new db, as fast as
possible, for 30 seconds; at the end of 30 seconds the total number
cycled through is reported. The DB clock doesn't start until the first
setsize items have been loaded (to prime the pump); the equivalent
step means nothing with CDB since the whole CDB file must be
rewritten each pass.
Hit control-C when you get tired of waiting.
=cut
use File::Basename;
my $tmpfile = "/tmp/" . basename($0) . '.' . $$;
$│ = 1;
print "\tCDB\tDB\n";
my $setsize = 1;
while (1) {
$setsize *= 10;
print $setsize, "\t";
my @load; # in-memory cache
for (my $i = 0; $i < 10*$setsize; $i++) {
push @load, join(".", int(rand(256)), int(rand(256)), int(rand(256)), int(rand(256)));
}
# CDB
use CDB_File;
my $count = 0;
my $end = time + 30;
while (time < $end) {
my $cdb = CDB_File->new($tmpfile, "$tmpfile.tmp") or die;
for (my $i = 0; $i < $setsize; $i++) {
$cdb->insert($load[$i], "ok");
}
$cdb->finish;
my $tmp = shift(@load);
push @load, $tmp;
$count++;
}
print $count, "\t";
unlink $tmpfile;
# DB
use DB_File;
use Fcntl qw(:flock);
use vars qw(%h);
my $dbh = tie %h, 'DB_File', $tmpfile;
my $fd = $dbh->fd;
open(DB_FH, "+<&=$fd") or die "$0: cannot open $tmpfile filehandle: $!\n";
for (my $i = 0; $i < $setsize; $i++) {
$h{$load[$i]} = "ok";
}
$count = 0;
$end = time + 30;
while (time < $end) {
flock(DB_FH, LOCK_EX) or die "$0: flock LOCK_EX failed: $!\n";
delete $h{$load0};
$h{$load[$setsize]} = "ok";
$dbh->sync;
flock(DB_FH, LOCK_UN) or die "$0: flock LOCK_UN failed: $!\n";
my $tmp = shift(@load);
push @load, $tmp;
$count++;
}
print $count, "\n";
close DB_FH;
undef $dbh;
untie %h;
unlink $tmpfile;
}
uAKRQypu60I7Lcqm
--DBIVS5p969aUjpLe
Content-Type: application/pgp-signature
Content-Disposition: inline
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.0.7 (GNU/Linux)
iD8DBQE9twcGHZWg9mCTffwRAu0UAJ0QSJKYqBm6HYE3/ypbvOlGEfyJyACgtLa0
H9svGNdTkcm/XNn/PrLOgSA=
=LtOG
-----END PGP SIGNATURE-----
DBIVS5p969aUjpLe
寄件者：Wietse Venema (wietse@porcupine.org)
主旨：Re: [ot] tinyCDB is available in packaged form
View this article only
新闻群组：mailing.postfix.users
日期：2002-10-23 14:06:04 PST
Bennett Todd:
> 2002-10-16-12:18:58 Bennett Todd:
> > 2002-10-15-15:43:50 Michael Tokarev:
> > > Bennett Todd wrote:
> > > >I'd have to do some experimenting, but I really suspect
> > > >pop-before-smtp would work very well indeed with cdb.
> > >
> > > One word: scalability. [...] It will work for small sites.
> >
> > Define "small".
> > [...]
> > Where's the crossover point? I.e. where does the cost of the CDB
> > generation grow to exceed the cost of doing the DB manipulation?
>
> Ok, in the interests of fairness, I've gotta come back and confess
> that my intuition was way, way off here, and Michael Tokarev's was
> right on.
>
> While cdb can work for small sites, where the load of doing things
> inefficiently doesn't make any difference anyway --- just what he
> suggested --- there's no "crossover point"; no matter how small
> the database, DB_Hash is faster than CDB_File for "updates" in all
> circumstances.
It may still be fast enough for small sites. With up to 1000
pop-before-smtp entries in a table, the cost of an update is only
1-2 milliseconds. On a system that is maintained by a private
person the impact would be negligible.
Thanks for measuring this.
Wietse
-
To unsubscribe, send mail to majordomo@postfix.org with content
(not subject): unsubscribe postfix-users
Post a follow-up to this message
Message 28 in thread
寄件者：Rahul Dhesi (dhesi@rahul.net)
主旨：Re: [ot] tinyCDB is available in packaged form
View this article only
新闻群组：mailing.postfix.users
日期：2002-10-23 15:30:07 PST
On Wed, Oct 23, 2002 at 04:31:02PM -0400, Bennett Todd wrote:
> While cdb can work for small sites, where the load of doing things
> inefficiently doesn't make any difference anyway --- just what he
> suggested --- there's no "crossover point"; no matter how small
> the database, DB_Hash is faster than CDB_File for "updates" in all
> circumstances.
If I were designing a pop-before-smtp system from scratch, I would do
it like this:
1. A function that creates (or updates the timestamp of) a file
representing an IP address. The filename is generated by replacing the
first N dots in the IP address with slashes. N is in the range 0 .. 3.
Number of system calls needed: two (an open and a close).
2. A function that returns true or false according as the file for a
given IP address exists or not. Number of system calls needed: one (a
stat).
3. Every half hour a cron job does this:
cd somedir; find . -type f -cmin +30 -print │ xargs rm -f
This is how I first implemented pop-before-smtp in 1997. I used N = 0.
As a possible optimization, the daemon that creates the files can
consult an in-memory hash table and avoid recreating/retouching the same
file more than once every 20 minutes.
As another optimization, we can run the cron job every 24 hours instead,
and have the lookup function (in item 2) return true only if the file is
no older than 30 minutes. This increases the number of system calls to
two: a gettimeofday and a stat.
I'm guessing that the reason we are discussing the efficiency of
different types of databases, instead of using the simple and efficient
scheme I have described above, is because Postfix's smtp server does not
have a built-in facility to look for a filename representing an IP
address. Thus I believe the cart is dragging the horse here.
Rahul
-
To unsubscribe, send mail to majordomo@postfix.org with content
(not subject): unsubscribe postfix-users
Post a follow-up to this message
Message 29 in thread
寄件者：Wietse Venema (wietse@porcupine.org)
主旨：Re: [ot] tinyCDB is available in packaged form
View this article only
新闻群组：mailing.postfix.users
日期：2002-10-23 15:34:28 PST
Wietse Venema:
> Bennett Todd:
> > 2002-10-16-12:18:58 Bennett Todd:
> > > 2002-10-15-15:43:50 Michael Tokarev:
> > > > Bennett Todd wrote:
> > > > >I'd have to do some experimenting, but I really suspect
> > > > >pop-before-smtp would work very well indeed with cdb.
> > > >
> > > > One word: scalability. [...] It will work for small sites.
> > >
> > > Define "small".
> > > [...]
> > > Where's the crossover point? I.e. where does the cost of the CDB
> > > generation grow to exceed the cost of doing the DB manipulation?
> >
> > Ok, in the interests of fairness, I've gotta come back and confess
> > that my intuition was way, way off here, and Michael Tokarev's was
> > right on.
> >
> > While cdb can work for small sites, where the load of doing things
> > inefficiently doesn't make any difference anyway --- just what he
> > suggested --- there's no "crossover point"; no matter how small
> > the database, DB_Hash is faster than CDB_File for "updates" in all
> > circumstances.
>
> It may still be fast enough for small sites. With up to 1000
> pop-before-smtp entries in a table, the cost of an update is only
> 1-2 milliseconds. On a system that is maintained by a private
> person the impact would be negligible.
Argh, that should be 30-60 milliseconds.
> Thanks for measuring this.
>
> Wietse
> -
> To unsubscribe, send mail to majordomo@postfix.org with content
> (not subject): unsubscribe postfix-users
>
>
-
To unsubscribe, send mail to majordomo@postfix.org with content
(not subject): unsubscribe postfix-users
Post a follow-up to this message
Message 30 in thread
寄件者：Matthias Andree (ma@dt.e-technik.uni-dortmund.de)
主旨：Re: [ot] tinyCDB is available in packaged form
View this article only
新闻群组：mailing.postfix.users
日期：2002-10-24 07:10:04 PST
Rahul Dhesi
> I'm guessing that the reason we are discussing the efficiency of
> different types of databases, instead of using the simple and efficient
> scheme I have described above, is because Postfix's smtp server does not
> have a built-in facility to look for a filename representing an IP
> address. Thus I believe the cart is dragging the horse here.
You can get all this for free with SMTP AUTH. No database required
except that one which holds the password and has a few writes a day
only, even at big sites. No file system sweeps necessary to expire old
relay permit files, and no cache to avoid touching the file system too
often.
--
Matthias Andree
-
To unsubscribe, send mail to majordomo@postfix.org with content
(not subject): unsubscribe postfix-users
Post a follow-up to this message
寄件者：Matthias Andree (ma@dt.e-technik.uni-dortmund.de)
主旨：Re: [ot] tinyCDB is available in packaged form
View this article only
新闻群组：mailing.postfix.users
日期：2002-10-28 02:54:08 PST
"Peter H. Coffin"
>> You can get all this for free with SMTP AUTH. No database required
>> except that one which holds the password and has a few writes a day
>> only, even at big sites. No file system sweeps necessary to expire old
>> relay permit files, and no cache to avoid touching the file system too
>> often.
>
> Unless setup and tuning time costs nothing, based on the number of
> difficulties people have setting SASL up, you have a different
> definition of "free" than I do.
"free" here means "free of additional system load".
I've been using SASL1 with the pwcheck daemon to authenticate users
against /etc/shadow or with sasldb without difficulties. When using the
pwcheck daemon, make sure the permissions to /var/pwcheck are right, I'm
using
drwx--x--- 2 root postfix 58 Sep 26 23:30 /var/pwcheck/
Note group ownership and execute permission (That's chown root:postfix
and then chmod 0710.)
I'm aware that this scheme has been superseded in the meanwhile, but it
still works with the latest Cyrus-SASL v1 version. It is disputable
whether this is safe enough to the local machine -- you get to balance
"local security" (use POP3-before-SMTP) against "network security" (use
SASL for SMTP AUTH).
--
Matthias Andree
Posted by hzqbbc at 03:31 PM | Comments (0)
November 11, 2002
对Postfix进行性能调整的高级策略
注意看viktor的发言
注意Wietse的性能优化 （强烈建议）
注意使用TRAM提高性能的方法
所有留言及回应
Message 1 in thread
寄件者：Simon White (simon@mtds.com)
主旨：Re: Postfix Choking?
View this article only
新闻群组：mailing.postfix.users
日期：2002-10-07 04:10:07 PST
07-Oct-02 at 00:54, Brian Wizard (brian@xtnd.net) wrote :
> >From what I have heard, postfix is supposedly a robust and scalable MTA.
> In the past 24 hours I have loaded postfix on a server running FreeBSD
> 4.7-RC with dual 2 gig Xeon procs with 1 gig of RAM and an SMP kernel.
> However, Postfix seems to be running out of resources as time goes on.
Did you see the post this morning re:
http://www.stahl.bau.tu-bs.de/~hildeb/postfix/bottleneck.shtml
--
[Simon White. vim/mutt. simon@mtds.com. GIMPS:1.930% see www.mersenne.org]
If it dies, it's biology. If it blows up, it's chemistry, and if it
doesn't work, it's physics.
[Linux user #170823 http://counter.li.org. Home cooked signature rotator.]
-
To unsubscribe, send mail to majordomo@postfix.org with content
(not subject): unsubscribe postfix-users
Message 2 in thread
寄件者：Brian Wizard (brian@xtnd.net)
主旨：Postfix Choking?
View this article only
新闻群组：mailing.postfix.users
日期：2002-10-07 04:20:25 PST
Greetings,
From what I have heard, postfix is supposedly a robust and scalable MTA.
In the past 24 hours I have loaded postfix on a server running FreeBSD
4.7-RC with dual 2 gig Xeon procs with 1 gig of RAM and an SMP kernel.
However, Postfix seems to be running out of resources as time goes on. I
am very new to Postfix and the only tuning options I have set in main.cf
are the following:
qmgr_message_active_limit = 7000
local_destination_concurrency_limit = 600
default_destination_concurrency_limit = 600
default_process_limit=600
process_limit=600
When I raise the process_limits/concurrency_limits to more than 600 the
server starts swapping for memory? And outputs the following message in
/var/log/messages:
Oct 7 08:38:39 server1 postfix/trivial-rewrite48810: fatal: accept
connection: Too many open files
If I leave the settings the way they are, postfix takes a long
time to respond on port 25 (more than 1 minute) after it has been running
for a few hours.
I have played with many kernel settings and nothing seems to help, could
this just be a limitation on Postfix? This server is processing close to 2
million emails/day.
I don't think postfix should need more than this...
root@server1:~# su - postfix
postfix@server1:~% ulimit -n
39218
I have maxfiles set to the maximum amount in the kernel as well.
root@server1:~# sysctl kern.maxfiles
kern.maxfiles: 65536
Has anyone run into a situation such as this before? Or can anyone make a
suggestions as far what optimization settings should be incorporated into
postfix for email systems with high loads? BTW, I have been running qmail
with no problems, but I would really like to switch to postfix if at all
possible. Thanks for your help.
Brian Wizard
-
To unsubscribe, send mail to majordomo@postfix.org with content
(not subject): unsubscribe postfix-users
Message 3 in thread
寄件者：Ralf Hildebrandt (Ralf.Hildebrandt@charite.de)
主旨：Re: Postfix Choking?
View this article only
新闻群组：mailing.postfix.users
日期：2002-10-07 04:26:03 PST
On Mon, Oct 07, 2002 at 12:54:49AM -1000, Brian Wizard wrote:
> When I raise the process_limits/concurrency_limits to more than 600 the
> server starts swapping for memory?
Shit happens. I guess you can never have enough RAM, can you?
> Oct 7 08:38:39 server1 postfix/trivial-rewrite48810: fatal: accept
> connection: Too many open files
The kernel cannot handle that many open files.
--
Ralf Hildebrandt (Im Auftrag des Referat V a) Ralf.Hildebrandt@charite.de
Charite Campus Mitte Tel. +49 (0)30-450 570-155
Referat V a - Kommunikationsnetze - Fax. +49 (0)30-450 570-916
Why you can't find your system administrators:
(S)he's out buying some caffeine.
-
To unsubscribe, send mail to majordomo@postfix.org with content
(not subject): unsubscribe postfix-users
寄件者：Chris Laco (claco@summitracing.com)
主旨：RE: Postfix Choking?
View this article only
新闻群组：mailing.postfix.users
日期：2002-10-07 06:36:16 PST
Have you done this stuff yet?
http://www.daemonnews.org/200108/benchmark.html
寄件者：lst_hoe (lst_hoe@kwsoft.de)
主旨：Re: Postfix Choking?
View this article only
新闻群组：mailing.postfix.users
日期：2002-10-07 07:34:01 PST
At 10:10 07.10.2002 -0400, Victor.Duchovni@morganstanley.com wrote:
>Two million messages a day is a high load: average of 23 messages per
>second, with presumably much higher peaks during the day. How high are
>the peaks? Are you sure your I/O subsystem is up to the task. What maps
>are you using for your transport table? Figure out the expected query rate
>(at least 3 per recipient, more if you are using the snapshot release) and
>query latency. Make sure that queries against the transport table can be
>completed in ~5ms (1000/23/10) perhaps less if the intra business day mail
>flow is substantially higher than 23 msgs/sec.
>
>Consider mjt's CDB patch for as many maps as possible. Show output of
>"postconf -n".
May i raise some questions :
Where do this mails go to?
Who has that great volume in the pacific provider area (xtnd.net)?
What kind of people who don′t even read the FAQ try to process 2 million
mails a day?
Strange sender-name, maybe related to this one http://www.brianwizard.com/ :-)
Regards
--
Andreas H?dle
Kühn & Weyh Software GmbH
WWW.KWSOFT.DE
-
To unsubscribe, send mail to majordomo@postfix.org with content
(not subject): unsubscribe postfix-users
Message 6 in thread
寄件者：Victor.Duchovni@morganstanley.com (Victor.Duchovni@morganstanley.com)
主旨：Re: Postfix Choking?
View this article only
新闻群组：mailing.postfix.users
日期：2002-10-07 07:52:03 PST
On Mon, 7 Oct 2002, Brian Wizard wrote:
> qmgr_message_active_limit = 7000
Don't lower this. If your machine has enough memory raise it to 20000 or
more, but not to solve this problem. It is unrelated to throughput.
> local_destination_concurrency_limit = 600
> default_destination_concurrency_limit = 600
Don't raise these.
> default_process_limit=600
Try 200-400.
> process_limit=600
>
This is not a postfix configuration parameter.
>
> When I raise the process_limits/concurrency_limits to more than 600 the
> server starts swapping for memory? And outputs the following message in
> /var/log/messages:
>
> Oct 7 08:38:39 server1 postfix/trivial-rewrite48810: fatal: accept
> connection: Too many open files
>
The trivial-rewrite daemon cannot handle that many concurrent connections.
You must be using very slow maps. Is LDAP or MySQL used for the transport
table by any chance? Keep the transport table fast, don't use slow
transport maps.
> If I leave the settings the way they are, postfix takes a long
> time to respond on port 25 (more than 1 minute) after it has been running
> for a few hours.
See above.
>
> I have played with many kernel settings and nothing seems to help, could
> this just be a limitation on Postfix? This server is processing close to 2
> million emails/day.
>
Two million messages a day is a high load: average of 23 messages per
second, with presumably much higher peaks during the day. How high are
the peaks? Are you sure your I/O subsystem is up to the task. What maps
are you using for your transport table? Figure out the expected query rate
(at least 3 per recipient, more if you are using the snapshot release) and
query latency. Make sure that queries against the transport table can be
completed in ~5ms (1000/23/10) perhaps less if the intra business day mail
flow is substantially higher than 23 msgs/sec.
Consider mjt's CDB patch for as many maps as possible. Show output of
"postconf -n".
--
Viktor.
-
To unsubscribe, send mail to majordomo@postfix.org with content
(not subject): unsubscribe postfix-users
Message 7 in thread
寄件者：Victor.Duchovni@morganstanley.com (Victor.Duchovni@morganstanley.com)
主旨：Re: Postfix Choking?
View this article only
新闻群组：mailing.postfix.users
日期：2002-10-07 11:50:03 PST
On Mon, 7 Oct 2002, Brian Wizard wrote:
>
>
> Hi Victor,
>
> Thanks for the helpful response. What you said sounds right, we have over
> 10,000 entries in a flat shadow file and trivial rewrite is probably
> choking there. So hashing the passwd file into a CDB should help you
> think? Thanks
No. Trivial rewrite does not consult the password file. It uses
"transport_maps" if set, but you do not set up transport_maps, so
trivial-rewrite is being starved for other reasons, see below, but do you
have enough disk I/O to handle the load? What is the nature of the
traffic, why are you processing 2 million messages a day. Are they inbound
or outbound?
> default_destination_concurrency_limit = 600
Lose this, leave set to default.
> default_process_limit = 600
Lower this (Try 400 or 200).
> local_destination_concurrency_limit = 600
Defitely don't do this! This is 600 concurrent connections to a
single mailbox! Leave at the default value of 2.
> local_recipient_maps = $alias_maps unix:passwd.byname
If your password file is not hashed (nsswitch.conf) and has 10,000
entries, this may cost you a lot here by substantially increasing
"smtpd" concurrency. Make sure to configure a fast random access path to
your user database. If you can't do this system-wide, create a separate
hashed database for use by Postfix and replace unix:passwd.byname with
hash:... (or CDB for even better performance).
> maps_rbl_domains = relays.visi.com
Is this providing timely responses? If not dump it and use someone who
does.
> qmgr_message_active_limit = 7000
Raise this to 20000 or leave the default alone at 10000
> smtpd_banner = $myhostname postfix
Why not leave the default alone. Putting "ESMTP" in the banner increases
the probability that the sender will use ESMTP and hence
predeclared size limits with pipelining.
--
Viktor.
Wietse关于Postfix优化的看法
寄件者：Wietse Venema (wietse@porcupine.org)主旨：Re: 1.000.000 email posted in a single delivery process
View: Complete Thread （共有 46 条留言）
Original Format
新闻群组：mailing.postfix.users
日期：2001-09-26 14:14:54 PST
If you want to deliver 1000000 queue files in 3 hours, that is 10
milliseconds per file. That seems a bit optimistic to me, considering
the latencies of real disks (every file needs to be read from disk,
the cache is cold) and of real networks (having a local DNS server
should really make a difference).
Neither qmail nor Postfix currently has connection caching, which
can make a difference sending mail to sites that have slow servers.
0 - In the /etc/syslog.conf file, put a - in front of the maillog
filename, so that the stupid syslogd does not hammer the disk
after writing each logfile record.
1 - The default settings of two levels of directory hashing in the
queue should suffice. If you disagree, change this with:
postconf -e hash_queue_depth=3
postfix reload
2 - Configure an adequate number of delivery agents in the master.cf
file: 500+. On your typical Linux box, this will run the kernel
out of resources and Postfix will slam on the brakes until you
fix it.
3 - Test the setup with the smtp-source and smtp-sink programs.
For the test, specify a relayhost name that points to the
smtp-sink program, and send messages with thousands of recipients
into Postfix, and have Postfix perform one-recipient deliveries.
On the Postfix host:
postconf -e relayhost=smtp-sink.host:9999 \
smtp_destination_recipient_limit=1
On the mail sink host (can be same as postfix host, if you have
enough memory and process slots and space for open files):
./smtp-sink -c :9999 1000
On the smtp-source host (can be same as postfix host):
./smtp-source -c -m 1000 -s 10 -r 10000 -t foo@postfix.host postfix.host
Postfix breaks up the deliveries into transactions of 1 recipient
each, so one message with 10000 recipients will keep lots of
delivery agents busy.
4 - Before submitting the real mail:
postconf -e defer_transports=smtp relayhost= \
smtp_destination_recipient_limit=\$default_destination_recipient_limit
postfix reload
(and don't let anyone execute a "sendmail -q" command).
5 - SUBMIT THE MAIL VIA SMTP, not via /usr/sbin/sendmail. Otherwise
you are throwing away all the performance.
6 - On Doomsday,
postconf -e defer_transports=
postfix reload
sendmail -q
7 - From here on, do not touch the box. Each time you stop or reload
Postfix, or do a sendmail -q, it slows down dramatically.
Wietse
Alexander:
> (I've posted this message in QMAIL and EXIM mailing list.
> Many people say me that POSTFIX can do a good work)
>
> This is a case study and it's NOT for create a spamming system or other
> terrible mail bombing system.
> I HATE them !
>
> Title: the best mode to deliver 1.000.000 email messages (recipient are ALL
> different and email are ALL different) in few hours during a single delivery
> session.
>
> You suppose to have a SMP Linux server (i.e. dual P-III 800 Mhz, 512 MB Ram,
> SCSI Raid 5, kernel 2.4.9 and RaiserFS) and a good internet connection (4
> Mbit/sec full bandwidth) and a fast DNS server near me.
>
> During the night a program running on the same server creates 1 million of
> email (getting user data and infos data from a SQL DB).
> This process need about 3 hours. Meanwhile email are created, they are
> "injected" in MTA queue (i.e. using qmail-inject or
> exim -odq -t) and they are NOT delivered: they stay in queue.
> The email are all different but short: about 800-1000 bytes each and the
> recipients are all different and are all remote.
> Yes, we can have a vary big number of different recipient domain but
> statistically we also have a large number of email directed on mass email
> system (Hotmail, Yahoo...)
>
> So 1.000.000 × 1.000 bytes = 1 GBytes data. Yes, like a technical
> information email system with personalized info for each recipient.
>
> Then every day at 8.00 AM the remote delivery needs to start (exim -q ,
> qmail-remote) and delivery is done once per day.
>
> Target: send 1 million of email in a very short time, shortest as possibile,
> it must end as soon as possibile. Bandwidth occupation is not a problem: we
> can use all 4 Mbit/sec (about 1 GB / 500 Kbytes/sec = about 2000 seconds , 1
> hour and few minutes. 2 or 3 hours are ok anyway).
>
> And now the questions:
>
> 1) QMAIL vs EXIM vs POSTFIX: which is the best MTA to manage 1.000.000 email
> in queue ?
> (EXIM with option "split_spool_directory" and QMAIL patched to have a more
> "deep" in queue tree, POSTIFIX ??)
>
> 2) EXIM can deliver all messages during a single SMTP session to the same
> email system (useful for mass system like HOTMAIL,YAHOO, ....), QMAIL no.
> Is it faster to have either for example 300 (more or less) qmail-remote
> process running and sending 1 messages each or for example 20 (more or
> less) "exim -q" process running sending more messages in a single SMTP
> connection ? Remember: the target is to use less time than possible.
>
> Thanks for any suggestion about.
> POSTFIX ,I've never used it, can be an alternative ?
>
>
>
另外一个人的性能调整建议，不过是产品化的建议。不错！虽然有些我不同意。
寄件者：Liviu Daia (Liviu.Daia@imar.ro)
主旨：Re: 1.000.000 email posted in a single delivery process
View this article only
新闻群组：mailing.postfix.users
日期：2001-09-27 02:11:08 PST
On 27 September 2001, Liviu Daia
[...]
> For a today's PC, something like 40-50 messages / second would still
> be an optimistic estimate;
That is, over a LAN. Expect a much lower rate over Intermet,
because of dead or slow destinations.
> that's about 6 times slower that what you want.
Regards,
Liviu Daia
--
Dr. Liviu Daia e-mail: Liviu.Daia@imar.ro
Institute of Mathematics web page: http://www.imar.ro/~daia
of the Romanian Academy PGP key: http://www.imar.ro/~daia/daia.asc
-
To unsubscribe, send mail to majordomo@postfix.org with content
(not subject): unsubscribe postfix-users
第 9 条留言
寄件者：Liviu Daia (Liviu.Daia@imar.ro)
主旨：Re: 1.000.000 email posted in a single delivery process
View this article only
新闻群组：mailing.postfix.users
日期：2001-09-27 02:11:20 PST
On 26 September 2001, Alexander
[...]
> Title: the best mode to deliver 1.000.000 email messages (recipient
> are ALL different and email are ALL different) in few hours during a
> single delivery session.
>
> You suppose to have a SMP Linux server (i.e. dual P-III 800 Mhz, 512
> MB Ram, SCSI Raid 5, kernel 2.4.9 and RaiserFS) and a good internet
> connection (4 Mbit/sec full bandwidth) and a fast DNS server near me.
Wietse already explained how to setup Postfix for best performance.
Here's an additional checklist, mainly for the rest of the system:
- if possible, use OpenBSD or FreeBSD with softupdates rather than
Linux;
- if you insist on using Linux, use a 2.2.x kernel, not a 2.4.x one, and
turn OFF the swap; buy more RAM if you feel you're running out of it;
- use RAID 0+1, not 5, and (if you have that choice) mirror the stripes
instead of striping the mirrors;
- use ext2; Xfs and Jfs are not yet ready for prime time, and everything
else is slower than ext2 for mail handling;
- install a caching DNS on the SAME machine as the mail server, a nearby
one is not good enough; use djbdns for that instead of bind; turn off
logging for lookups, and give it plenty of memory to keep the cache;
- if you trust your UPS, turn off sync writes to the queue
("chattr -R -S /var/spool/postfix")
- if possible, send the logs to another machine instead of writing them
to a local file, or at least put the logs on different disk spindles
than the queue;
- apply Michael Tokarev's patch for CDB maps, and use CDB maps instead
of hash.
(I'm sure I forgot a few things too.)
[...]
> Target: send 1 million of email in a very short time, shortest as
> possibile, it must end as soon as possibile. Bandwidth occupation is
> not a problem: we can use all 4 Mbit/sec (about 1 GB / 500 Kbytes/sec
> = about 2000 seconds , 1 hour and few minutes. 2 or 3 hours are ok
> anyway).
[...]
As Wietse said, that's overly optimistic. With careful tuning,
Postfix would perform better than Exim and Qmail, but IMHO, no MTA would
be able to send 1 million messages in an hour, even over a LAN. No
offense intended, but people who set themselves such a target have never
managed a mail systems in the real world. For a today's PC, something
like 40-50 messages / second would still be an optimistic estimate;
that's about 6 times slower that what you want.
Regards,
Liviu Daia
寄件者：Wietse Venema (wietse@porcupine.org)
主旨：Re: 1.000.000 email posted in a single delivery process
View this article only
新闻群组：mailing.postfix.users
日期：2001-09-27 06:45:59 PST
Matthias Andree:
> Well, in a LAN, it might actually be possible to get half a million
> mails through per hour without extensive tuning - however, that's
> without VERP and it assumes your destination machines are FAST.
To give you a data point: running smtp-source against smtp-sink
on a 750MHz pentium FreeBSD 4.3 machine without going over a LAN,
with 100 parallel connections:
tail% /usr/bin/time ./smtp-source -d -m 1000 -s 100 tail:9999
1.21 real 0.06 user 0.50 sys
That's 1 million messages in 20 minutes.
Same email load, two hosts connected by a 100BaseT ethernet switch,
target is P550 FreeBSD 4.1, with 100 parallel connections:
tail% /usr/bin/time ./smtp-source -d -m 1000 -s 100 spike:9999
1.05 real 0.06 user 0.23 sys
The times are pretty much comparable.
Do not expect anything near this performance with a queue on rotating
disk drives, though.
Wietse
使用内存RAM来提高Postfix的性能
如何使用TRAM提高你计算机的速度？英文的哦。不过太恐怖了。用ram。。TRAM
以下是邮件列表的一个人说的话，确实是“化整为零”将负载分布到廉价的*nix box
============
寄件者：Craig Sanders (cas@taz.net.au)
主旨：Re: Making postfix faster?
View this article only
新闻群组：mailing.postfix.users
日期：2001-07-05 21:54:15 PST
On Thu, Jul 05, 2001 at 08:08:28PM -0400, Myke Olson wrote:
> Does anyone have any suggestions on making Postfix faster at sending mail?
> [...]
> We're running the latest version on a Sun Ultra1/140 and are only able
> to send about 160 emails/minute.
for this application, you may be better of with lots of cheap little
linux or freebsd boxes running postfix than one big Sun box.
one of the best ways is to have multiple cheap boxes rather than one big
expensive box.
this may require modifying your mailing list software so that it sends
part of each list to different machines. e.g. if you have 5 postfix smtp
sender boxes, then split the list in 5.
if you send individual messages to each subscriber rather than one message
bcc-ed to everyone then you could use a load balancer to spread the mail
load across multiple machine. e.g. linux virtual server (LVS) makes an
excellent load balancer - http://www.linuxvirtualserver.org/
alternatively, use a large RAID array of as many disks as you can
afford. the more drives in the array, the better. e.g. if you want 72GB
of total space, it is much faster to have 8 × 9GB drives than 2 × 36GB,
or 4 × 18GB.
a solid-state disk (SSD) will be even faster than a raid array.
the way to scale up a disk I/O-bound application (such as sending to
enormous mailing lists) is to speed up the disk by adding more physical
disks and/or more machines or by using faster disks (e.g. SSDs).
combining the alternatives above will get even better results - e.g. an
array of 5 linux + postfix boxes, each with a 1GB SSD queue drive.
> Some ideas that I had were to somehow make it give up on the first
> try if the DNS record for a particular host didn't exist (i.e. a user
> signed up with jdflkl@kjdjdflk.com as their email address) or somehow
> moving the spool directory into a RAM disk. But I couldn't figure out
> how to do that....
you could do that if you didn't care whether the messages got lost if
the system crashed or lost power.
a safer alternative is to use a non-volatile solid-state disk for the
postfix queue. in essence, they're a RAM disk with a battery backup
and a SCSI interface, in a standard disk form factor. they cost a
fortune, though, for any reasonable size....several thousand dollars per
gigabyte, last time i looked into them.
if 1 or 2 GB is enough for your postfix queue, then an SSD may be your
best option.
craig
第 5 条留言
寄件者：Len Conrad (LConrad@Go2France.com)
主旨：Re: Multiple frontend Postfix - One RAID5 storage
View this article only
新闻群组：mailing.postfix.users
日期：2003-01-14 06:47:20 PST
>mailserver-front--│
>mailserver-front--│--> store spool on 1 raid5 dev, seperate spools
>mailserver-front--│
It sounds like your are getting ready for very high volumes, so why have
you chosen the RAID that is slowest for writing? RAID1 has faster writing.
And of course going across a network to the spool dir is horribly slower
than local disk i/o.
RAID5 is fine for database work where read:write it 90:10, but mail is
closer to 50:50.
构筑高可靠的app，这个东西可以做个随时监测smtp 机器的mon。。不错不错。
Wackamole
Posted by hzqbbc at 10:07 PM | Comments (0)
为实现大容量邮件系统所设计LDAP ldif 的构思
在如何构Postfix(MTA)+Maildrop(MDA)+SqWebMail(web-MUA)+IMAP/POP3+OpenLDAP 组成的邮件系统时，自己考虑了不少问题，现在并没完成。
良好的postfix配置及优化好的ldap结构设计能大幅提高性能。例如：
◎适当的进行压力测试（例如postal）可以估算出系统能承受的负载量，通过结果来适当调整smtp(output)/smtpd(input)的最大进程数来达到一个综合的最优化结果。
同时注意系统的调节，Linux的注意打开文件数/用户进程上限及内存、资源分配，FreeBSD则注意对内核进行微调，如kern.maxprocperfiles等（详细看freebsd的tunning）
◎注意对Postfix里的一些小参数仔细设置，如一些timeout值，进程上限,lock_delay, queue的lifetime,refresh time,ipc等的timeout 和idle timeout值，这些可仔细看man
◎如果virtual_domains不多或者更新较少的话，强烈建议使用hash来保存，而不是ldap。因为至少每一封信，postfix都至少要查询3-5次ldap，如果对virtual_maps的查询改成了hash后，就减少了20%-25%的查询量，意义非比寻常。
◎尽量在SMTP会话过程中就reject掉垃圾信，可以减少很多无用工作。因此各种check及限制的手段就显必不可少了。
◎尽量向本地的ldap查询，并且严格注意查询的timeout设置，并安排多个ldap的server做冗余（提高可靠性）并使用快速的网络连接。最好能设计成并发的query就比较好了。
◎注意使用高速的DB（例如最新的berkeley db）而不是用陈旧的缓慢的db。并且配置适当的slave ldap服务器提高分布能力。
◎对virtual_mailbox_maps及access_maps等分别指定不同的ldap服务器，可人为的进行分布查询，将负载分担。由于基本上每个mail都要同时查询这些maps，所以负载是相当均衡的。
◎对pipe改造。利用ldap查询时得到的用户目录（前提是本地的虚拟用户）直接传递给MDA (maildrop)进行直接投递，免除了maildrop再次查询ldap所带来的重复问题。
part2
补充一点点：
◎使用Berkerly DB代替GDBM，作为ldap的backend，db3/db4等的速度大大的快于gdbm，甚至gdbm使用了优化的index也没办法达到dbx的一般速度。
◎关闭log功能，即"loglevel 0"，这样可以大幅度提高ldap查询速度，难以置信的快！
详细见：
Close loglevel to improve performance
◎mydestination部分，如果要自动辨别新加的domain，那么使用$virtual_maps进去，但这可能会导致postfix多次查询ldap（具体待验证）如果是，那么最好取消掉，换之以确定的数据（如显式的填写好domain，或者以hash的形式作为virtual_maps比较好，好处同上）
part3
有人做了个mydestination用ldap的简单测试结果以及一些讨论，可以看看：
part4 (updated!)
一些有关于Email及LDAP的RFC/Draft，值得一看，也是我的参考文章。
MIME Directory Profile for LDAP Schema
有关Postfix VDA讨论，其中不少关于LDAP schema的内容
LDAP Schema for Intranet Mail Routing
LDAP-based Routing of SMTP Messages
LDAP-based Routing of SMTP Messages
Using LDAP for SMTP Mailing Lists & Aliases
LDAPv3: A Collection of User Schema (最新的user schema设计)
Schema Functions for the LDAP C API (不错的c API参考)
Posted by hzqbbc at 07:39 PM | Comments (1)
有关Postfix 的虚拟用户支持的大讨论
原始地址：
virtualizing local users
--------------
Message 1 in thread
寄件者：Phil Howard (phil-postfix-users@ipal.net)
主旨：virtualizing local users
View this article only
新闻群组：mailing.postfix.users
日期：2002-07-01 02:43:07 PST
I am trying to virtualize the users on a new mail server.
What I mean by that is that the user names will not be listed
in the /etc/passwd file. There is another file which looks
like the first 2 fields of /etc/passwd or /etc/shadow which
contains the list of valid users. The main question is how
might I get Postfix to check this file for known users?
I do have the option to make this be virtual or not virtual
since this mail server is serving only one space of usernames
(that is, xyzzy@foo and xyzzy@bar are the same mailbox and
same user). But I have been looking through the various ways
to configure either approach and I simply don't find a way to
do this. Mail is being delivered OK, and picked up OK using
vm-pop3d as the pop3 server (which reads the alternate file
to get user passwords). So once I can get Postfix to know
the users are existing (if they are in that file), then it
should all be working.
I definitely do not want to add these users to /etc/passwd or
/etc/shadow. There are conflicts between the user space for
mail and the user space for shell logins (which have no local
mail), so this isn't an option.
I also want to avoid (means if there is no other way, then this
can be done, but I definitely want to exhaust all other possible
approaches first) having a separate file to maintain just for
Postfix to do user lookups. Basically this means if there is
a way to use the shadow file format, that's preferred, but any
map format Postfix can handle will do as a last resort.
No single configuration description pops out at me and says this
will do what I want. And I can't see any combinations that
might create the same effect. Anyone have any ideas on this?
Note that there is no virtual translation. User "xyzzy" has a
mailbox named "xyzzy" in (via symlink) /var/spool/mail in the
traditional mailbox format.
To unsubscribe, send mail to majordomo@postfix.org with content
(not subject): unsubscribe postfix-users
Message 2 in thread
寄件者：Jean-Pierre Schwickerath (lists@schwicky.net)
主旨：Re: virtualizing local users
View this article only
新闻群组：mailing.postfix.users
日期：2002-07-01 03:03:54 PST
Phil Howard
> What I mean by that is that the user names will not be listed
> in the /etc/passwd file. There is another file which looks
> like the first 2 fields of /etc/passwd or /etc/shadow which
> contains the list of valid users. The main question is how
> might I get Postfix to check this file for known users?
There is a parameter called local_recipient_maps
cut
local_recipient_maps = $alias_maps unix:passwd.byname
cut
There you could add your file(s) in any format that postfix
understands...
Jean-Pierre
--
Powered by Linux From Scratch - http://schwicky.net
PGP Key ID: 0xEE6F49B4 - AIM/Jabber: Schwicky - ICQ: 4690141
Nothing is impossible... Everything is relative!
-
To unsubscribe, send mail to majordomo@postfix.org with content
(not subject): unsubscribe postfix-users
Message 3 in thread
寄件者：Phil Howard (phil-postfix-users@ipal.net)
主旨：Re: virtualizing local users
View this article only
新闻群组：mailing.postfix.users
日期：2002-07-01 04:41:46 PST
On Mon, Jul 01, 2002 at 12:02:14PM +0200, Jean-Pierre Schwickerath wrote:
│ Phil Howard
│
│ > What I mean by that is that the user names will not be listed
│ > in the /etc/passwd file. There is another file which looks
│ > like the first 2 fields of /etc/passwd or /etc/shadow which
│ > contains the list of valid users. The main question is how
│ > might I get Postfix to check this file for known users?
│
│ There is a parameter called local_recipient_maps
│ cut
│ local_recipient_maps = $alias_maps unix:passwd.byname
│ cut
│
│ There you could add your file(s) in any format that postfix
│ understands...
So what do I put there to make it understand /etc/passwd format?
Is there a document or man page that lists all the formats?
--
-----------------------------------------------------------------
│ Phil Howard - KA9WGN │ Dallas │ http://linuxhomepage.com/ │
│ phil-nospam@ipal.net │ Texas, USA │ http://phil.ipal.org/ │
-----------------------------------------------------------------
-
To unsubscribe, send mail to majordomo@postfix.org with content
(not subject): unsubscribe postfix-users
Message 4 in thread
寄件者：Victor.Duchovni@morganstanley.com (Victor.Duchovni@morganstanley.com)
主旨：Re: virtualizing local users
View this article only
新闻群组：mailing.postfix.users
日期：2002-07-01 07:57:36 PST
On Mon, 1 Jul 2002, Phil Howard wrote:
> I am trying to virtualize the users on a new mail server.
> What I mean by that is that the user names will not be listed
> in the /etc/passwd file. There is another file which looks
> like the first 2 fields of /etc/passwd or /etc/shadow which
> contains the list of valid users. The main question is how
> might I get Postfix to check this file for known users?
This is the wrong question. The right question is:
What delivery agent and POP/IMAP server combination should I use to
deliver mail to users without a shell account (not listed in /etc/passwd)?
For a specific choice of delivery agent and POP/IMAP server, what is the
best way to manage the virtual user account information?
Is it possible to enter the virtual user accounts into just one
database that will be used by all components ("smtpd" user validation,
delivery agent, POP/IMAP server)?
The most important factor is your choice of POP/IMAP server. Some can be
integrated tightly with Postfix and some cannot.
Your best bet is likely (I am not an expert on this topic, this is
my impression from recent posts to the list) Courier IMAP. There is a
patch for Postfix to enable it to use "userdb" as a Postfix map type.
Check some of the Courier HOWTO documents recently advertised on the list.
If none provide sufficient detail, perhaps someone can volunteer the
missing recipes.
Postfix has no support for "password-like" maps except /etc/passwd
accessed via getpwnam(3).
--
Viktor.
-
To unsubscribe, send mail to majordomo@postfix.org with content
(not subject): unsubscribe postfix-users
寄件者：Phil Howard (phil-postfix-users@ipal.net)
主旨：Re: virtualizing local users
View: Complete Thread (36 articles)
Original Format
新闻群组：mailing.postfix.users
日期：2002-07-01 08:55:35 PST
On Mon, Jul 01, 2002 at 10:56:21AM -0400, Victor.Duchovni@morganstanley.com wrote:
│ On Mon, 1 Jul 2002, Phil Howard wrote:
│
│ > I am trying to virtualize the users on a new mail server.
│ > What I mean by that is that the user names will not be listed
│ > in the /etc/passwd file. There is another file which looks
│ > like the first 2 fields of /etc/passwd or /etc/shadow which
│ > contains the list of valid users. The main question is how
│ > might I get Postfix to check this file for known users?
│
│ This is the wrong question. The right question is:
│
│ What delivery agent and POP/IMAP server combination should I use to
│ deliver mail to users without a shell account (not listed in /etc/passwd)?
The POP server is already chosen.
The answer to this question will be interesting for the next project.
But there is a different set of requirements for that one. I can post
them if you would like to offer advice on that.
│ For a specific choice of delivery agent and POP/IMAP server, what is the
│ best way to manage the virtual user account information?
│
│ Is it possible to enter the virtual user accounts into just one
│ database that will be used by all components ("smtpd" user validation,
│ delivery agent, POP/IMAP server)?
│
│
│ The most important factor is your choice of POP/IMAP server. Some can be
│ integrated tightly with Postfix and some cannot.
│
│ Your best bet is likely (I am not an expert on this topic, this is
│ my impression from recent posts to the list) Courier IMAP. There is a
│ patch for Postfix to enable it to use "userdb" as a Postfix map type.
│ Check some of the Courier HOWTO documents recently advertised on the list.
│ If none provide sufficient detail, perhaps someone can volunteer the
│ missing recipes.
Courier IMAP did not meet migration requirements for this server.
Mailbox format is required in this case so that user mail is not
lost. I will be using rsync to syncronize the mailboxes from the
old machine to the new one, then transfer IP addresses and restart.
The POP server used is vm-pop3d. It is past the time to allocate
the additional time to change the software used or to work out the
strategy to change from mailbox to maildir format. Physical move of
servers takes place 13 July. The migration must take place at the
maintenance window allocated for least user disruption.
For another project as mentioned above, Courier IMAP appears to be
a good choice. I don't know what you mean by "userdb", but then I
haven't gotten to the point of fully RTFM-ing Courier IMAP, yet.
Maybe this will all be obvious at that time.
│ Postfix has no support for "password-like" maps except /etc/passwd
│ accessed via getpwnam(3).
So the workaround is to have 2 maps, one for vm-pop3d to look up its
passwords, another for postfix to verify the existance of users. It's
something I wanted to avoid, but I have changed to another strategy
to manage accounts on this machine. It's quick and dirty. Each user
has a single file in a directory. The file contains their password.
These files are collected to construct the two needed files (and then
postmap is run to generate the hash db file Postfix will use).
At this point, Postfix is not recognizing users in the file even
though it is specified in local_recipient_maps. I just started
working on this so I haven't vetted all the possible errors, yet.
But it looks as though local_recipient_maps has no effect as it
still recognizes users that do happen to be in /etc/passwd.
--
-----------------------------------------------------------------
│ Phil Howard - KA9WGN │ Dallas │ http://linuxhomepage.com/ │
│ phil-nospam@ipal.net │ Texas, USA │ http://phil.ipal.org/ │
-----------------------------------------------------------------
寄件者：Ralf Hildebrandt (Ralf.Hildebrandt@charite.de)
主旨：Re: virtualizing local users
View this article only
新闻群组：mailing.postfix.users
日期：2002-07-01 10:09:57 PST
On Mon, Jul 01, 2002 at 10:48:03AM -0500, Phil Howard wrote:
> Courier IMAP did not meet migration requirements for this server.
> Mailbox format is required in this case so that user mail is not
> lost. I will be using rsync to syncronize the mailboxes from the
> old machine to the new one, then transfer IP addresses and restart.
Hello? There are conversion scripts for mbox -> maildir
> The POP server used is vm-pop3d. It is past the time to allocate
> the additional time to change the software used or to work out the
> strategy to change from mailbox to maildir format. Physical move of
> servers takes place 13 July. The migration must take place at the
> maintenance window allocated for least user disruption.
You can even convert on the fly! New delivery to Maildir, then, while
new mail is coming in you can still convert the old mailboxes in the
background.
--
Ralf Hildebrandt (Im Auftrag des Referat V A) Ralf.Hildebrandt@charite.de
Charite Campus Virchow-Klinikum Tel. +49 (0)30-450 570-155
Referat V A - Kommunikationsnetze - Fax. +49 (0)30-450 570-916
"Java is, in many ways, C++--." - Michael Feldman.
-
To unsubscribe, send mail to majordomo@postfix.org with content
(not subject): unsubscribe postfix-users
Message 7 in thread
寄件者：Phil Howard (phil-postfix-users@ipal.net)
主旨：Re: virtualizing local users
View this article only
新闻群组：mailing.postfix.users
日期：2002-07-01 11:09:25 PST
On Mon, Jul 01, 2002 at 07:09:32PM +0200, Ralf Hildebrandt wrote:
│ On Mon, Jul 01, 2002 at 10:48:03AM -0500, Phil Howard wrote:
│
│ > Courier IMAP did not meet migration requirements for this server.
│ > Mailbox format is required in this case so that user mail is not
│ > lost. I will be using rsync to syncronize the mailboxes from the
│ > old machine to the new one, then transfer IP addresses and restart.
│
│ Hello? There are conversion scripts for mbox -> maildir
│
│ > The POP server used is vm-pop3d. It is past the time to allocate
│ > the additional time to change the software used or to work out the
│ > strategy to change from mailbox to maildir format. Physical move of
│ > servers takes place 13 July. The migration must take place at the
│ > maintenance window allocated for least user disruption.
│
│ You can even convert on the fly! New delivery to Maildir, then, while
│ new mail is coming in you can still convert the old mailboxes in the
│ background.
It would be nice if there was the luxury of time to go study all that.
But when an old server starts to die, you don't generally think about
"Oh, let's change the file format so we can change the POP server".
Maybe these things could be done, but it would also be prudent to test
things first, too. Given the short amount of time in this case, it is
barely enough to test a machine, OS, MTA, and POP server. Perhaps I
shouldn't have even tried to virtualize the domain at all, but it will
deal with some issues this server has had.
I do prefer the maildir format. But this wasn't the time for it.
--
-----------------------------------------------------------------
│ Phil Howard - KA9WGN │ Dallas │ http://linuxhomepage.com/ │
│ phil-nospam@ipal.net │ Texas, USA │ http://phil.ipal.org/ │
-----------------------------------------------------------------
-
To unsubscribe, send mail to majordomo@postfix.org with content
(not subject): unsubscribe postfix-users
Message 8 in thread
寄件者：Phil Howard (phil-postfix-users@ipal.net)
主旨：Re: virtualizing local users
View this article only
新闻群组：mailing.postfix.users
日期：2002-07-01 11:38:42 PST
On Mon, Jul 01, 2002 at 12:02:14PM +0200, Jean-Pierre Schwickerath wrote:
│ Phil Howard
│
│ > What I mean by that is that the user names will not be listed
│ > in the /etc/passwd file. There is another file which looks
│ > like the first 2 fields of /etc/passwd or /etc/shadow which
│ > contains the list of valid users. The main question is how
│ > might I get Postfix to check this file for known users?
│
│ There is a parameter called local_recipient_maps
│ cut
│ local_recipient_maps = $alias_maps unix:passwd.byname
│ cut
│
│ There you could add your file(s) in any format that postfix
│ understands...
I now have:
local_recipient_maps = hash:/etc/virtual/users
The file named "users" has each LHS with the username by itself and
the RHS with just "OK". But the names are not recognized. Names
that are in /etc/passwd are recognized even though I did not code
unix:passwd.byname on local_recipient_maps at all. It's like it
just doesn't see it at all and still uses the default.
What do I look for next?
--
-----------------------------------------------------------------
│ Phil Howard - KA9WGN │ Dallas │ http://linuxhomepage.com/ │
│ phil-nospam@ipal.net │ Texas, USA │ http://phil.ipal.org/ │
-----------------------------------------------------------------
-
To unsubscribe, send mail to majordomo@postfix.org with content
(not subject): unsubscribe postfix-users
Message 9 in thread
寄件者：Jean-Pierre Schwickerath (lists@schwicky.net)
主旨：Re: virtualizing local users
View this article only
新闻群组：mailing.postfix.users
日期：2002-07-01 12:12:46 PST
Phil Howard
> he file named "users" has each LHS with the username by itself and
> the RHS with just "OK". But the names are not recognized.
I had a look into the VIRTUAL_README file and saw you might want to try
this:
virtual_mailbox_base = /var/mail/vhosts
virtual_mailbox_maps = hash:/etc/postfix/vmailbox
and fill your /etc/postfix/vmailbox file with records like
usernameusernameMailbox
so that mail for username is put into the
"/var/mail/vhosts/usernameMailbox" File
Jean-Pierre
--
Powered by Linux From Scratch - http://schwicky.net
PGP Key ID: 0xEE6F49B4 - AIM/Jabber: Schwicky - ICQ: 4690141
Nothing is impossible... Everything is relative!
-
To unsubscribe, send mail to majordomo@postfix.org with content
(not subject): unsubscribe postfix-users
Message 10 in thread
寄件者：Victor.Duchovni@morganstanley.com (Victor.Duchovni@morganstanley.com)
主旨：Re: virtualizing local users
View this article only
新闻群组：mailing.postfix.users
日期：2002-07-01 12:26:50 PST
On Mon, 1 Jul 2002, Phil Howard wrote:
> local_recipient_maps = hash:/etc/virtual/users
>
> The file named "users" has each LHS with the username by itself and
> the RHS with just "OK". But the names are not recognized. Names
> that are in /etc/passwd are recognized even though I did not code
> unix:passwd.byname on local_recipient_maps at all. It's like it
> just doesn't see it at all and still uses the default.
>
> What do I look for next?
>
logs.
--
Viktor.
-
To unsubscribe, send mail to majordomo@postfix.org with content
(not subject): unsubscribe postfix-users
寄件者：Jean-Pierre Schwickerath (lists@schwicky.net)
主旨：Re: virtualizing local users
View this article only
新闻群组：mailing.postfix.users
日期：2002-07-01 13:05:06 PST
Phil Howard
> This is not going in the right direction. I'm trying to find a way
> to have LESS data files hanging around.
I thought you were looking for a way not to list local users in your
/etc/passwd file but somewhere else so as to trully seperate login users
and mail users.
Jean-Pierre
--
Powered by Linux From Scratch - http://schwicky.net
PGP Key ID: 0xEE6F49B4 - AIM/Jabber: Schwicky - ICQ: 4690141
Nothing is impossible... Everything is relative!
-
To unsubscribe, send mail to majordomo@postfix.org with content
(not subject): unsubscribe postfix-users
Message 12 in thread
寄件者：Phil Howard (phil-postfix-users@ipal.net)
主旨：Re: virtualizing local users
View this article only
新闻群组：mailing.postfix.users
日期：2002-07-01 13:58:15 PST
On Mon, Jul 01, 2002 at 03:25:22PM -0400, Victor.Duchovni@morganstanley.com wrote:
│ On Mon, 1 Jul 2002, Phil Howard wrote:
│
│ > local_recipient_maps = hash:/etc/virtual/users
│ >
│ > The file named "users" has each LHS with the username by itself and
│ > the RHS with just "OK". But the names are not recognized. Names
│ > that are in /etc/passwd are recognized even though I did not code
│ > unix:passwd.byname on local_recipient_maps at all. It's like it
│ > just doesn't see it at all and still uses the default.
│ >
│ > What do I look for next?
│ >
│
│ logs.
It says bounced, unknown user. I would be looking for something like
maybe why it can't read the file? I checked and double checked the
file permissions, and it should be able to read it. Should I make it
owned by the postfix user?
Here's what I get:
=========================================================================
Jul 1 15:55:12 rack4 postfix/smtpd6364: connect from vega.ipal.net[209.102.192.64]
Jul 1 15:55:12 rack4 postfix/smtpd6364: 72025F5: client=vega.ipal.net[209.102.192.64]
Jul 1 15:55:12 rack4 postfix/cleanup6365: 72025F5: message-id=<20020701205511.85C28293@vega.ipal.net>
Jul 1 15:55:12 rack4 postfix/qmgr6337: 72025F5: from=
Jul 1 15:55:12 rack4 postfix/smtpd6364: disconnect from vega.ipal.net[209.102.192.64]
Jul 1 15:55:12 rack4 postfix/local6367: 72025F5: to=
Jul 1 15:55:12 rack4 postfix/cleanup6365: 7E069125: message-id=<20020701205512.7E069125@rack4.intur.net>
Jul 1 15:55:12 rack4 postfix/qmgr6337: 7E069125: from=<>, size=2025, nrcpt=1 (queue active)
Jul 1 15:55:14 rack4 postfix/smtp6369: 7E069125: to=
=========================================================================
--
-----------------------------------------------------------------
│ Phil Howard - KA9WGN │ Dallas │ http://linuxhomepage.com/ │
│ phil-nospam@ipal.net │ Texas, USA │ http://phil.ipal.org/ │
-----------------------------------------------------------------
-
To unsubscribe, send mail to majordomo@postfix.org with content
(not subject): unsubscribe postfix-users
Message 13 in thread
寄件者：Ralf Hildebrandt (Ralf.Hildebrandt@charite.de)
主旨：Re: virtualizing local users
View this article only
新闻群组：mailing.postfix.users
日期：2002-07-01 14:03:52 PST
On Mon, Jul 01, 2002 at 03:56:20PM -0500, Phil Howard wrote:
> Jul 1 15:55:12 rack4 postfix/qmgr6337: 72025F5: from=
> Jul 1 15:55:12 rack4 postfix/smtpd6364: disconnect from vega.ipal.net[209.102.192.64]
> Jul 1 15:55:12 rack4 postfix/local6367: 72025F5: to=
At least we now know that "local" is involved.
Wasn't the whole idea NOT to use "local" but some other LDA, like
"virtual"?
--
Ralf Hildebrandt (Im Auftrag des Referat V A) Ralf.Hildebrandt@charite.de
Charite Campus Virchow-Klinikum Tel. +49 (0)30-450 570-155
Referat V A - Kommunikationsnetze - Fax. +49 (0)30-450 570-916
I wish you'd tell me what kind of systems they're using instead,
because HP can't be doing much worse than Sun "would you like the
compiler or internet options with that" Microsystems, or Silicon "hey
be glad the support-contract number isn't a 1-900" Graphics. Then
there's Digital "It sucks in 64 bits, you can't suck in 64 bits
anywhere else" Equipment Corp (Did we mention it's 64 bits?).
-
To unsubscribe, send mail to majordomo@postfix.org with content
(not subject): unsubscribe postfix-users
Message 14 in thread
寄件者：Phil Howard (phil-postfix-users@ipal.net)
主旨：Re: virtualizing local users
View this article only
新闻群组：mailing.postfix.users
日期：2002-07-01 14:39:00 PST
On Mon, Jul 01, 2002 at 10:03:54PM +0200, Jean-Pierre Schwickerath wrote:
│ Phil Howard
│
│ > This is not going in the right direction. I'm trying to find a way
│ > to have LESS data files hanging around.
│
│ I thought you were looking for a way not to list local users in your
│ /etc/passwd file but somewhere else so as to trully seperate login users
│ and mail users.
There were two, and now three. But I don't see the point in having
a map that translates x -> x for every different user.
--
-----------------------------------------------------------------
│ Phil Howard - KA9WGN │ Dallas │ http://linuxhomepage.com/ │
│ phil-nospam@ipal.net │ Texas, USA │ http://phil.ipal.org/ │
-----------------------------------------------------------------
-
To unsubscribe, send mail to majordomo@postfix.org with content
(not subject): unsubscribe postfix-users
Message 15 in thread
寄件者：Phil Howard (phil-postfix-users@ipal.net)
主旨：Re: virtualizing local users
View this article only
新闻群组：mailing.postfix.users
日期：2002-07-01 14:40:39 PST
On Mon, Jul 01, 2002 at 10:59:28PM +0200, Ralf Hildebrandt wrote:
│ On Mon, Jul 01, 2002 at 03:56:20PM -0500, Phil Howard wrote:
│
│ > Jul 1 15:55:12 rack4 postfix/qmgr6337: 72025F5: from=
│ > Jul 1 15:55:12 rack4 postfix/smtpd6364: disconnect from vega.ipal.net[209.102.192.64]
│ > Jul 1 15:55:12 rack4 postfix/local6367: 72025F5: to=
│
│ At least we now know that "local" is involved.
│ Wasn't the whole idea NOT to use "local" but some other LDA, like
│ "virtual"?
But it is local. Just not the same user base as the system.
--
-----------------------------------------------------------------
│ Phil Howard - KA9WGN │ Dallas │ http://linuxhomepage.com/ │
│ phil-nospam@ipal.net │ Texas, USA │ http://phil.ipal.org/ │
-----------------------------------------------------------------
-
To unsubscribe, send mail to majordomo@postfix.org with content
(not subject): unsubscribe postfix-users
Message 16 in thread
寄件者：Victor.Duchovni@morganstanley.com (Victor.Duchovni@morganstanley.com)
主旨：Re: virtualizing local users
View this article only
新闻群组：mailing.postfix.users
日期：2002-07-01 14:45:29 PST
On Mon, 1 Jul 2002, Phil Howard wrote:
> It says bounced, unknown user. I would be looking for something like
> maybe why it can't read the file? I checked and double checked the
> file permissions, and it should be able to read it. Should I make it
> owned by the postfix user?
>
> Here's what I get:
> =========================================================================
> Jul 1 15:55:12 rack4 postfix/smtpd6364: connect from vega.ipal.net[209.102.192.64]
> Jul 1 15:55:12 rack4 postfix/smtpd6364: 72025F5: client=vega.ipal.net[209.102.192.64]
> Jul 1 15:55:12 rack4 postfix/cleanup6365: 72025F5: message-id=<20020701205511.85C28293@vega.ipal.net>
> Jul 1 15:55:12 rack4 postfix/qmgr6337: 72025F5: from=
> Jul 1 15:55:12 rack4 postfix/smtpd6364: disconnect from vega.ipal.net[209.102.192.64]
> Jul 1 15:55:12 rack4 postfix/local6367: 72025F5: to=
> Jul 1 15:55:12 rack4 postfix/cleanup6365: 7E069125: message-id=<20020701205512.7E069125@rack4.intur.net>
> Jul 1 15:55:12 rack4 postfix/qmgr6337: 7E069125: from=<>, size=2025, nrcpt=1 (queue active)
> Jul 1 15:55:14 rack4 postfix/smtp6369: 7E069125: to=
> =========================================================================
>
This is finally useful. Please examine your logs closely. The bounce is
from the "local" delivery agent, not from "smtpd". So local_recipient_maps
is not the issue. Delivery for the domain is being passed to the "local"
delivery agent. The local delivery agent does deliver to non-shell users
(what uid should the mailbox belong to? what user should pipes in .forward
files run as? ...)
What delivery agent are you expecting to use?
Your choices are:
virtual- Postfix virtual delivery agent
maildrop - Courier delivery agent
deliver - Cyrus delivery agent
whatever comes with your POP server?
...
Each of these delivery agents has its own database format. Set up the
appropriate transport table entries (or redefine local_transport) and
configure any necessary delivery agent tables.
As I said the first time, the answer depends on your choice of delivery
agent which ties into your choice of POP/IMAP server.
--
Viktor.
-
To unsubscribe, send mail to majordomo@postfix.org with content
(not subject): unsubscribe postfix-users
Message 17 in thread
寄件者：Victor.Duchovni@morganstanley.com (Victor.Duchovni@morganstanley.com)
主旨：Re: virtualizing local users
View this article only
新闻群组：mailing.postfix.users
日期：2002-07-01 14:48:19 PST
On Mon, 1 Jul 2002, Phil Howard wrote:
> On Mon, Jul 01, 2002 at 10:59:28PM +0200, Ralf Hildebrandt wrote:
>
> │ On Mon, Jul 01, 2002 at 03:56:20PM -0500, Phil Howard wrote:
> │
> │ > Jul 1 15:55:12 rack4 postfix/qmgr6337: 72025F5: from=
> │ > Jul 1 15:55:12 rack4 postfix/smtpd6364: disconnect from vega.ipal.net[209.102.192.64]
> │ > Jul 1 15:55:12 rack4 postfix/local6367: 72025F5: to=
> │
> │ At least we now know that "local" is involved.
> │ Wasn't the whole idea NOT to use "local" but some other LDA, like
> │ "virtual"?
>
> But it is local. Just not the same user base as the system.
>
Sadly you misundertood the "local" delivery agent. Its input namespace
is the set of local shell accounts + any aliases. Its output namespace
(mailboxes it delivers to) is just the space of shell acounts.
--
Viktor.
-
To unsubscribe, send mail to majordomo@postfix.org with content
(not subject): unsubscribe postfix-users
Message 18 in thread
寄件者：Phil Howard (phil-postfix-users@ipal.net)
主旨：Re: virtualizing local users
View this article only
新闻群组：mailing.postfix.users
日期：2002-07-01 14:54:51 PST
On Mon, Jul 01, 2002 at 10:59:28PM +0200, Ralf Hildebrandt wrote:
│ On Mon, Jul 01, 2002 at 03:56:20PM -0500, Phil Howard wrote:
│
│ > Jul 1 15:55:12 rack4 postfix/qmgr6337: 72025F5: from=
│ > Jul 1 15:55:12 rack4 postfix/smtpd6364: disconnect from vega.ipal.net[209.102.192.64]
│ > Jul 1 15:55:12 rack4 postfix/local6367: 72025F5: to=
│
│ At least we now know that "local" is involved.
│ Wasn't the whole idea NOT to use "local" but some other LDA, like
│ "virtual"?
Obviously I've picked the wrong concept here. If "local" and
"virtual" can't be the same thing, I'm not sure how to get this
done.
When a very basic simple one-host setup is done, I don't have to
use a map which translates a username to path (file for mailbox or
directory for maildir). So why should it be different for virtual?
Oh I know, because of the traditional kludge to fake virtual that
originated with sendmail, where user@domain was translated to a
local system user. But I don't need that. If local can deliver
to /var/spool/mail/${user} why not allow something to deliver to
/var/spool/vmail/${domain}/${user}. This would be such a simple
concept and not need a map file (but certain you can have one that
can override the default for the cases where you want something
different done).
It looks like what I need to do is abandon virtualizing on this
server, and address how I want to set up the NEXT project. It seems
figuring things out is going to take more time than I thought. The
next project does have more time, though not enough time to develop
a new MTA for it. Now would it be better for me to try to figure it
out and have others fix my mistakes, or explain it up front and let
you tell me what concept in Postfix matches up to it (if any)?
--
-----------------------------------------------------------------
│ Phil Howard - KA9WGN │ Dallas │ http://linuxhomepage.com/ │
│ phil-nospam@ipal.net │ Texas, USA │ http://phil.ipal.org/ │
-----------------------------------------------------------------
-
To unsubscribe, send mail to majordomo@postfix.org with content
(not subject): unsubscribe postfix-users
Message 19 in thread
寄件者：Phil Howard (phil-postfix-users@ipal.net)
主旨：Re: virtualizing local users
View this article only
新闻群组：mailing.postfix.users
日期：2002-07-01 15:05:16 PST
On Mon, Jul 01, 2002 at 05:46:42PM -0400, Victor.Duchovni@morganstanley.com wrote:
│ On Mon, 1 Jul 2002, Phil Howard wrote:
│
│ > On Mon, Jul 01, 2002 at 10:59:28PM +0200, Ralf Hildebrandt wrote:
│ >
│ > │ On Mon, Jul 01, 2002 at 03:56:20PM -0500, Phil Howard wrote:
│ > │
│ > │ > Jul 1 15:55:12 rack4 postfix/qmgr6337: 72025F5: from=
│ > │ > Jul 1 15:55:12 rack4 postfix/smtpd6364: disconnect from vega.ipal.net[209.102.192.64]
│ > │ > Jul 1 15:55:12 rack4 postfix/local6367: 72025F5: to=
│ > │
│ > │ At least we now know that "local" is involved.
│ > │ Wasn't the whole idea NOT to use "local" but some other LDA, like
│ > │ "virtual"?
│ >
│ > But it is local. Just not the same user base as the system.
│ >
│
│ Sadly you misundertood the "local" delivery agent. Its input namespace
│ is the set of local shell accounts + any aliases. Its output namespace
│ (mailboxes it delivers to) is just the space of shell acounts.
So what should I be using to deliver the mail into mailboxes on the local
machine, where the mailbox is a name exactly the same as the user part
of the address, appended to a path (/var/spool/mail), where every mailbox
is owned by one specific system user. The mailed-to users do not have
home directories, so there is no .forward file. If there are any addresses
that need to be forwarded to some other, a map for that is good, but if
the address is not found in the map, it should just be delivered normally.
--
-----------------------------------------------------------------
│ Phil Howard - KA9WGN │ Dallas │ http://linuxhomepage.com/ │
│ phil-nospam@ipal.net │ Texas, USA │ http://phil.ipal.org/ │
-----------------------------------------------------------------
-
To unsubscribe, send mail to majordomo@postfix.org with content
(not subject): unsubscribe postfix-users
Message 20 in thread
寄件者：Victor.Duchovni@morganstanley.com (Victor.Duchovni@morganstanley.com)
主旨：Re: virtualizing local users
View this article only
新闻群组：mailing.postfix.users
日期：2002-07-01 15:05:29 PST
On Mon, 1 Jul 2002, Phil Howard wrote:
> Obviously I've picked the wrong concept here. If "local" and
> "virtual" can't be the same thing, I'm not sure how to get this
> done.
This is the first step, knowing what you don't know.
>
> When a very basic simple one-host setup is done, I don't have to
> use a map which translates a username to path (file for mailbox or
> directory for maildir). So why should it be different for virtual?
> Oh I know, because of the traditional kludge to fake virtual that
> originated with sendmail, where user@domain was translated to a
> local system user. But I don't need that. If local can deliver
> to /var/spool/mail/${user} why not allow something to deliver to
> /var/spool/vmail/${domain}/${user}. This would be such a simple
> concept and not need a map file (but certain you can have one that
> can override the default for the cases where you want something
> different done).
>
The "virtual" delivery agent is not very sophisticated. I believe that
this is in part because most sites don't use it! It is not very useful by
itself, and a complete POP/IMAP product (such as Cyrus or Courier) comes
with a dedicated delivery agent.
> It looks like what I need to do is abandon virtualizing on this
> server, and address how I want to set up the NEXT project. It seems
> figuring things out is going to take more time than I thought. The
> next project does have more time, though not enough time to develop
> a new MTA for it. Now would it be better for me to try to figure it
> out and have others fix my mistakes, or explain it up front and let
> you tell me what concept in Postfix matches up to it (if any)?
>
Yes, for now it may be simplest to deliver using "local" to shell accounts
listed in /etc/passwd. Some of us think that migrating to Courier is not
too difficult given the availability of decent HOWTO documents, but a
conservative step-by-step approach is wise. First get your server working,
then teach it new tricks.
--
Viktor.
-
To unsubscribe, send mail to majordomo@postfix.org with content
(not subject): unsubscribe postfix-users
寄件者：Victor.Duchovni@morganstanley.com (Victor.Duchovni@morganstanley.com)
主旨：Re: virtualizing local users
View this article only
新闻群组：mailing.postfix.users
日期：2002-07-01 15:21:44 PST
On Mon, 1 Jul 2002, Phil Howard wrote:
> So what should I be using to deliver the mail into mailboxes on the local
> machine, where the mailbox is a name exactly the same as the user part
> of the address, appended to a path (/var/spool/mail), where every mailbox
> is owned by one specific system user. The mailed-to users do not have
> home directories, so there is no .forward file. If there are any addresses
> that need to be forwarded to some other, a map for that is good, but if
> the address is not found in the map, it should just be delivered normally.
>
Use a better (than "virtual") delivery agent which comes with your virtual
user POP/IMAP server.
The simplest configuration with the bundled VDA is to use the "static" map
for virtual_uid_maps and virtual_gid_maps. Then use a suitable map type
(perhaps MySQL or "userdb" to share data with the POP/IMAP server) to
manage just virtual_mailbox_maps.
There is no support for a "/some/path/%d/%u" format template in the
virtual delivery agent which ships with Postfix.
--
Viktor.
寄件者：Phil Howard (phil-postfix-users@ipal.net)
主旨：Re: virtualizing local users
View: Complete Thread (36 articles)
Original Format
新闻群组：mailing.postfix.users
日期：2002-07-01 15:22:16 PST
On Mon, Jul 01, 2002 at 06:03:42PM -0400, Victor.Duchovni@morganstanley.com wrote:
│ On Mon, 1 Jul 2002, Phil Howard wrote:
│
│ > Obviously I've picked the wrong concept here. If "local" and
│ > "virtual" can't be the same thing, I'm not sure how to get this
│ > done.
│
│ This is the first step, knowing what you don't know.
│
│ >
│ > When a very basic simple one-host setup is done, I don't have to
│ > use a map which translates a username to path (file for mailbox or
│ > directory for maildir). So why should it be different for virtual?
│ > Oh I know, because of the traditional kludge to fake virtual that
│ > originated with sendmail, where user@domain was translated to a
│ > local system user. But I don't need that. If local can deliver
│ > to /var/spool/mail/${user} why not allow something to deliver to
│ > /var/spool/vmail/${domain}/${user}. This would be such a simple
│ > concept and not need a map file (but certain you can have one that
│ > can override the default for the cases where you want something
│ > different done).
│ >
│
│ The "virtual" delivery agent is not very sophisticated. I believe that
│ this is in part because most sites don't use it! It is not very useful by
│ itself, and a complete POP/IMAP product (such as Cyrus or Courier) comes
│ with a dedicated delivery agent.
How does a delivery agent "hook up" to Postfix? Is LMTP the only way,
or can it be a dynamic library?
What about local_recipient_maps and/or making smtpd reject unknown users
when the delivery is keeping separate users spaces for each domain (which
apparently is what is called virtual).
│ > It looks like what I need to do is abandon virtualizing on this
│ > server, and address how I want to set up the NEXT project. It seems
│ > figuring things out is going to take more time than I thought. The
│ > next project does have more time, though not enough time to develop
│ > a new MTA for it. Now would it be better for me to try to figure it
│ > out and have others fix my mistakes, or explain it up front and let
│ > you tell me what concept in Postfix matches up to it (if any)?
│ >
│
│ Yes, for now it may be simplest to deliver using "local" to shell accounts
│ listed in /etc/passwd. Some of us think that migrating to Courier is not
│ too difficult given the availability of decent HOWTO documents, but a
│ conservative step-by-step approach is wise. First get your server working,
│ then teach it new tricks.
Once the server is working, it won't be changed for a while.
But, there is the next project. This is what it needs:
1. Support for many domains.
2. Support for separate user name space for each domain (except where
a domain is linked/aliased to another, then they share the user
name space).
3. NOT one giant map with every user@domain. That will be too big to
maintain. A separate map for each domain is best.
4. Mail delivered to ${prefix}/${domain}/${user}/ in maildir format
5. If ${prefix}/${domain}/${user}/ exists, the address is valid for
delivery. If it doesn't, then the address is non-existant.
6. If ${prefix}/${domain}/${user}/.forward exists, obey it.
7. One single system user owns everything from ${prefix}/ on down.
To carry out some of these things, the thought I had was to write a new
map type handler which does a lookup for an existing directory or file.
The "name" for the map will actually be a complex specification that
tells the path, what to return if the file object does not exist, what
to return if it is a directory, and what to return if it is a file with
a special code to indicate that the file should be read and its content
be returned. Then Postfix can think it is a map, but it's just a directory.
Users will be added/deleted/changed by web CGI programs. Rebuilding a
whole map is a bad idea in this case.
--
-----------------------------------------------------------------
│ Phil Howard - KA9WGN │ Dallas │ http://linuxhomepage.com/ │
│ phil-nospam@ipal.net │ Texas, USA │ http://phil.ipal.org/ │
-----------------------------------------------------------------
寄件者：Victor.Duchovni@morganstanley.com (Victor.Duchovni@morganstanley.com)
主旨：Re: virtualizing local users
View: Complete Thread (36 articles)
Original Format
新闻群组：mailing.postfix.users
日期：2002-07-01 18:43:47 PST
On Mon, 1 Jul 2002, Phil Howard wrote:
> How does a delivery agent "hook up" to Postfix? Is LMTP the only way,
> or can it be a dynamic library?
Either LMTP, or a UNIX command line invocation via "pipe" (the
delivery agents that come with Postfix use the internal queue manager <->
delivery agent IPC, they also directly touch the queue file and bounce
logs). The internal interfaces are not documented and are subject to
change, so at this time only "LMTP" and "pipe" are suitable for
"third-party" delivery agents. In practice this is good enough.
>
> What about local_recipient_maps and/or making smtpd reject unknown users
> when the delivery is keeping separate users spaces for each domain (which
> apparently is what is called virtual).
>
If the VDA recipient tables are available via a suitable Postfix map type,
"smtpd" can validate recipients against these tables. The "userdb" format
used by Courier is available (via a patch) as a Postfix map type. One day
there may be an interface for dynamically loading new map types.
> 1. Support for many domains.
>
> 2. Support for separate user name space for each domain (except where
> a domain is linked/aliased to another, then they share the user
> name space).
Any VDA will do 1 & 2.
>
> 3. NOT one giant map with every user@domain. That will be too big to
> maintain. A separate map for each domain is best.
Your intuition is wrong here. Searching multiple (possible hundreds) of
maps for a single key scales poorly. In fact one single map is best.
Making a sensible front-end tool that provides per domain views is a
separate problem (can use say "select ... where ..."). A good IMAP server
comes with virtual user and virtual domain administration tools.
>
> 4. Mail delivered to ${prefix}/${domain}/${user}/ in maildir format
Any VDA will do this, but for some the mailboxes for each user may need to
be explicitly mapped to the appropriate path.
>
> 5. If ${prefix}/${domain}/${user}/ exists, the address is valid for
> delivery. If it doesn't, then the address is non-existant.
This cannot be done from "smtpd", it needs to be able to run "chrooted"
and with low privileges. You need a map to validate the users. Surely
users need a password entry to read their mail, so a path entry for their
mailbox is not asking too much. Any multi-column database (MySQL, LDAP,
userdb, ...) will be able to all the necessary data in one row per user.
>
> 6. If ${prefix}/${domain}/${user}/.forward exists, obey it.
This is available with some VDAs (but not the "virtual" delivery
agent bundled with Postfix).
>
> 7. One single system user owns everything from ${prefix}/ on down.
$virtual_uid_maps = static:12345
$virtual_gid_maps = static:12345
> To carry out some of these things, the thought I had was to write a new
> map type handler which does a lookup for an existing directory or file.
> The "name" for the map will actually be a complex specification that
> tells the path, what to return if the file object does not exist, what
> to return if it is a directory, and what to return if it is a file with
> a special code to indicate that the file should be read and its content
> be returned. Then Postfix can think it is a map, but it's just a directory.
>
This will not work because of "chroot" and security issues. Just drive
both the VDA and Postfix from a shared "userdb", MySQL or LDAP database.
> Users will be added/deleted/changed by web CGI programs. Rebuilding a
> whole map is a bad idea in this case.
>
Berkeley DB maps support incremental insertion, but this is not
crashproof unless you turn on logging and transactions, doing Postfix
compatible locking may also be an issue, also Postfix may need to open
the map with logging and transactions turned on in order to do crash
recovery, this is not the case with the current Berkeley DB <-> Postfix
glue. For live incremental updates your best may be LDAP, MySQL or (via a
patch) PostSQL.
Note on the other hand that even large Berkeley DB maps can be rebuilt
from scratch very quickly (5s on my machine for ~16MB on disk .db file
with ~140000 records). If you build a copy of the map, open it, sync it,
and rename it, you have a light weight crash-proof (atomic) Berkeley DB
rebuild. If this happens infrequently (less than once every 5 minutes) you
will be better off with Berkeley DB than with any complex/slow DBMS than
supports incremental insertion of records.
Does anyone else have good experiences with other dynamic map types that
support efficient frequent incremental rebuilds and are robust against
crashes?
[ Michael Tokarev: how quickly does your tinycdb build large maps ~100000
rows ~16 MB image ].
--
Viktor.
-
To unsubscribe, send mail to majordomo@postfix.org with content
(not subject): unsubscribe postfix-users
寄件者：Ralf Hildebrandt (Ralf.Hildebrandt@charite.de)
主旨：Re: virtualizing local users
View this article only
新闻群组：mailing.postfix.users
日期：2002-07-01 22:52:06 PST
On Mon, Jul 01, 2002 at 05:21:15PM -0500, Phil Howard wrote:
> How does a delivery agent "hook up" to Postfix? Is LMTP the only way,
> or can it be a dynamic library?
You can always call it using "mailbox_command" or "local_transport" or
"mailbox_transport".
> What about local_recipient_maps and/or making smtpd reject unknown users
> when the delivery is keeping separate users spaces for each domain (which
> apparently is what is called virtual).
These are just syntactical sugar.
Merely maps that rell smtpd to reject the mail right away.
> 1. Support for many domains.
>
> 2. Support for separate user name space for each domain (except where
> a domain is linked/aliased to another, then they share the user
> name space).
virtual_maps
> 3. NOT one giant map with every user@domain. That will be too big to
> maintain. A separate map for each domain is best.
That's what you think.
It performs best wiuth one map (which can be constructed/merged from
many small map). OTOH, sql or LDAP come to mind.
> 4. Mail delivered to ${prefix}/${domain}/${user}/ in maildir format
procmail can do that. Dunno about "virtual"
> 5. If ${prefix}/${domain}/${user}/ exists, the address is valid for
> delivery. If it doesn't, then the address is non-existant.
Interesting. That would be using the FS as map.
In that case, smtpd couldn't run chrooted.
> 6. If ${prefix}/${domain}/${user}/.forward exists, obey it.
So no "virtual" LDA then.
> 7. One single system user owns everything from ${prefix}/ on down.
OK.
> To carry out some of these things, the thought I had was to write a new
> map type handler which does a lookup for an existing directory or file.
It already exists (I thinK!) as a patch.
--
Ralf Hildebrandt (Im Auftrag des Referat V A) Ralf.Hildebrandt@charite.de
Charite Campus Virchow-Klinikum Tel. +49 (0)30-450 570-155
Referat V A - Kommunikationsnetze - Fax. +49 (0)30-450 570-916
Eh? Linux is luserproof? What kind of "proper" set up is that, ripping
out all removable media devices and ethernet, freezing the hard drive
spindle, encasing it in concrete and dropping it off a pier?
-
To unsubscribe, send mail to majordomo@postfix.org with content
(not subject): unsubscribe postfix-users
寄件者：Phil Howard (phil-postfix-users@ipal.net)
主旨：Re: virtualizing local users
View: Complete Thread (36 articles)
Original Format
新闻群组：mailing.postfix.users
日期：2002-07-02 02:06:18 PST
Just to point out first, unlike the current project, the next mail based
project does have more time available to it and doesn't yet have a fixed
schedule. Thus there is time to write some C code and do things like
adding a new map type.
On Mon, Jul 01, 2002 at 09:42:53PM -0400, Victor.Duchovni@morganstanley.com wrote:
│ On Mon, 1 Jul 2002, Phil Howard wrote:
│
│ > 3. NOT one giant map with every user@domain. That will be too big to
│ > maintain. A separate map for each domain is best.
│
│ Your intuition is wrong here. Searching multiple (possible hundreds) of
│ maps for a single key scales poorly. In fact one single map is best.
│ Making a sensible front-end tool that provides per domain views is a
│ separate problem (can use say "select ... where ..."). A good IMAP server
│ comes with virtual user and virtual domain administration tools.
If each map is for a specific domain, why search them all? Why not just
search the one map for that domain, using only the username part as the
key, and stop there?
I want to be able to be able to link maps together so that example.com
and example.net are the same namespace, without there being duplicates
for every user in a big single giant map. But the biggest reason is
load balancing. I want to be able to easily and quickly move a group of
domains from one server to another one.
As a result of this, these maps would have only the user part as the
key, so even if you wanted to search all the maps for some user@domain,
you can't actually carry out a reasonable search. If you searching
for foo@example.com, and there exists an entry for "foo" in the map
for example.org, is that considered a match?
As for the administration tools in an IMAP server, this project won't
be able to use them, as the administration will be done by a separate
project. Manual administration is intended to be very rare, so any
manual oriented administration tools are not applicable.
│ > 4. Mail delivered to ${prefix}/${domain}/${user}/ in maildir format
│
│ Any VDA will do this, but for some the mailboxes for each user may need to
│ be explicitly mapped to the appropriate path.
If the regular expression map can do substitution, then maybe it can
be used to construct the path from the user@domain key. Otherwise a
new map type could easily solve the problem. One that did read a map
file but applied the substitution if the map file didn't have a match
would make sense to me. Then you could still have a mapping for the
rare cases of an unusual relationship, without having to build and
keep a giant identity map.
│ > 5. If ${prefix}/${domain}/${user}/ exists, the address is valid for
│ > delivery. If it doesn't, then the address is non-existant.
│
│ This cannot be done from "smtpd", it needs to be able to run "chrooted"
│ and with low privileges. You need a map to validate the users. Surely
│ users need a password entry to read their mail, so a path entry for their
│ mailbox is not asking too much. Any multi-column database (MySQL, LDAP,
│ userdb, ...) will be able to all the necessary data in one row per user.
Obviously it needs to know in some way. Can the specific chroot path be
configured (even if at patch time)? That can be a path where all the maps
are located and it just reads them. In Linux I can also do bind mounting
to put one directory inside another, and that works fine in chroot (I know
because I have a script to do Slackware installs in chroot that uses this
capability to "pre-mount" the target directory into the chroot as "/mnt"
where the Slackware installer installs to, and it works).
│ > 6. If ${prefix}/${domain}/${user}/.forward exists, obey it.
│
│ This is available with some VDAs (but not the "virtual" delivery
│ agent bundled with Postfix).
Is Postfix's VDA considered useful?
│ > 7. One single system user owns everything from ${prefix}/ on down.
│
│ $virtual_uid_maps = static:12345
│ $virtual_gid_maps = static:12345
│
│ > To carry out some of these things, the thought I had was to write a new
│ > map type handler which does a lookup for an existing directory or file.
│ > The "name" for the map will actually be a complex specification that
│ > tells the path, what to return if the file object does not exist, what
│ > to return if it is a directory, and what to return if it is a file with
│ > a special code to indicate that the file should be read and its content
│ > be returned. Then Postfix can think it is a map, but it's just a directory.
│ >
│
│ This will not work because of "chroot" and security issues. Just drive
│ both the VDA and Postfix from a shared "userdb", MySQL or LDAP database.
What directory will it be chroot-ed into?
│ > Users will be added/deleted/changed by web CGI programs. Rebuilding a
│ > whole map is a bad idea in this case.
│ >
│
│ Berkeley DB maps support incremental insertion, but this is not
│ crashproof unless you turn on logging and transactions, doing Postfix
│ compatible locking may also be an issue, also Postfix may need to open
│ the map with logging and transactions turned on in order to do crash
│ recovery, this is not the case with the current Berkeley DB <-> Postfix
│ glue. For live incremental updates your best may be LDAP, MySQL or (via a
│ patch) PostSQL.
│
│ Note on the other hand that even large Berkeley DB maps can be rebuilt
│ from scratch very quickly (5s on my machine for ~16MB on disk .db file
│ with ~140000 records). If you build a copy of the map, open it, sync it,
│ and rename it, you have a light weight crash-proof (atomic) Berkeley DB
│ rebuild. If this happens infrequently (less than once every 5 minutes) you
│ will be better off with Berkeley DB than with any complex/slow DBMS than
│ supports incremental insertion of records.
│
│ Does anyone else have good experiences with other dynamic map types that
│ support efficient frequent incremental rebuilds and are robust against
│ crashes?
│
│ [ Michael Tokarev: how quickly does your tinycdb build large maps ~100000
│ rows ~16 MB image ].
With reiserfs, it is now more practical to use the filesystem as a database.
While it has journaling, the key factors here is that it uses hashing to
manage the directories, and has a feature called tail-packing for efficient
space utilization with small files. It's not quite as fast as ext2, but it
sure beats ext2 when you have 3.7 million files in a single directory (which
I have done).
With Linux bind mounts I can present the tree of such a filesystem based db
into any directory which is to serve as the chroot jail for some process.
The only risk is that bind mounts inherit the rw vs ro property of the whole
filesystem it is picked from. Making sure the process has no uid rights to
write on those directories or files should deal with it.
I haven't ruled out using a db file rebuilt periodically from other data.
But I probably would go for cdb in that case.
--
-----------------------------------------------------------------
│ Phil Howard - KA9WGN │ Dallas │ http://linuxhomepage.com/ │
│ phil-nospam@ipal.net │ Texas, USA │ http://phil.ipal.org/ │
-----------------------------------------------------------------
-
寄件者：Ralf Hildebrandt (Ralf.Hildebrandt@charite.de)
主旨：Re: virtualizing local users
View this article only
新闻群组：mailing.postfix.users
日期：2002-07-02 02:11:06 PST
On Tue, Jul 02, 2002 at 04:03:07AM -0500, Phil Howard wrote:
> If each map is for a specific domain, why search them all? Why not just
> search the one map for that domain, using only the username part as the
> key, and stop there?
Because all data could be in all maps. What you want would probably an
LDAP server (search only in the ou=domain or something like that)
> load balancing. I want to be able to easily and quickly move a group of
> domains from one server to another one.
Simply change what user@domain1.com points to in your map.
> As a result of this, these maps would have only the user part as the
> key, so even if you wanted to search all the maps for some user@domain,
> you can't actually carry out a reasonable search. If you searching
> for foo@example.com, and there exists an entry for "foo" in the map
> for example.org, is that considered a match?
man 5 virtual
(yes)
> │ This will not work because of "chroot" and security issues. Just drive
> │ both the VDA and Postfix from a shared "userdb", MySQL or LDAP database.
>
> What directory will it be chroot-ed into?
$queue_directory
> With reiserfs, it is now more practical to use the filesystem as a database.
If you want to lose all data, yes. I get sick when I see the weekly
patches for various flaws. I don't let Reiser anywhere close to a
system which needs to be reliable.
> I haven't ruled out using a db file rebuilt periodically from other data.
> But I probably would go for cdb in that case.
Yup.
--
Ralf Hildebrandt (Im Auftrag des Referat V A) Ralf.Hildebrandt@charite.de
Charite Campus Virchow-Klinikum Tel. +49 (0)30-450 570-155
Referat V A - Kommunikationsnetze - Fax. +49 (0)30-450 570-916
"Internet is so big, so powerful and pointless that for some people it
is a complete substitute for life."-Andrew Brown
-
To unsubscribe, send mail to majordomo@postfix.org with content
(not subject): unsubscribe postfix-users
Message 27 in thread
寄件者：Phil Howard (phil-postfix-users@ipal.net)
主旨：Re: virtualizing local users
View this article only
新闻群组：mailing.postfix.users
日期：2002-07-02 02:17:49 PST
On Tue, Jul 02, 2002 at 07:50:30AM +0200, Ralf Hildebrandt wrote:
│ > 3. NOT one giant map with every user@domain. That will be too big to
│ > maintain. A separate map for each domain is best.
│
│ That's what you think.
│ It performs best wiuth one map (which can be constructed/merged from
│ many small map). OTOH, sql or LDAP come to mind.
Accessing one large map is faster than accessing one smaller map? If
that's true, then I'll add a million dummy entries to make it faster.
Actually, the performance of accessing the map isn't the reason I want
to split by domain. It's a maintenance issue. Suppose there are 1000
domains with an average of 500 users each. The program that generates
the maps when there is a change would be exposed to 1000 times more
updates, and have 1000 times more work to do, each time a change is
made. That's O(N^2) for N = number of domains. Then there is the
issue of maintenance permission per domain. I need to leave open the
possibility that different domains will be managed by different user
access permissions. That might become part of the design.
│ > 5. If ${prefix}/${domain}/${user}/ exists, the address is valid for
│ > delivery. If it doesn't, then the address is non-existant.
│
│ Interesting. That would be using the FS as map.
│ In that case, smtpd couldn't run chrooted.
Why not? I think it can be. Linux's bind mounts work fine across
chroot.
│ > To carry out some of these things, the thought I had was to write a new
│ > map type handler which does a lookup for an existing directory or file.
│
│ It already exists (I thinK!) as a patch.
I'll look around for it.
--
-----------------------------------------------------------------
│ Phil Howard - KA9WGN │ Dallas │ http://linuxhomepage.com/ │
│ phil-nospam@ipal.net │ Texas, USA │ http://phil.ipal.org/ │
-----------------------------------------------------------------
-
To unsubscribe, send mail to majordomo@postfix.org with content
(not subject): unsubscribe postfix-users
Message 28 in thread
寄件者：Phil Howard (phil-postfix-users@ipal.net)
主旨：Re: virtualizing local users
View this article only
新闻群组：mailing.postfix.users
日期：2002-07-02 02:24:39 PST
On Tue, Jul 02, 2002 at 11:08:57AM +0200, Ralf Hildebrandt wrote:
│ On Tue, Jul 02, 2002 at 04:03:07AM -0500, Phil Howard wrote:
│
│ > If each map is for a specific domain, why search them all? Why not just
│ > search the one map for that domain, using only the username part as the
│ > key, and stop there?
│
│ Because all data could be in all maps. What you want would probably an
│ LDAP server (search only in the ou=domain or something like that)
I have no plans to add an LDAP server, nor an SQL server.
│ > load balancing. I want to be able to easily and quickly move a group of
│ > domains from one server to another one.
│
│ Simply change what user@domain1.com points to in your map.
How will I transfer the file to the new machine? I want it to be a
separate file for a different domain
│ > As a result of this, these maps would have only the user part as the
│ > key, so even if you wanted to search all the maps for some user@domain,
│ > you can't actually carry out a reasonable search. If you searching
│ > for foo@example.com, and there exists an entry for "foo" in the map
│ > for example.org, is that considered a match?
│
│ man 5 virtual
│ (yes)
Is there a Postfix man page that lists all the other Postfix man pages?
│ > │ This will not work because of "chroot" and security issues. Just drive
│ > │ both the VDA and Postfix from a shared "userdb", MySQL or LDAP database.
│ >
│ > What directory will it be chroot-ed into?
│
│ $queue_directory
│
│ > With reiserfs, it is now more practical to use the filesystem as a database.
│
│ If you want to lose all data, yes. I get sick when I see the weekly
│ patches for various flaws. I don't let Reiser anywhere close to a
│ system which needs to be reliable.
Your experience and my experience aren't the same, apparently. Now I've
only been using it for half a year, but it has been rock solid reliable
and I have not lost a single thing. I have lost data occaisionally with
ext2.
--
-----------------------------------------------------------------
│ Phil Howard - KA9WGN │ Dallas │ http://linuxhomepage.com/ │
│ phil-nospam@ipal.net │ Texas, USA │ http://phil.ipal.org/ │
-----------------------------------------------------------------
-
To unsubscribe, send mail to majordomo@postfix.org with content
(not subject): unsubscribe postfix-users
Message 29 in thread
寄件者：Ralf Hildebrandt (Ralf.Hildebrandt@charite.de)
主旨：Re: virtualizing local users
View this article only
新闻群组：mailing.postfix.users
日期：2002-07-02 02:29:13 PST
On Tue, Jul 02, 2002 at 04:15:48AM -0500, Phil Howard wrote:
> Accessing one large map is faster than accessing one smaller map? If
> that's true, then I'll add a million dummy entries to make it faster.
I was not talking about adding garbage to the map.
> Actually, the performance of accessing the map isn't the reason I want
> to split by domain. It's a maintenance issue. Suppose there are 1000
> domains with an average of 500 users each. The program that generates
> the maps when there is a change would be exposed to 1000 times more
> updates, and have 1000 times more work to do, each time a change is
> made. That's O(N^2) for N = number of domains. Then there is the
> issue of maintenance permission per domain. I need to leave open the
> possibility that different domains will be managed by different user
> access permissions. That might become part of the design.
Valid point. But why not use a map instead that does NOT require
rebuilding (like LDAP!)
Looking up something in one btree:
O(n) = ln(n) (well, sort of)
Looking up something in x btrees of size (n/x):
O(n) = x*ln(n/x)
which is bigger than ln(n)
--
Ralf Hildebrandt (Im Auftrag des Referat V A) Ralf.Hildebrandt@charite.de
Charite Campus Virchow-Klinikum Tel. +49 (0)30-450 570-155
Referat V A - Kommunikationsnetze - Fax. +49 (0)30-450 570-916
"Real programmers can write assembly code in any language."
-- Larry Wall
-
To unsubscribe, send mail to majordomo@postfix.org with content
(not subject): unsubscribe postfix-users
Message 30 in thread
寄件者：Ralf Hildebrandt (Ralf.Hildebrandt@charite.de)
主旨：Re: virtualizing local users
View this article only
新闻群组：mailing.postfix.users
日期：2002-07-02 03:13:30 PST
On Tue, Jul 02, 2002 at 04:23:27AM -0500, Phil Howard wrote:
> │ Because all data could be in all maps. What you want would probably an
> │ LDAP server (search only in the ou=domain or something like that)
>
> I have no plans to add an LDAP server, nor an SQL server.
But it would be exactly what your requirements dictate.
> Is there a Postfix man page that lists all the other Postfix man pages?
Unfortunately no. maybe "man postfix" should do that!
> Your experience and my experience aren't the same, apparently. Now I've
> only been using it for half a year, but it has been rock solid reliable
> and I have not lost a single thing. I have lost data occaisionally with
> ext2.
We've been using ext3 for over a year now on our main server
(hauptpostamt.charite.de). During that time I've been following
ReiserFS development closely -- I think ReiserFS's complexity (too
much code) is just too high.
Although reliability has risen in the past (a recent test in the
German c't mag showed that ReiserFS, ext3 and XFS are probably the
most reliable choices).
--
Ralf Hildebrandt (Im Auftrag des Referat V A) Ralf.Hildebrandt@charite.de
Charite Campus Virchow-Klinikum Tel. +49 (0)30-450 570-155
Referat V A - Kommunikationsnetze - Fax. +49 (0)30-450 570-916
"The danger from computers is not that they will eventually get as smart
as men, but we will meanwhile agree to meet them halfway."-Bernard Avishai
-
To unsubscribe, send mail to majordomo@postfix.org with content
(not subject): unsubscribe postfix-users
Message 31 in thread
寄件者：Phil Howard (phil-postfix-users@ipal.net)
主旨：Re: virtualizing local users
View this article only
新闻群组：mailing.postfix.users
日期：2002-07-02 03:15:54 PST
On Tue, Jul 02, 2002 at 11:26:49AM +0200, Ralf Hildebrandt wrote:
│ On Tue, Jul 02, 2002 at 04:15:48AM -0500, Phil Howard wrote:
│
│ > Accessing one large map is faster than accessing one smaller map? If
│ > that's true, then I'll add a million dummy entries to make it faster.
│
│ I was not talking about adding garbage to the map.
│
│ > Actually, the performance of accessing the map isn't the reason I want
│ > to split by domain. It's a maintenance issue. Suppose there are 1000
│ > domains with an average of 500 users each. The program that generates
│ > the maps when there is a change would be exposed to 1000 times more
│ > updates, and have 1000 times more work to do, each time a change is
│ > made. That's O(N^2) for N = number of domains. Then there is the
│ > issue of maintenance permission per domain. I need to leave open the
│ > possibility that different domains will be managed by different user
│ > access permissions. That might become part of the design.
│
│ Valid point. But why not use a map instead that does NOT require
│ rebuilding (like LDAP!)
I don't want to add the administrative complexity of having yet another
server with all of its permissions and access schemes to make sure that
different administrators can only administer the subset of data they are
allowed to. While I am certain LDAP implementations can do that, it is
my experience that these things add a new complex layer of management
control to achieve it. I'm trying to keep it simple.
│ Looking up something in one btree:
│ O(n) = ln(n) (well, sort of)
│
│ Looking up something in x btrees of size (n/x):
│
│ O(n) = x*ln(n/x)
│ which is bigger than ln(n)
What is x? I think you are making the same mistake Victor did and are
trying to sequentially search each map separately. Such a search can't
even work since a map per-domain should have only the user part as the
key. So looking for foo@example.com by searching for foo in example.com
would be right, but what would you search for when searching the map for
example.org? foo? You might find foo in example.org, but it's the wrong
foo. And what if you are searching for foo@example.org but found it first
in the map for example.com?
Just search ONE map, the one for the appropriate domain. Then we're
doing 1*ln(n/x) ... times whatever it takes for the filesystem to
find that map ... which makes it ln(n/x)+ln(x) ... or ln(max(n/x,x))
as far as big-O is concerned.
Then you can also symlink one domain to another at the maps. With the
user part by itself as the key, this would work.
--
-----------------------------------------------------------------
│ Phil Howard - KA9WGN │ Dallas │ http://linuxhomepage.com/ │
│ phil-nospam@ipal.net │ Texas, USA │ http://phil.ipal.org/ │
-----------------------------------------------------------------
-
To unsubscribe, send mail to majordomo@postfix.org with content
(not subject): unsubscribe postfix-users
Message 32 in thread
寄件者：Ralf Hildebrandt (Ralf.Hildebrandt@charite.de)
主旨：Re: virtualizing local users
View this article only
新闻群组：mailing.postfix.users
日期：2002-07-02 03:50:03 PST
On Tue, Jul 02, 2002 at 05:14:36AM -0500, Phil Howard wrote:
> │ Looking up something in one btree:
> │ O(n) = ln(n) (well, sort of)
> │
> │ Looking up something in x btrees of size (n/x):
> │
> │ O(n) = x*ln(n/x)
> │ which is bigger than ln(n)
>
> What is x? I think you are making the same mistake Victor did and are
x << n
(x is the number of maps)
> trying to sequentially search each map separately. Such a search can't
> even work since a map per-domain should have only the user part as the
> key.
All maps are searched. So if you key is in the last map, it has to
search each maps until a match is found.
> Just search ONE map, the one for the appropriate domain. Then we're
Postfix doesn't do that (yet)
--
Ralf Hildebrandt (Im Auftrag des Referat V A) Ralf.Hildebrandt@charite.de
Charite Campus Virchow-Klinikum Tel. +49 (0)30-450 570-155
Referat V A - Kommunikationsnetze - Fax. +49 (0)30-450 570-916
Why you can't find your system administrators:
They've gone to find some more coffee. Sysadmin has left the building! -- Matt Haswell ccdmlh@zippy.dct.ac.uk
-
To unsubscribe, send mail to majordomo@postfix.org with content
(not subject): unsubscribe postfix-users
Message 33 in thread
寄件者：Phil Howard (phil-postfix-users@ipal.net)
主旨：Re: virtualizing local users
View this article only
新闻群组：mailing.postfix.users
日期：2002-07-02 05:49:44 PST
On Tue, Jul 02, 2002 at 11:30:29AM +0200, Ralf Hildebrandt wrote:
│ On Tue, Jul 02, 2002 at 04:23:27AM -0500, Phil Howard wrote:
│
│ > │ Because all data could be in all maps. What you want would probably an
│ > │ LDAP server (search only in the ou=domain or something like that)
│ >
│ > I have no plans to add an LDAP server, nor an SQL server.
│
│ But it would be exactly what your requirements dictate.
I didn't list all the requirements. They aren't all written down
in one place, and they didn't all seem appropriate here, either.
Some of the requirements do include having a minimum of things to
administer, a minimum of things to debug, and a minimum of things
exposed to failing.
│ > Is there a Postfix man page that lists all the other Postfix man pages?
│
│ Unfortunately no. maybe "man postfix" should do that!
It would be a help, even if it just gives subtopics when then have
the list of other man pages.
│ > Your experience and my experience aren't the same, apparently. Now I've
│ > only been using it for half a year, but it has been rock solid reliable
│ > and I have not lost a single thing. I have lost data occaisionally with
│ > ext2.
│
│ We've been using ext3 for over a year now on our main server
│ (hauptpostamt.charite.de). During that time I've been following
│ ReiserFS development closely -- I think ReiserFS's complexity (too
│ much code) is just too high.
│
│ Although reliability has risen in the past (a recent test in the
│ German c't mag showed that ReiserFS, ext3 and XFS are probably the
│ most reliable choices).
How well does ext3 deal with 3000000 files in a single directory?
How well does ext3 deal with 3000000 files that have no more than
16 bytes each? These capabilities in ReiserFS have helped in a few
projects already. I am at 3.6 in Linux 2.4.18 and it has been good
to me so far. It's even dealt with messy issues like files that are
the executeable images of running processes being deleted, rebooting,
and cleaning up properly afterwards. Maybe some of that extra code
is what helps do things like tail packing. But I consider it a good
feature to have.
--
-----------------------------------------------------------------
│ Phil Howard - KA9WGN │ Dallas │ http://linuxhomepage.com/ │
│ phil-nospam@ipal.net │ Texas, USA │ http://phil.ipal.org/ │
-----------------------------------------------------------------
-
To unsubscribe, send mail to majordomo@postfix.org with content
(not subject): unsubscribe postfix-users
Message 34 in thread
寄件者：Phil Howard (phil-postfix-users@ipal.net)
主旨：Re: virtualizing local users
View this article only
新闻群组：mailing.postfix.users
日期：2002-07-02 06:14:11 PST
On Tue, Jul 02, 2002 at 12:49:19PM +0200, Ralf Hildebrandt wrote:
│ On Tue, Jul 02, 2002 at 05:14:36AM -0500, Phil Howard wrote:
│
│ > │ Looking up something in one btree:
│ > │ O(n) = ln(n) (well, sort of)
│ > │
│ > │ Looking up something in x btrees of size (n/x):
│ > │
│ > │ O(n) = x*ln(n/x)
│ > │ which is bigger than ln(n)
│ >
│ > What is x? I think you are making the same mistake Victor did and are
│
│ x << n
│
│ (x is the number of maps)
│
│ > trying to sequentially search each map separately. Such a search can't
│ > even work since a map per-domain should have only the user part as the
│ > key.
│
│ All maps are searched. So if you key is in the last map, it has to
│ search each maps until a match is found.
That would produce the wrong results.
│ > Just search ONE map, the one for the appropriate domain. Then we're
│
│ Postfix doesn't do that (yet)
So it might be me who makes it do it.
--
-----------------------------------------------------------------
│ Phil Howard - KA9WGN │ Dallas │ http://linuxhomepage.com/ │
│ phil-nospam@ipal.net │ Texas, USA │ http://phil.ipal.org/ │
-----------------------------------------------------------------
-
To unsubscribe, send mail to majordomo@postfix.org with content
(not subject): unsubscribe postfix-users
Message 35 in thread
寄件者：Victor.Duchovni@morganstanley.com (Victor.Duchovni@morganstanley.com)
主旨：Re: virtualizing local users
View this article only
新闻群组：mailing.postfix.users
日期：2002-07-02 06:21:43 PST
On Tue, 2 Jul 2002, Phil Howard wrote:
> Just search ONE map, the one for the appropriate domain. Then we're
> doing 1*ln(n/x) ... times whatever it takes for the filesystem to
> find that map ... which makes it ln(n/x)+ln(x) ... or ln(max(n/x,x))
> as far as big-O is concerned.
>
To avoid running out of file descriptors this would need to be implemented
as a new (to Postfix) hierarchical map type. It will complicate chroot
setup, because the map hierarchy would need to be loopback (bind) mounted
into the chroot jain (always the Postfix queue directory). Additionally
the submaps for each domain would need to opened and closed on the fly,
with a suitable cache of open maps to avoid opening and closing maps for
every lookup. The underlying per domain map would need to be "cdb", not
Berkeleyu DB, since allocating and deallocating page pools and other map
open overhead in Berkeley DB potentially impacts lookup performance in a
negative way). What is the relative cost of map open vs. key lookup with
"cdb"?
Postfix maps do not know whether the lookup key is a user name, address,
domain name, ..., this happens at a higher layer accross all the map
types. Postfix maps just look up strings. Your map needs to parse its
keys into a domain and a user part. This conflicts with the Postfix
keyed map abstraction. Like "regexp" maps your map might need to avoid
"partial" keys so as not to get confused by source routes. There may be
other semantic subleties.
Actually LDAP maps with ldapsource_domain set also violate the abstraction
when filtering the input keys by domain, perhaps such maps should also
avoid "partial" keys.
Your proposal is a non-trivial development effort. Are you sure that your
energy and enthusiasm will be best spent in this direction? It would be
much easier to deploy something existing of the shelf even if slightly
imperfect the solutions will have been tested by others and known to work.
Of course all real progress is made by people who are wisely or otherwise
unhappy with the status quo, I do not want to discourage creativity, just
be aware that innovation is a risky business... :-(
--
Viktor.
寄件者：Phil Howard (phil-postfix-users@ipal.net)
主旨：Re: virtualizing local users
View: Complete Thread (36 articles)
Original Format
新闻群组：mailing.postfix.users
日期：2002-07-02 07:51:24 PST
On Tue, Jul 02, 2002 at 09:20:44AM -0400, Victor.Duchovni@morganstanley.com wrote:
│ On Tue, 2 Jul 2002, Phil Howard wrote:
│
│ > Just search ONE map, the one for the appropriate domain. Then we're
│ > doing 1*ln(n/x) ... times whatever it takes for the filesystem to
│ > find that map ... which makes it ln(n/x)+ln(x) ... or ln(max(n/x,x))
│ > as far as big-O is concerned.
│ >
│
│ To avoid running out of file descriptors this would need to be implemented
│ as a new (to Postfix) hierarchical map type. It will complicate chroot
│ setup, because the map hierarchy would need to be loopback (bind) mounted
│ into the chroot jain (always the Postfix queue directory). Additionally
│ the submaps for each domain would need to opened and closed on the fly,
│ with a suitable cache of open maps to avoid opening and closing maps for
│ every lookup. The underlying per domain map would need to be "cdb", not
│ Berkeleyu DB, since allocating and deallocating page pools and other map
│ open overhead in Berkeley DB potentially impacts lookup performance in a
│ negative way). What is the relative cost of map open vs. key lookup with
│ "cdb"?
My understanding is that DJBDNS opens cdb for every request.
│ Postfix maps do not know whether the lookup key is a user name, address,
│ domain name, ..., this happens at a higher layer accross all the map
│ types. Postfix maps just look up strings. Your map needs to parse its
│ keys into a domain and a user part. This conflicts with the Postfix
│ keyed map abstraction. Like "regexp" maps your map might need to avoid
│ "partial" keys so as not to get confused by source routes. There may be
│ other semantic subleties.
I might create a map type that simply constructs strings from keys. Maybe
the regexp map can already do what I need.
│ Actually LDAP maps with ldapsource_domain set also violate the abstraction
│ when filtering the input keys by domain, perhaps such maps should also
│ avoid "partial" keys.
│
│ Your proposal is a non-trivial development effort. Are you sure that your
│ energy and enthusiasm will be best spent in this direction? It would be
│ much easier to deploy something existing of the shelf even if slightly
│ imperfect the solutions will have been tested by others and known to work.
│
│ Of course all real progress is made by people who are wisely or otherwise
│ unhappy with the status quo, I do not want to discourage creativity, just
│ be aware that innovation is a risky business... :-(
Another option on the table being explored is developing my own SMTP daemon.
But this would be a very specialized one, not for general use. For example
it would not do any relaying whatsoever, so if you want to be able to inject
mail and have it delivered on the internet, or run an MX other than the final
destination, run another MTA, e.g. Postfix. It could also do LMTP. It would
only do maildir so that it wouldn't face having to otherwise queue mail, so
it would be a deliver-during-receipt type daemon, too. Its primitiveness
and lack of intent to support various other sendmail-ish things means it
would be a smaller project, and potentially viable. I'm not sure if would
go this route, but it's still an option. I have ruled out other MTAs for
the most part.
--
-----------------------------------------------------------------
│ Phil Howard - KA9WGN │ Dallas │ http://linuxhomepage.com/ │
│ phil-nospam@ipal.net │ Texas, USA │ http://phil.ipal.org/ │
-----------------------------------------------------------------
Posted by hzqbbc at 03:20 PM | Comments (1)
November 01, 2002
对qmgr_message_active_limit 与 default_process_limit的高级分析
First the summary:
------------------
The deferred messages dominate the queue. In steady state the
"deliverable" messages just sail by and can be ignored. Size the process
limit to accommodate the "slow" messages + a reasonable estimate of active
sessions that the hardware and network will allow. Better yet use a
separate active queue, which for now requires a second Postfix that gets all
deferred mail (via fallback_relay???).
Since the queue manager knows which messages came in fresh from
"incoming/maildrop" and which from "deferred", it could keep two directory
hierarchies "active" and "reactive" and apply different process limits to
each. This would also help the folks with the solid-state disks, who do not
want deferred mail to consume any of the precious storage, and finally
"flush" would be much less tragic, as it would interfere with the delivery
of fresh mail.
Author: Victor Duchovni (Victor.Duchovni@morganstanley.com)
And now the details:
--------------------
The limit on the active queue size is intended to bound the CPU and memory
cost of the queue manager scheduling algorithms. Think of the queue manager
as being similar to an OS scheduler, it has a concurrency limit for running
jobs (here active instances of "smtp" and other transports rather than
physical CPU) and a limit for runnable jobs (at a stretch analogous to the
size of the process table). There is a difference between a running job
(actually being processed by one of your 400 allocated transport slots) and
a runnable job (a message in the active queue). The limit on the total
number of "runnable" jobs should be as large as possible without expending
unreasonable CPU cost in the scheduling algorithm. A large active queue
limit will let the queue manager deal gracefully with bursts of new mail
that briefly overwhelm the sustained output rate the queue.
A healthy MTA is on average able to deliver mail faster than the input rate
(or it dies of congestive collapse). The active queue consists of just
arrived messages that are being tried for the first time (average occupancy
= input message rage * average time to deliver, bounce or defer) and also
deferred messages being retried (average occupancy = deferred queue size *
fraction of time deferred messages spend in active queue).
It turns that for large sites the (previously) deferred messages typically
constitute the majority of the "running" messages and contribute most to the
active queue population. Suppose for example that your message input rate
is 100 messages per second, and you are able to process the average message
with a 1 second delay. This means that the active queue in steady state
will have 100 newly arrived messages.
On the other hand with 100 messages per second your deferred queue may have
20000 or more deferred messages. With a maximum backoff of 4000 seconds,
each deferred message "visits" the active queue every 4000 seconds or so.
As the message was deferred it is likely to be deferred again, typically due
to a multi-minute timeout (say 3 minutes == 180 seconds). This means that
you typically have (180/4000)*20000 = 900 messages in the active queue that
have previously been deferred and are likely to be deferred again. These
900 messages will tie up all the available delivery slots and the queue will
stall. It is very important to make sure that the default process limit
exceeds the expected population of reactivated deferred mail.
There are a couple things you can do to improve throughput on machines that
have large quantities of deferred mail. One (the best for really high
volume sites IMHO) option is to hand off the deferred mail to a different
postfix instance that only handles deferred mail. This keeps the
reactivated deferred mail from competing with the new mail for delivery
slots, but requires a second copy of Postfix which still needs some tuning.
I have not yet had to go to that "extreme". Here are a few simpler things
you can do:
1. If reasonable for your user community reduce "maximal_queue_lifetime"
(I use 2 days rather than 5). This should proportionally reduce the active
queue footprint of the deferred mail.
2. Increase the "maximal_backoff_time" (I use 14400s), this will again
proportionally reduce the average number of reactivated messages.
3. Use "nqmgr" and use a different (in name only, I use "relay") SMTP
transport for your "relay" domains if you forward mail for your domains via
SMTP to an internal gateway. "nqmgr" implements fairness between
transports, and you do not want to round-robin the high inbound message rate
(50% of all traffic for many sites) against all the hundreds of "slow"
domains in the active queue. This can keep inbound mail flowing even when
the active queue is congested.
4. Avoid "flush" (a.k.a. ETRN a.k.a. "sendmail -q") like the plague.
Disable ETRN for all clients except those on the "fast_flush_domains" list.
There is nothing worse than thousands of undeliverable messages in the
active queue (instant DoS). The deferred messages are deferred for a
reason, and while one wants to retry them, doing so all at once is a recipe
for disaster.
5. Size default_process_limit with the expected number of deferred
messages being retried as a baseline. Such messages spend most of their
time idle, so they don't really burn CPU, the actual output is being
performed by the additional delivery slots allocated over and above the
active count of previously deferred messages.
My own experience:
------------------
My initial configuration with the default timers and process limit (of 50)
had anywhere from 40 to 60 "slow" messages in the active queue at a time.
This was not fun to watch, a fast machine completely idle, not delivering
any mail. Scaling the queue age and backoff timers reduced the "slow"
occupancy to around 10, and raising the default_process_limit from 50 to 500
was more in line with the capabilities of the hardware (8 × 400MHz CPU
E4500).
Just as an illustration: right now (the slowest time for mail at my site)
the numbers on one of the relays are as follows:
# find `postconf -h queue_directory`/active -type f -print│ wc -l
12
# netstat -an -f inet │ grep SYN_SENT │ wc -l
9
# find `postconf -h queue_directory`/deferred -type f -print │ wc -l
733
This shows at least 9 out of 12 active delivery slots going to sites with
connections in SYN_SENT (host not responding). 9/733 is within 4% of
180/14400 (three minutes to time out a TCP connection and then another 14400
second backoff). Your mileage may vary based on OS TCP settings and
differences in the average number of A records corresponding to MX records
for the dead domains.
I hope this helps.
--
Viktor.
>
> This should be an easy one :-) I'm trying to tune a
> MX spooler. It has about 21,000 queued messages, all in
> various stages of delivery. I'm running
> Postfix-20010228-pl01 on it. qmgr_message_active_limit
> defaults to 10,000, and I usually have default_process_limit
> set to something like 400 (dual p3 w/ 512mb ram, so given
> the size of smtp processes, I figured 400 would be nice and
> safe).
>
> Here's the question. If postfix is pulling messages out of
> deferred and incoming, and putting them in active, how are
> 400 smtp process (obviously not all 400 are smtp, but most
> are) going to process 10,000 in a timely manner? shouldn't
> the active queue match the number of processes taking mail
> out of it? Please, be brutal. Tell me where my logic is
> going wrong.
>
>
> --
> Joshua Warchol
> -
Posted by hzqbbc at 01:12 AM | Comments (0)
October 31, 2002
Ext3文件系统如何应用于Postfix
EXT3文件系统可以看作是EXT2的升级版本，在兼容EXT2的前提下实现了日志，性能也不至于下降很多，为数据的可靠与连续性提供了较好的保证。
为了使Postfix在EXT3文件系统下发挥最大作用，必须仔细的调整（tuning）。以下是Ralf Hildebrandt（德国）一篇简短的HOWTO，介绍如何调整和优化，非常值得一看。
Postfix on an ext3 filesystem
Warning
Hein Roehrig notes: According to http://www.redhat.com/mailing-lists/ext3-users/msg07522.html there is a bug in the Linux kernel 2.4.18 and 2.5.x which makes ext3 data=journal unsafe when both mmap() and write are used on a file. db3 does exhibit this behavior.
IMHO it should be safe to use data=journal on /var/spool/postfix.
Recommended Setup
use data=journal journalling mode for the $queue_directory (usually /var/spool/postfix).
ext3 was designed to perform full data and metadata journaling. In this mode (called "data=journal" mode), the JBD journals all changes to the filesystem, whether they are made to data or metadata. Because both data and metadata are journaled, JBD can use the journal to bring both metadata and data back to a consistent state.
For the $queue_directory we get no performance penalty, because most queuefiles dont stay in the queue very long, thus they're written to the journal only and deleted from there.
mount $queue_directory using the option noatime.
Postfix doesn't care about the atime of a queuefile. Since atime updates affect the file and all directories above it, we get a performance boost by that.
Since we're using data=journal, we can safely say
chattr -R -S /var/spool/postfix
ext3 guarantees the integrity of the directory
make sure that /var/log is not using data=journal journalling mode.
data=ordered (the default) gives the same guarantees as data=journal when just appending (and you're appending to logs only!) -- while being faster.
This usually means you have to have a partition for /var/log and another one for /var/spool/postfix
Disable the write-cache for your drive:
Linux cannot currently deal with write caches in drives properly, because it cannot tell the drive when to flush the cache.
Current disk drives have big caches (2 MB and in excess of that) and can easily swallow several hundred mails at a time. You don't want that. Tagged Command Queueing (SCSI only in Linux at the moment) partially makes up for that.
Write barrier support is on its way (Chris Mason posted an alpha-quality patch to the linux-kernel and reiserfs mailing lists, it only supports ATA and AIC7XXX SCSI, but no qlogic or LSI SCSI, not sure what file systems are supported), which is supposed to run Linux safely with write caches on, but before this is solid and in the regular kernel, using write caches can kill your file system. All ATA drives I bought recently have had their caches turned on, so watch out. On SCSI, I've seen to opposite, all SCSI drives I know of have their fast write cache off.
hdparm -W0 /dev/hda
Use
elvtune -r 4096 -w 8192
for the drive containing your $queue_directory. On soft-RAID devices, you need to use elvtune on each physical drive!
I usually use:
/dev/hda3 on / type auto (rw,errors=remount-ro) /dev/hda1 on /boot type ext3 (rw) /dev/hda6 on /var/spool type ext3 (rw,data=journal,noatime) /dev/hda7 on /var/log type ext3 (rw)
References
The benefits of the ext3 filesystem (Part 7)
Surprises in ext3 (part 8)
Visit Andrew Morton's ext3 and 2.4 usage page to complete your ext3 setup.
Posted by hzqbbc at 04:35 PM | Comments (0)
Postfix邮件列表上关于大容量系统的讨论
本文是2002年秋季，我在Postfix邮件列表里搜集的一些有关Postfix大容量Email系统的讨论，其中讨论的人不乏Postfix开发的Hacker，当然少不了作者Wietse Venema 。
讨论的焦点大致有：
O 如何提高队列的性能
O 如何提高投递的速度
O 有关LDAP的优化
O 有关I/O的瓶颈分析
......
寄件者：Gianugo Rabellino (gianugo@apache.org)
主旨：Scalability issues
View: Complete Thread (14 articles)
Original Format
新闻群组：mailing.postfix.users
日期：2002-10-30 01:54:04 PST
(note: I sent this message already two days ago, but it didn't show up
on the list. I'm sending it again, hoping this time will get through.
Just in case, sorry for the duplicate)
Hi,
I'm running a huge email gateway (~1.000.000+ e-mail per day). The
system has postfix running as a frontend SMTP gateway, dealing with
approx. half of the traffic (there is another - commercial - mail router
splitting the load with postfix).
~80% of the email traffic is sent to a farm of inbound hosts, the rest
is sent to the outside world by users authenticated via SMTP-AUTH. The
system relies heavily on LDAP lookups both for canonicalization and
transport lookup.
The postfix machine is a Linux 2.4.18-3, queue files are on a separate
disk, as an ext3 fs with data=journal and noatime. The kernel file-max
and threads have been risen up. The machine is running its own dnscache
process to speed up lookups. I've spent several days tuning up the
system, reading documentation and lurking the list and the archives: I'm
pretty confident, although I might have done something stupid on the
way, that I optimized the system quite heavily, following Ralf
Hildebrandt's most helpful hints as well as other documentation I found
on the net (Victor Duchovni really enlightened me with
http://archives.neohapsis.com/archives/postfix/2001-07/0871.html), but
still the system sometimes just screws up. When a peak load of inbound
traffic hits the system (peak = ~150 concurrent and sustained smtp
connections), Postfix queues start to grow from the average of 800-1300
up to 5-10.000.
When Postfix hits this number of mails in queue, it starts spiraling
down: email still flows in, but the outgoing rate slows dramatically.
The system climbs up to 20, 30, 40.000 emails in queue and there is no
way out, with loads of e-mails in the active queue. The workaround I
found is to stop postfix, mount the queue directory on another location,
start postfix with a fresh queue and fire a second postfix instance on
the old queue directory to clean up the remains.
The good news is that the two postfix instances keep up nicely: in a
matter of a few hours the queues have been spooled out completely and I
can rollback to the original setup.
My first suspect was LDAP, but since the two postfix instance happily do
their job with a huge load on the LDAP server. The problem isn't the MTA
behind postfix too, for the same reason among others. I strongly suspect
in this scenario an heavy I/O issue, but I'm wondering if I did
something dead wrong or if there is a real scalability issue with
postfix on large amounts of traffic, so that there might be next to
nothing to do to solve this issue in a clean way (true, I can put more
servers, buy faster disks and so on, but iron-based only scalability
worries me a lot).
Following is the relevant part of main.cf. In master.cf I'm using nqmgr
and I have 100 smtpd and 700 smtp processes.
default_process_limit = 2000
smtp_destination_concurrency_limit = 100
trigger_timeout = 1
in_flow_delay = 10s
qmgr_message_active_limit = 10000
qmgr_message_recipient_limit = 10000
hash_queue_names = defer,deferred,incoming,active
queue_run_delay = 120s
minimal_backoff_time = 60s
maximal_backoff_time = 180s
smtp_connect_timeout = 15
qmgr_site_hog_factor = 100
Is there anything stupid that I did? Is this a (un)known postfix limit,
of which I doubt? Would using the two instance trick as a permanent
setup help? Where can I start working on this issue?
Thanks in advance for your assistance, fire at will :-)
--
Gianugo Rabellino
寄件者：Ralf Hildebrandt (Ralf.Hildebrandt@charite.de)
主旨：Re: Scalability issues
View: Complete Thread (14 articles)
Original Format
新闻群组：mailing.postfix.users
日期：2002-10-30 02:00:05 PST
On Wed, Oct 30, 2002 at 10:50:08AM +0100, Gianugo Rabellino wrote:
> The postfix machine is a Linux 2.4.18-3, queue files are on a separate
> disk, as an ext3 fs with data=journal and noatime.
You read http://www.stahl.bau.tu-bs.de/~hildeb/postfix/ext3.shtml ?
(especially the "chattr -R -S /var/spool/postfix" bit?)
> The kernel file-max and threads have been risen up. The machine is
> running its own dnscache process to speed up lookups.
Excellent. This all sounds good.
> I've spent several days tuning up the system, reading documentation
> and lurking the list and the archives: I'm pretty confident, although
> I might have done something stupid on the way, that I optimized the
> system quite heavily, following Ralf Hildebrandt's most helpful hints
> as well as other documentation I found on the net (Victor Duchovni
> really enlightened me with
> http://archives.neohapsis.com/archives/postfix/2001-07/0871.html),
> but still the system sometimes just screws up.
oops.
> When a peak load of
> inbound traffic hits the system (peak = ~150 concurrent and sustained
> smtp connections), Postfix queues start to grow from the average of
> 800-1300 up to 5-10.000.
OK. That means the incoming traffic starves the outgoing traffic.
in_flow_delay = 5s
> When Postfix hits this number of mails in queue, it starts spiraling
> down: email still flows in, but the outgoing rate slows dramatically.
Yup. Two ideas:
Reduce the max. number of smtpd processes so the qmgr can keep up and
the smtp's can send out mail.
> My first suspect was LDAP, but since the two postfix instance happily do
> their job with a huge load on the LDAP server. The problem isn't the MTA
> behind postfix too, for the same reason among others. I strongly suspect
> in this scenario an heavy I/O issue, but I'm wondering if I did
Probably.
> Following is the relevant part of main.cf. In master.cf I'm using nqmgr
> and I have 100 smtpd and 700 smtp processes.
Sounds sane.
> default_process_limit = 2000
> smtp_destination_concurrency_limit = 100
> trigger_timeout = 1
> in_flow_delay = 10s
According to the docs, this can be only in the range 0s-5s!
> queue_run_delay = 120s
You run the queue quite often!
--
Ralf Hildebrandt Ralf.Hildebrandt@charite.de
Postfix Tips: http://www.arschkrebs.de/postfix/ Tel. +49 (0)30-450 570-155
perl -e 'print $i=pack(c5,(41*2),sqrt(7056),(unpack(c,H)-2),oct(115),10);'
寄件者：Gianugo Rabellino (gianugo@apache.org)
主旨：Re: Scalability issues
View: Complete Thread (14 articles)
Original Format
新闻群组：mailing.postfix.users
日期：2002-10-30 02:28:04 PST
Ralf Hildebrandt wrote:
> >The postfix machine is a Linux 2.4.18-3, queue files are on a separate
> >disk, as an ext3 fs with data=journal and noatime.
>
>
>
> You read http://www.stahl.bau.tu-bs.de/~hildeb/postfix/ext3.shtml ?
> (especially the "chattr -R -S /var/spool/postfix" bit?)
Yep:
- mount │ grep postfix
/dev/sdb1 on /var/spool/postfix type ext3 (rw,noatime,data=journal)
- lsattr /var/spool/postfix
-------------- /var/spool/postfix/lost+found
-------------- /var/spool/postfix/active
-------------- /var/spool/postfix/bounce
-------------- /var/spool/postfix/corrupt
-------------- /var/spool/postfix/defer
-------------- /var/spool/postfix/deferred
-------------- /var/spool/postfix/flush
-------------- /var/spool/postfix/hold
-------------- /var/spool/postfix/incoming
-------------- /var/spool/postfix/private
-------------- /var/spool/postfix/maildrop
-------------- /var/spool/postfix/public
-------------- /var/spool/postfix/pid
> Yup. Two ideas:
>
> Reduce the max. number of smtpd processes so the qmgr can keep up and
> the smtp's can send out mail.
That's what I did ATM. My issue is that lowering the smtpd processes I'm
slowing down the incoming email traffic, so I'm trying to understand the
figures I'm able to keep up with, in order to understand if I have to
plan for more powerful iron, more servers, or just another architecture.
I basically need to understand what is the peak load of inbound traffic
that I can sustain.
> >default_process_limit = 2000
> >smtp_destination_concurrency_limit = 100
> >trigger_timeout = 1
> >in_flow_delay = 10s
>
>
> According to the docs, this can be only in the range 0s-5s!
I couldn't find any doc mentioning that limit, just some mailing list
archives. I resorted to the source code and read in in
src/global/mail_params.c
static CONFIG_TIME_TABLE time_defaults[] = {
[...]
VAR_IN_FLOW_DELAY, DEF_IN_FLOW_DELAY, &var_in_flow_delay, 0, 10,
0,
};
So I assumed that I could go as high as 10 sec.
> >queue_run_delay = 120s
>
>
> You run the queue quite often!
My point was to not let many incoming mails accumulate in the queue.
Does it make sense or is it actually doing more harm than good to the
system?
Thanks for your assistance,
--
Gianugo Rabellino
寄件者："Edward Wildgoose" (Edward.Wildgoose@FRMHedge.com)
主旨：RE: Scalability issues
View: Complete Thread (14 articles)
Original Format
新闻群组：mailing.postfix.users
日期：2002-10-30 03:12:04 PST
Suggest investigating choice of raid controllers before adding disks. A decent SCSI controller with write back cache should make a big difference. On IDE, the 3ware controllers are allegedly pretty good - don't have one, but some people have suggested that transaction rates can increase quite a bit (although they don't have any write back cache, so will never compete with the SCSI stuff).
Also can you refuse any of the incoming email...
A few body checks to eliminate top 3 virus's perhaps, plus use some conservative DNSBL or blocking based on FROM address. Ensure that FROM and TO are real domains and fully qualified, etc...
If you want to go the simple IDE route, then consider that you have several different disk write patterns and simply moving directories over different disks gives you a lot of the benefit of striping, eg put queue dir on a stripe, log on another disk, etc. (Make sure logging is asynchronous!)
Just a few thoughts. Perhaps you have been here already.
Ed W
-----Original Message-----
From: Gianugo Rabellino [mailto:gianugo@apache.org]
Sent: 30 October 2002 10:58
To: postfix-users@postfix.org
Subject: Re: Scalability issues
Ralf Hildebrandt wrote:
> >That's what I did ATM. My issue is that lowering the smtpd processes I'm
> >slowing down the incoming email traffic, so I'm trying to understand the
> >figures I'm able to keep up with, in order to understand if I have to
> >plan for more powerful iron, more servers, or just another architecture.
>
> Try putting the queue_directory on a fast striping RAID
You know what? I'm starting to think whether, scalability wise, this can
be the best solution. I see two different approaches in general to
issues like these:
1. the "big iron" approach. I can buy expensive n-way hardware, load it
with gigs of RAM, put RAID controllers, 15k disk drives, and spend big
money for one machine, hoping that it might be able to keep up with my
traffic peaks;
2. the "Google way": inexpensive hardware added "on demand". Two
machines with a fast SCSI drive, or four machines with a fast IDE drive.
More fault tolerant, more scalability, possibly at the same cost.
I tend to like 2) more. I'm just trying to understand, on my average
machine, how many inbound connections I can sustain before my outgoing
rate is unable to keep up. ATM I tend to think that the figure goes
beyond 50-80, but I want to be sure that this is a reasonable limit and
not a misconfiguration issue (besides, I think that this is a quite
small figure). I was astonished when I saw the dramatic improvement in
performance by "simply" chattr'ing the spool directory, so I'm wondering
if there is another sweet spot still untouched.
> Both incoming and outgoing mail compete for the disk, as does the
> qmgr. That means by running the qmgr too often, you're slowing the
> whole thing down.
Makes absolutely sense. I removed the setting, I'll see what happens in
the next hours.
One more question, though. Since I'm sending 80% of all my email to the
same inbound server, which is currently capable of handling gracefully
the incoming traffic, is there any other setting I can tweak to make
sure that I'm "hammering" it as fast and as much as I can?
Again, thanks for your answers and support.
--
Gianugo Rabellino
-
To unsubscribe, send mail to majordomo@postfix.org with content
(not subject): unsubscribe postfix-users
-
寄件者：lst_hoe (lst_hoe@kwsoft.de)
主旨：Re: Scalability issues
View: Complete Thread (14 articles)
Original Format
新闻群组：mailing.postfix.users
日期：2002-10-30 03:14:01 PST
At 10:50 30.10.2002 +0100, Gianugo Rabellino wrote:
>(note: I sent this message already two days ago, but it didn't show up
>on the list. I'm sending it again, hoping this time will get through.
>Just in case, sorry for the duplicate)
>
>Hi,
>
>I'm running a huge email gateway (~1.000.000+ e-mail per day). The
>system has postfix running as a frontend SMTP gateway, dealing with
>approx. half of the traffic (there is another - commercial - mail router
>splitting the load with postfix).
>
>~80% of the email traffic is sent to a farm of inbound hosts, the rest
>is sent to the outside world by users authenticated via SMTP-AUTH. The
>system relies heavily on LDAP lookups both for canonicalization and
>transport lookup.
>
>The postfix machine is a Linux 2.4.18-3, queue files are on a separate
>disk, as an ext3 fs with data=journal and noatime. The kernel file-max
>and threads have been risen up. The machine is running its own dnscache
>process to speed up lookups. I've spent several days tuning up the
>system, reading documentation and lurking the list and the archives: I'm
>pretty confident, although I might have done something stupid on the
>way, that I optimized the system quite heavily, following Ralf
>Hildebrandt's most helpful hints as well as other documentation I found
>on the net (Victor Duchovni really enlightened me with
>http://archives.neohapsis.com/archives/postfix/2001-07/0871.html), but
>still the system sometimes just screws up. When a peak load of inbound
>traffic hits the system (peak = ~150 concurrent and sustained smtp
>connections), Postfix queues start to grow from the average of 800-1300
>up to 5-10.000.
>
>When Postfix hits this number of mails in queue, it starts spiraling
>down: email still flows in, but the outgoing rate slows dramatically.
>The system climbs up to 20, 30, 40.000 emails in queue and there is no
>way out, with loads of e-mails in the active queue. The workaround I
>found is to stop postfix, mount the queue directory on another location,
>start postfix with a fresh queue and fire a second postfix instance on
>the old queue directory to clean up the remains.
>
>The good news is that the two postfix instances keep up nicely: in a
>matter of a few hours the queues have been spooled out completely and I
>can rollback to the original setup.
>
>My first suspect was LDAP, but since the two postfix instance happily do
>their job with a huge load on the LDAP server. The problem isn't the MTA
>behind postfix too, for the same reason among others. I strongly suspect
>in this scenario an heavy I/O issue, but I'm wondering if I did
>something dead wrong or if there is a real scalability issue with
>postfix on large amounts of traffic, so that there might be next to
>nothing to do to solve this issue in a clean way (true, I can put more
>servers, buy faster disks and so on, but iron-based only scalability
>worries me a lot).
>
>Following is the relevant part of main.cf. In master.cf I'm using nqmgr
>and I have 100 smtpd and 700 smtp processes.
>
>default_process_limit = 2000
>smtp_destination_concurrency_limit = 100
>trigger_timeout = 1
>in_flow_delay = 10s
>qmgr_message_active_limit = 10000
>qmgr_message_recipient_limit = 10000
>hash_queue_names = defer,deferred,incoming,active
>
>queue_run_delay = 120s
>minimal_backoff_time = 60s
>maximal_backoff_time = 180s
>smtp_connect_timeout = 15
>qmgr_site_hog_factor = 100
>
>Is there anything stupid that I did? Is this a (un)known postfix limit,
>of which I doubt? Would using the two instance trick as a permanent
>setup help? Where can I start working on this issue?
>
>Thanks in advance for your assistance, fire at will :-)
Just some ideas (you don′t mention your hardware):
1.) qmgr_message_active_limit = 10000 is a lot of stuff to process/sort in
the active queue. Maybe less would be better for overall throughput.
2.) smtp_destination_concurrency_limit = 100 means you do up to 100
connections to one destination. Depending on the target hosts you should
lower this. Maybe have a look at smtp_destination_recipient_limit for you
internal hosts.
3.) For LDAP have a look at "ldap_bind" and "ldap_domain" parameter.
4.) Transport entries for your internal servers should be with IP-Adress
and "[]" around to save DNS lookups. Or even better dedicated services in
master.cf with dns lookups disabled. There have been discussion about
LDAP-maps for transport and a memory leak some time ago.
Helpful would be if you could provide the numbers for memory/processor load
and maybe some entries from the logfiles (warnings/errors etc.) from the
point your machine starts spiraling down.
Regards
--
Andreas H?dle
Kühn & Weyh Software GmbH
WWW.KWSOFT.DE
寄件者：Galen Johnson (gjohnson@trantor.org)
主旨：Re: Scalability issues
View this article only
新闻群组：mailing.postfix.users
日期：2002-10-30 05:48:06 PST
One thing I have yet to see mentioned, and since you are looking at the
possibilty of multiple machines anyway, is what about having separate
servers for incoming and outgoing. That way you split the load between
separate machines and forego the contention of incoming and outgoing
fighting for i/o (mostly).
=G=
-
寄件者：Gianugo Rabellino (gianugo@apache.org)
主旨：Re: Scalability issues
View this article only
新闻群组：mailing.postfix.users
日期：2002-10-30 07:02:05 PST
Ralf Hildebrandt wrote:
> >if there is another sweet spot still untouched.
>
>
> Check /etc/resolv.conf and /etc/nsswitch.conf
What for? In resolv.conf I only have my local (127.0.0.1) nameserver
configured plus another two for failover. nsswitch.conf is "dns files"
for hosts.
What exactly should I look for?
--
Gianugo Rabellino
寄件者：Ralf Hildebrandt (Ralf.Hildebrandt@charite.de)
主旨：Re: Scalability issues
View this article only
新闻群组：mailing.postfix.users
日期：2002-10-30 07:16:04 PST
On Wed, Oct 30, 2002 at 04:00:04PM +0100, Gianugo Rabellino wrote:
> >Check /etc/resolv.conf and /etc/nsswitch.conf
>
> What for? In resolv.conf I only have my local (127.0.0.1) nameserver
> configured plus another two for failover. nsswitch.conf is "dns files"
> for hosts.
>
> What exactly should I look for?
For exactly this.
--
Ralf Hildebrandt Ralf.Hildebrandt@charite.de
Postfix Tips: http://www.arschkrebs.de/postfix/ Tel. +49 (0)30-450 570-155
If you give someone a program, you will frustrate them for a day; if
you teach them how to program, you will frustrate them for a lifetime.
寄件者：Gianugo Rabellino (gianugo@apache.org)
主旨：Re: Scalability issues
View this article only
新闻群组：mailing.postfix.users
日期：2002-10-30 07:44:01 PST
Victor.Duchovni@morganstanley.com wrote:
> Do you have a dedicated transport for relaying inbound mail? This was the
> most important piece of advice in my Jul 2001 posting. From what follows
> below, the answer appears to be no. The dedicated inbound transport needs
> to have a high (O(100)) destination concurrency.
Unfortunately I'm unable to do it. The problem is that the overall
architecture is based on a mailhost: attribute in the LDAP server that
is being used by other parts of the systems. This atribute contains just
the plain host, not in the postfix syntax transport:[host]. I had to put
a two-liner patch in transport.c to fix this, and I resorted to
default_transport to be put in place of the transport: keyword.
This, however, forbids me from using a dedicated transport. Or do you
see a way to implement it?
> How many "smtp" processes are actually observed to be running when the
> queue is full?
>
Approx. 100, but I want to ckeck better when (if) the problem rises up
again.
As per your suggestions, I will try them all. Just a few comments:
>
> >trigger_timeout = 1
>
> Don't.
OK. But why? :-)
> >queue_run_delay = 120s
> >minimal_backoff_time = 60s
> >maximal_backoff_time = 180s
>
>
> This is insane. I recommended raising maximal backoff time from 4000 to
> 14400 and you take it down to 180s! All the deferred messages are retried
> every 3 minutes, a sure recipe for disaster.
Makes sense, must have missed that. Done.
> >smtp_connect_timeout = 15
>
> Good, but 30s is more reasonable for the Internet at large. Set to
> 5s or 1s only when forwarding to your internal mailhubs.
I set it to 15, as a compromise between outgoing mail and dedicated
inbound, since I'm not able to set up the dedicated transport.
>
> Make sure your LDAP server is fast (<5ms query latency). Use
> ldapsource_domain = mydom.ain to limit the domains for which queries are
> sent to LDAP.
This makes a huge lot of sense. Will do it as well.
Thanks a lot for your hints,
--
Gianugo Rabellino
寄件者：Victor.Duchovni@morganstanley.com (Victor.Duchovni@morganstanley.com)
主旨：Re: Scalability issues
View this article only
新闻群组：mailing.postfix.users
日期：2002-10-30 08:10:05 PST
On Wed, 30 Oct 2002, Wietse Venema wrote:
> Gianugo Rabellino:
> > still the system sometimes just screws up. When a peak load of inbound
> > traffic hits the system (peak = ~150 concurrent and sustained smtp
> > connections), Postfix queues start to grow from the average of 800-1300
> > up to 5-10.000.
>
> With that many clients pounding the disk with incoming mail,
> nothing goes out unless you switch to a fater disk.
>
> To keep mail flowing out, reduce inflow. For example, reduce
> the number of SMTPD processes to something that your disk can
> actually handle.
>
I have ~1000000 messages a day, but this is accross 4 machines, each with
a 3 way stripe of fibre attached RAID-5 disks with a battery backed cache.
So I/O almost never hits raw disk and there are effectively 40 disks of
I/O bandwidth to soak up the traffic. Not surprisingly given the 3 layers
of conent filters on my system, it is CPU bound, I/O is handled quite
nicely.
So it is possible the system in question is running out of I/O muscle
(which iostat should easily reveal). That said, my heavy iron was also not
handling the load until it was configured correctly, the main ingredient
was the dedicated inbound transport.
--
Viktor.
寄件者：Russell Mosemann (mose@ns.cune.edu)
主旨：Re: Scalability issues
View this article only
新闻群组：mailing.postfix.users
日期：2002-10-30 08:12:05 PST
On Wed, 30 Oct 2002, Gianugo Rabellino wrote:
> Unfortunately I'm unable to do it. The problem is that the overall
> architecture is based on a mailhost: attribute in the LDAP server that
> is being used by other parts of the systems. This atribute contains just
> the plain host, not in the postfix syntax transport:[host]. I had to put
> a two-liner patch in transport.c to fix this, and I resorted to
> default_transport to be put in place of the transport: keyword.
The patch is not necessary. Besides, you don't want to be customizing
postfix, because then you are locked into maintaining it.
ldapsource_result_attribute = mailHost
ldapsource_result_filter = smtp:[%s]
----
Russell Mosemann, Ph.D. * Computing Services * Concordia University, Nebraska
"A thousand people will stop smoking today. Their funerals will be held
sometime in the next three or four days." C. E. Koop, Former Surg. Gen.
寄件者：Gianugo Rabellino (gianugo@apache.org)
主旨：Re: Scalability issues
View this article only
新闻群组：mailing.postfix.users
日期：2002-10-30 08:28:04 PST
Russell Mosemann wrote:
> The patch is not necessary. Besides, you don't want to be customizing
> postfix, because then you are locked into maintaining it.
>
> ldapsource_result_attribute = mailHost
> ldapsource_result_filter = smtp:[%s]
Ouch, where is my dumbass hat? I blatantly overlooked that setting.
Thanks for showing me the way, you solved one of my greatest PITA with
this solution. Now I think I can set up that dedicated transport after
all. :-)
Thanks again, you sorted me out.
寄件者：Wietse Venema (wietse@porcupine.org)
主旨：Re: Scalability issues
View this article only
新闻群组：mailing.postfix.users
日期：2002-10-30 09:20:03 PST
Gianugo Rabellino:
> still the system sometimes just screws up. When a peak load of inbound
> traffic hits the system (peak = ~150 concurrent and sustained smtp
> connections), Postfix queues start to grow from the average of 800-1300
> up to 5-10.000.
With that many clients pounding the disk with incoming mail,
nothing goes out unless you switch to a fater disk.
To keep mail flowing out, reduce inflow. For example, reduce
the number of SMTPD processes to something that your disk can
actually handle.
Wietse
寄件者：Victor.Duchovni@morganstanley.com (Victor.Duchovni@morganstanley.com)
主旨：Re: Scalability issues
View this article only
新闻群组：mailing.postfix.users
日期：2002-10-30 09:32:07 PST
On Wed, 30 Oct 2002, Russell Mosemann wrote:
> The patch is not necessary. Besides, you don't want to be customizing
> postfix, because then you are locked into maintaining it.
>
> ldapsource_result_attribute = mailHost
> ldapsource_result_filter = smtp:[%s]
>
This is what I was about to suggest, but is this available in stock
1.1.11? I know that result_filter is in the snapshot release, but I have a
feeling that this is not available in the official release.
This is critical for high volume inbound relay throughput. Gianugo should
either switch to non-LDAP transport maps or upgrade to the snapshot or both.
Given that 1.1.11 does not support per user transports, using LDAP for
transports seems quite pointless. Perhaps he is using the experimental
release after all.
--
Viktor.
Posted by hzqbbc at 04:29 PM | Comments (0)
September 03, 2002
如何在postfix队列里删除以特定字符开头的邮件？
作为Postfix MTA的管理员，维护队列是家常便饭，但如何能够方便的按自己的意愿控制对列呢？这需要一点perl和regexp的知识。
以下提供一个perl的小程序，由一个国外朋友写的，通过命令行传递正则表达式，匹配的邮件将被删除。
queue_mgr.pl
#!/usr/bin/perl
$REGEXP = shift || die "no email-adress given (regexp-style, e.g.
bl.*\@yahoo.com)!";
@data = qx</usr/sbin/postqueue -p>;
for (@data) {
if (/^(\w+)\*?\s/) {
$queue_id = $1;
}
if($queue_id) {
if (/$REGEXP/i) {
$Q{$queue_id} = 1;
$queue_id = "";
}
}
}
open(POSTSUPER,"| postsuper -d -") || die "couldn't open postsuper" ;
foreach (keys %Q) {
print POSTSUPER "$_\n";
};
close(POSTSUPER);
Posted by hzqbbc at 12:57 PM | Comments (0)
June 30, 2002
如何用Postfix取代Qmail呢？
qmail自1998年以来，DJB一直都没有任何更新，他认为qmail已经足够好了。可实际却不尽然。在邮件列表上，经常有不少人抱怨qmail，诸如要吃这样那样的补丁，补丁的不兼容，功能不够等等。
以下是在邮件列表上关于这个话题的一些精选内容：
一个人问：
========
I've been using qmail for a couple years, mainly installing it in
ISPs. It has a lot of features. But sometimes, it stops working,
without any hint. And support it's not so easy as other open source
projects, specially because the logs didn't show what is really
happening.
I've heard good opinions about Postfix and expect start using it instead of
qmail. But I have some minimum requirements to be able to switch.
My usual setup is:
1) qmail 1.03 without patches
Consequences:
--one unix user for each account
--maildir incredible performance
--pop and smtp in same package
--sendmail command line compatible
2) vpopmail with mysql support
Consequences:
--very easy virtual domains administration
--one unix user for all thousands account users
--all virtual domains are Mysql tables
--all virtual users em Mysql database
3) qmailadmin/autorespond/ezmlm Web Interface
Consequences:
--web admin interface to manage users, forwards, aliases, mailing lists
--web interface for users to change their passwords
--web interface for users to change vacation/forwards
--every domain owner can deal with their accounts
4) IMAP with all above support
-- Very dificult to implement
Except for item 4, I can setup all this using a plain RH 7.x base install,
some RPMs (mysql, gcc, apache, and few others) and items 1-3 compiling from
the tar.gz files, in a couple hours. Ie, no complicated configs,
compilings, etc.
If someone who is using Postfix and was a qmail user and have time to
discuss this, I will appreciate.
I don't want to start a flame war about this. I just want to discuss
technically the previous aspects.
Thanks everyone.
-Gilson
有一个人回答到：
====================================================================
all of the *features* you use are available with postfix (and more), but
the mechanisms are different. but that's part of the point, isn't it?
if you install postfix + parts of courier (especially courier-maildrop
courier-pop, courier-imap, and sqwebmail) then you will have a mail
server which can do system and/or virtual users, pop, imap, and webmail.
there are SSL encryption options for postfix, courier imap & courier pop
and, of course, any webmail service can (and probably should) be run on
a https server.
you can configure postfix and/or courier to get account details from
system files (/etc/passwd, nsswitch, pam) AND/OR from mysql, postgres,
hashed db map files or whatever you like.
the last remaining piece is the mailing list manager. the one that
probably comes closest to ezmlm's functionality would be GNU MailMan
(www.list.org), followed closely by ecartis (aka "listar") at
www.ecartis.org. neither are identical to ezmlm, but they do a
comparable job.
> Except for item 4, I can setup all this using a plain RH 7.x base
> install, some RPMs (mysql, gcc, apache, and few others) and items 1-3
> compiling from the tar.gz files, in a couple hours. Ie, no
> complicated configs, compilings, etc.
all of the above can be installed and configured easily on a debian box
just by apt-getting a handful of packages. dunno about RH, but i would
assume that packages are available for RH too (if not in RH itself, then
in contrib packages...which are of notoriously variable quality, some
are excellent, some are very poor).
my strongest recomendation would have to be to build a test machine and
install postfix and all of the above packages, and play with it for a
few days or a few weeks. my bet is that you will become a postfix
convert in short order :-), postfix really sells itself, especially to
experienced mail systems administrators who can easily see the benefits.
if you do switch to postfix, you will have to plan the transition very
carefully - changing MTA is not something to be undertaken lightly.
there are some significant conceptual differences to the way qmail works
and the way postfix (or any other MTA for that matter) works. qmail
tends to expect you to do everything DJB's One True Way, whether you
like it or not...other MTAs like postfix tend to be far more flexible.
also, most MTAs other than qmail have some core similarities because
they all provide some level of backwards-compatibility (or at least an
easy migration path) with sendmail. qmail really is the odd one out.
i say these things as statements of fact, not in an attempt to disparage
qmail...although i am obviously biased towards postfix. i've been using
postfix for several years now. i had previously used qmail for a few
years (and sendmail and smail and others for years before that). i
would never switch back to any of the others.
craig
Posted by hzqbbc at 12:40 AM | Comments (0)
June 29, 2002
Postfix实现转发/自动回复
在系统用户(Unix account)的$HOME下建立.forward文件，内容：
user@your-forward-mail-address
同时chmod 0640 .forward
小心的是属性为：user.usergroup
注意在/etc/postfix/main.cf中myhostname 加多一行，例如：
myhostname = smtp.hzqbbc.com myhostname = smtp.otherdomain.org
然后在mydestination后加要forward的域就基本可以了。
在postfix的maillist里提到实现如何转发时给自己留备份，或转发并删除的讨论：
Google group discuss
Message 13 in thread
寄件者："Renato A. Q. Salles" (linuxbr@onda.com.br)
主旨：RE: forwards
新闻群组：mailing.postfix.users
View this article only
日期：2002-06-27 21:16:20 PST
-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1
$ pico .forward
Put on the first line, without hitting enter, the email address you wish to have
your mail forwarded to:
me@counterexample.com
If you wish to forward just to another user whose account is @example.com,
you would just enter:
ernie
If you wish to leave a copy for yourself, but also forward a message, be sure
to put a backslash in front of your username so as to avoid infinite forwarding
loops. Use a comma between addresses:
\me, me@counterexample.com
To delete mail so that no one gets it:
/dev/null
To delete mail and send an autoreply (see the vacation page for other
requirements of sending autoreplies)
/dev/null, "│vacation andrew"
Source: Google
Posted by hzqbbc at 05:00 PM | Comments (0)