June 28, 2005
qmail sendmail postfix - 三种MTA的比较
关于sendmail/qmail/postfix孰优孰劣，以及部署邮件系统的时候该选哪一个的讨论已经重复了千百次了。但事实往往并不是A好B坏，或B好A坏，必须根据场合和应用的要求来定。但虽然如此，大多数人还是需要一个相对公平的评价，以引导邮件系统的部署。
自己一直很慎重于回答这类问题，以免引发不必要的争论甚至矛盾，但还是必须面对这个问题做一定的分析和比较的，否则很多朋友经常会问“到底用哪个好？”，却拿不出完整的答案。
首先看看三个MTA的历史...
MTAs的发展历史
Sendmail
毫无疑问，sendmail是最古老的MTA之一。它比qmail和postfix要古老得多。最早它诞生的时候，Internet还没有被标准化，当时主机之间使用的是UUCP技术来交换邮件。
它被设计得比较灵活，便于配置和运行于各种类型的机器。
Qmail
qmail是新生一代的MTA代表，它以速度快、体积小、易配置安装等特性而著称。作者D. J. Bernstein(djb)是一个数学教授，富有传奇色彩。djb于1995年开发qmail，1996年发布0.70版，并使用了多种当时比较先进的技术，包括Maildir，与sendmail单个binary不同的模块化设计，权限分离，以及使用了大量由djb编写的配套工具，如daemontools，ucsip-tcp等。
qmail迅速成为了Internet上最有名的MTA，使用者众。
Postfix
Postfix作者是Wietse Venema，一名著名的安全专家。最早postfix起源于1996年，当时venema 在美国IBM研究中心负责研究更安全的邮件系统，当时称为Vmailer。后因为商标问题于1998年11月正式更名为Postfix
Postfix以替代sendmail为目的，并提供了一个更安全、更高性能的灵活的体系。它同样也采用模块化设计，使用了大量优秀的技术，以达到安全的目的。由于作者的设计理念独到，经过7，8年时间，Postfix现今已发展成为功能非常丰富，扩展性和安全性强的优秀MTA。
概括的比较
以下的分析主要基于我在CASA上发的一个小文章，对sendmail/qmail/postfix做了一个概括性的比较。sendmail
sendmai功能非常强大，很多先进功能在sendmail上都最先有实现。sendmail里的Milter技术是一个非常好的框架，目前postfix及qmail仍然没有官方发布的方案比milter要好。
但sendmail也有典型的历史问题，只有一个binary程序，需要sid权限，m4配置文件复杂难懂。这些是是阻碍sendmail更好发展的一些客观问题。客观来说，调教得好的sendmail，其性能也是相当不俗的，据一个国外的Unix杂志称，在solaris+内存文件系统+带电池的raid系统下，sendmail能达到惊人的287封/秒的注入速度！
目前sendmail比较适合那些老用户，因为他们习惯了sendmail的应用环境和配置。
qmail
qmail体积非常小巧，source的gz包大概只有260多K，是三大MTA中最小的！模块化设计，避免了sid问题，基本功能齐全。配置相对sendmail而言，简单了很多，而且用户非常广泛。而且补丁和插件非常多，例如著名的vpopmail，netqmail，以及qmail-ldap等。
但qmail有几个问题，一是djb已经5，6年没有继续开发了，补丁的良莠不齐及版本依赖是非常麻烦的事，这对初学者极为不利。二是功能扩充需要补丁来完成，扩展能力不足。
总体上qmail依然是个非常不错的选择。对于希望了解mta原理，或希望修改mta代码的爱好者，qmail是值得推荐的。对于需要建立中小型邮件系统的用户也同样适合。而对于需要丰富功能却不想面对补丁困难，或者需要建立大型的系统，qmail不太合适，需要更丰富的经验和技术。
postfix
postfix如今已经独树一帜，流水线、模块化的设计，兼顾了效率和功能。灵活的配置和扩展，使得配置postfix变得富有趣味。其主要的特点是速度快、稳定，而且配置/功能非常强大，并和sendmail类似，提供了与外部程序对接的API/protocol。尤其是配置部分，可以说是一扫qmail和sendmail的各自缺点。
但postfix管理及配置的入门依然需要一定的工夫，必须仔细阅读官方文档。postfix另一个优势是至今依然保持活跃的开发工作，而且稳步发展，适合高流量大负载的系统，扩充能力较强。
大规模应用例子
国内若干个大型email ISP（如163.net/tom.com/163.com及sohu等）过去都使用qmail，后来全部更换成postfix。
新浪使用qmail，yahoo使用qmail。但这些已经不是普通的qmail了。
技术层面的分析
这里仅探讨一些典型的技术特点，从这些特点可以看出每个MTA设计的异同，主要讨论的焦点是qmail和postfix。磁盘I/O
从队列文件的读写来看，qmail处理每一封邮件时，都至少需要建立3个文件，mess, intd, info等。而Postfix使用的是单队列文件设计，因此磁盘I/O的开销要比qmail小得多，如果仅仅从这个方面考虑，postfix的队列是qmail的2-4倍那么快。
从我过去的一个qmail vs postfix对比测试中，也可以发现这个问题。
数据同步
如果从MTA对待操作系统的文件是否安全写入磁盘的策略来看，qmail和postfix也是不同的。Postfix使用的是随机写，并且需要写入完成并安全同步到磁盘后才算完成。而qmail的写入则是即刻执行的，因此它将等待数据安全写入磁盘后才返回。对于高流量的系统而言，这将导致性能问题。
此外，Postfix的队列对于FreeBSD的softupdate是安全的，而qmail则是不安全的，qmail作者明确警告用户不要使用softupdate，除非是有磁盘后写电池。
扩充能力
sendmail有着非常好的扩充能力，支持众多的特性，功能可谓豪华。包括频率控制到集群支持应有尽有。而milter API则更加使sendmail的灵活性发挥至极，通过milter，用户可以对邮件几乎所有的参数进行控制！但是在存储方面，由于只支持mbox，会有一定的问题。
qmail在系统容量扩展上有着独到的设计，配合qmail-ldap补丁，可以充分利用qmqp及分布存储的优势。现今已有各式各样的qmail扩展方案，最著名的是qmail-ldap。但qmail缺乏类似milter的设计，功能扩展需要各种补丁，而补丁的设计水平参差不齐，配置能力有限。实施起来相对是最复杂的。
Postfix同样有着非常好的容量扩充能力，利用LMTP或transport的/alias的方法，可以分布式的存储邮件，扩充容量。同时postfix的功能扩展也非常强，通过灵活的配置即可实现复杂的功能，这是其最突出的优点之一，是qmail望尘莫及的。此外，类似sendmail的milter，postfix拥有content_filter和policy 两个与外部程序/应用对接的接口，但不如milter那样功能集中和灵活，也没有完整实现qmail的qmqp及类似qmail-ldap的机制。
可配置性
sendmail 使用m4语法，单一的主配置文件（sendmail.cf）是三个mta中最难使用的，但是如果熟悉使用的话却能实现复杂的功能。
qmail使用的是大量小配置文本，格式最简单，每个配置一个文件，存放在/var/qmail/control目录里。
postfix也使用单一的主配置文件（main.cf），同时还有对应master主服务进程的配置文件master.cf，但使用的是简明易懂的key = value 格式。
总体而言，qmail的配置文件较易管理（格式最简单）但配置文件多（10个以上），而postfix的格式简单只有2个配置文件，并配备强大的postconf工具，sendmail的配置文件最复杂。
数据库支持
sendmail通过一些插件/补丁，可以支持mysql/pgsql/oracle等，ldap及小型的dbm/cdb等数据存储格式。
qmail默认只支持cdb，需通过补丁才可支持ldap，mysql，pgsql及oracle等。
postfix可以支持的数据库应该是最多的，默认就包括了mysql/pgsql/ldap及dbm/cdb和cidr/nis*/btree等一堆。还支持特殊的tcp_table(仅在snapshot里支持)
稳定性/负载能力
sendmail, qmail, postfix都比较稳定。在高负载下，配置不佳或没有打足够补丁的qmail容易被DOS攻击打跨，而postfix在遇到超过配置的限制时会降低处理能力，但系统依然有一定资源可用。
作者介绍
sendmail - Eric Allman Unix专家、学者
qmail - DJB 数学教授，科学家
Postfix - wietse venema 安全专家 学者
Recommentaion - 建议
我建议在使用Postfix MTA，无论是小型系统，还是大中型系统，能带来最高的性价比。一些有用的link
在足够好的硬件条件下Postfix比qmail更快的原因分析benchmark，无聊还是骗局？
qmail/postfix/sendmail 比较
Sendmail 历史
有关mta benchmark
Sendmail性能调整
qmail可靠性FAQ
Posted by hzqbbc at 12:15 PM | Comments (3)
June 27, 2005
Mbox vs Maildir - 两者原理和区别
在开放源代码的世界里，电子邮件服务器最主流的目前有三种，分别是sendmail, qmail, postfix。而存储格式最流行的有两种，Mbox和Maildir，它们都是开放的存储格式，因此兼容性比较好。
而mbox和maildir的历史，却不为大多数用户所了解。今天和一个朋友聊到mbox和maildir哪个好，顺便搜集一下mbox和maildir的资料，谈谈这两者的原理和区别，以及应该如何做选择。
Mbox的历史较Maildir悠久，sendmail支持mbox，qmail和postfix都支持mbox，其主要特点就是“所有邮件都存放到一个文件里”。每个邮件之间以特定的标记分割。
而Maildir则相反，每一封邮件保存成一个文件，每个文件名称一般有一定的规律，例如会包含时间戳、pid及inode节点号等。
除了mbox和maildir格式外，还有一个叫mbx的格式，它是对mbox的改进版本。主要用在UW-IMAP server里。它最大的特点是有一个针对mbox文件的索引，能改善读/写性能。但依然需要file lock。
随着qmail/postfix的普及，以及mbox的一些问题暴露，maildir得到了越来越多的应用。其中mbox的最主要问题是文件缩定（file lock），其次是大多数update操作的效能问题。
以下是引自courier-mta.org上的mbox和maildir的测试对比文章，介绍了mbox和maildir：
mbox mail storage format
This is the traditional way to store mail on UNIX-based mail servers.
Individual messages are simply concatenated together, and saved in a
single file. A special marker is placed where one message ends and the
next message begins. Only one process can access the mbox file in
read/write mode. Concurrent access requires a locking mechanism. Anytime
someone needs to update the mbox file, everyone else must wait for the
update to complete.maildir mail storage format
Maildirs were originally implemented in the Qmail mail server, supposedly
to address the inadequacies of mbox files. Individual messages are saved
in separate files, one file per message. There is a defined method for
naming each file. There's a defined procedure for adding new messages to
the maildir. No locking is required. Multiple processes can use maildirs
at the same time.mbx mail storage format
This is a slightly modified version of the original mbox format that's offered by the UW-IMAP server. mbx mailboxes still require locking. The
main difference from the mbox format is that each message in the file is
preceded by a record that carries some message-specific metadata. As
such, certain operations that used to require the entire mbox file to be
rewritten can now be implemented by updating the fixed-size header
record.This benchmark focuses mainly on the mbox and maildirs formats. In March of 2003 an unrelated party conducted a similar benchmark for mbx
formats. See http://www.decisionsoft.com/pdw/mailbench.html
for more details.
Mbox vs Maildir之优缺点比较
这里给出一个基本的特性对比，读者很容易就清楚根据自己的应用到底应该选什么存储格式：可靠性
优选是Maildir，因为mbox只有一个文件，一旦出问题之后，所有邮件都将损毁。
更新速度
这里主要指的是删除/增加邮件的能力，无疑Maildir完胜Mbox
搜索速度
这点Mbox因为是单文件，因此搜索的能力要强于maildir
并发访问能力
对于繁忙的邮件系统，多个进程同时访问同一封邮件是可能的事情，Mbox需要flock()的支持，而且如果某一个进程操作时间长，则其他所有进程都堵塞了。Maildir没有这个问题。在NFS等网络文件系统上，Maildir相对安全，Mbox不能用于此类型环境
扩充能力
现在的邮箱已经不是5年前甚至10年前的1MB,2MB而是100,200甚至1G/2G，Mbox应付那么大的容量已力不从心，无疑Maildir是比较适合的。
文件系统依赖
Maildir较依赖文件系统，尤其是依赖对目录的索引能力，用ReiserFS会比较快，对于超大型的maildir，读写性能将受到考验。相对而言Mbox则不存在这个问题。
综合结论
推荐使用Maildir格式，安全可靠，绝大部分操作都远快于Mbox。而且现今支持Maildir的软件越来越多，qmail/Postfix 都支持。一些有用的链接
原理标准
http://cr.yp.to/proto/maildir.html
http://www.qmail.org/qmail-manual-html/man5/mbox.html
Benchmark/比较
http://www.courier-mta.org/mbox-vs-maildir/
http://www.decisionsoft.com/pdw/mailbench.html
转换工具
http://www.qmail.org/mbox2maildir
http://batleth.sapienti-sat.org/projects/mb2md/
Posted by hzqbbc at 10:45 PM | Comments (2)
May 12, 2004
Postfix Ecartis HOWTO - 集成ecartis + Postfix
English version: 0.02alpha
Chinese caution: 本文是过去用英文写的，有兴趣的朋友可以翻译成中文，谢谢。
Introduce
There are a lot of maillist softwares compatible with the Postfix, but we need a simple configure and powerful one, Ecartis will meet our need.Ecartis is a certain kind of mailing list manager software, designing for small binary and fast operation, with modular archtecture. It support sendmail/Postfix Exim4 and qmail, very easy to configure, but it's bit regreat that ecartis lacks documentation :-(
Requirement
Before you begin your work, make sure the following softwares or configuration are done:Postfix (1.x/2.x, 2.x are better)
gcc and gcc-devel package
GNU make and other tools
root account
Unix like OS, linux is prefer
Make and build
Getting ecartis
Ecartis developed since 1998, but it's not 1.0 release ready. The latest version is 1.0.0-snap20040426, use wget or ncftp to get the tar ball from http://web.archive.org/web/20041010224951/ftp://ftp.ecartis.org/.
Complie ecartis
Unpack the tar ball with GNU tar, like this:
$ tar xfz ecartis-1.0.0-snap20040426.tar.gz
Then get into the src directory, there is no autoconf/automake style configure file ship with ecartis, only a Makefile.dist, copy the Makefile.dist to Makefile:
$ cp Makefile.dist Makefile
Make sure that gcc and make can work properly before you begin to complie, type make to complie:
$ make
Installation
If nothing exception occur, ecartis and modules complie finished, binary will in the build direcotry. Install ecartis when you want to use it:$ make install
The default location of ecartis installation will at the top of source directory, if you want to use ecartis in another location, copy them to the different path.
At the top of source directory, assume you want to place ecartis and modules in /home/ecartis, do as the following steps:
$ su root # useradd -d /home/ecartis -s /bin/false ecartis # cp ecartis /home/ecartis/ # cp ecartis.cfg.dist /home/ecartis/ecartis.cfg # cp ecartis.aliases.dist /home/ecartis/ecartis.aliases # cp -rf modules lists queue /home/ecartis/
Configuration - create a testlist
You should remember that use full path to run ecartis or things will be corupt! Please refer to ecartis documentation if you've some doubt.
# /home/ecartis/ecartis -newlist lists
This will create a list name "lists" for you, ecartis will prompt for list admin/owner, enter email address response for it. Then ecartis will create "lists" directory in the /home/ecartis/lists/lists and user/config file, at last ecartis display aliases content:
# Aliases for 'lists' mailing list. lists: "|/home/ecartis/ecartis -s lists" lists-request: "|/home/ecartis/ecartis -r lists" lists-repost: "|/home/ecartis/ecartis -a lists" lists-admins: "|/home/ecartis/ecartis -admins lists" lists-moderators: "|/home/ecartis/ecartis -moderators lists" lists-bounce: "|/home/ecartis/ecartis -bounce lists"
Configuration - Postfix
Use vi or emacs or other editor, open main.cf, paste the ecartis aliases content into the default aliases file, which commonly located at /etc/postfix/aliases, but i highly recommend that use a seperate aliases file, it's more clean and easy to manage:) my configuration shown as follow:
alias_maps = hash:/etc/postfix/aliases hash:/home/ecartis/ecartis.aliases mydestination = $myhostname, $mydomain mydomain = postfix.org.cn myhostname = lists.postfix.org.cn
Save main.cf and ecartis.aliases, then run postalias to rebuild ecartis.aliases, last reload postfix:
# postfix reload
Test lists
It's the time to test ecartis now. Send an email to [listname]-request@domain.tld as title or body contain "subscribe" to test subscribe function, in my case, send email to lists-request@lists.postfix.org.cn. Look at the maillog to get detail information.Almost immediately you will receive reply from ecartis, if nothing unexpected, you've been subscribed. The email will contains:
>>subscribe Subscribed.
Virtual Hosting
Ecartis can easily support virtual hosting, with postfix we need some tricks to get ecartis to work.
By some means virtual hosting equal to virtual domain, the Postfix MTA support virtual domain via alias, but virtual_alias_maps(2.0) or virtual_maps(1.0) could not support sendmail style aliases, then it means we can't simpliy put some lines like:
lists-request@lists.postfix.org.cn "| /home/ecartis/ecartis -r lists"
into those maps/files, how could we implement virtual domain? The trick is according to sendmail style aliases, the processing flow show as below:
mail to lists@domain.tld -> lookup virtual_alias_maps -> found lists@domain.tld point to "lists-domain.tld" -> lookup aliases maps -> found lists-domain.tld point to "|/home/ecartis/ecartis -s listname" -> OK
Assume we want to support thisdomain.tld and thatdomain.tld at the same machine, follow the steps:
1) DNS setup, point both the thisdomain.tld and thatdomain.tld's mx to the machine running ecartis
2) We only make thisdomain.tld as an example, add the following lines into the Postfix virtual_alias_maps file:
lists@thisdomain.tld lists-thisdomain.tld lists-request@thisdomain.tld lists-request-thisdomain.tld lists-repost@thisdomain.tld lists-repost@thisdomain.tld lists-admins@thisdomain.tld lists-admins-thisdomain.tld lists-moderators@thisdomain.tld lists-moderators-thisdomain.tld lists-bounce@thisdomain.tld lists-bounce-thisdomain.tld
Then run postmap to rebuild it
3)add the following lines into sendmail style aliases file:
lists-thisdomain.tld: "|/home/ecartis/ecartis -s lists-thisdomain.tld" lists-request-thisdomain.tld: "|/home/ecartis/ecartis -r lists-thisdomain.tld" lists-repost-thisdomain.tld: "|/home/ecartis/ecartis -a lists-thisdomain.tld" lists-admins-thisdomain.tld: "|/home/ecartis/ecartis -admins lists-thisdomain.tld" lists-moderators-thisdomain.tld: "|/home/ecartis/ecartis -moderators lists-thisdomain.tld" lists-bounce-thisdomain.tld: "|/home/ecartis/ecartis -bounce lists-thisdomain.tld"
Then run postalias to rebuild it
4)run this command:
#/home/ecartis/ecartis -newlist lists-thisdomain.tld
and enter the listowner address, then ecartis will create config and users file for the list "lists-thisdomain.tld" in /home/ecartis/lists direcotry.
5)configure /home/ecartis/lists/lists-thisdomain.tld/config to meet your need, refer to ecartis documentation or varible difinition for details.
If everything works ok, then we done it :-)
Trouble Shooting
When the first time you install and configure ecartis, there will be a lot of problem, see this section carefully.Q:ecartis can't work!
A:Turn on debug mode, in /path/to/ecartis/ecartis.cfg, uncomment "debug = 0" and set it large than 0, for example 10(the highest level), ecartis will log anything it do and see in ecartis.log, look at it for detail error.
Q:ecartis always send email as unkonw sender domain
A:check your system hostname, if it's not fqdn, ecartis will append the domain after the hostname, and make it as sender domain, if this name can't resolve, then recipient's MTA will bounce/reject the mail.
Q:How to config ecartis to confirm subscrition?
A: set "subscribe-mode = confirm" and save the config file, that's ok!
Reference
Docs of ecartis
ecartis.hlp in source
ecartis source file
ecartis mailing list
Postfix documentation
Posted by hzqbbc at 05:25 PM | Comments (0)
March 14, 2004
Postfix smtpd的性能瓶颈分析
smtpd performance ...(zz) ( 2004年3月14日9:25 )
List: postfix-users
Subject: Re: smtpd performance
From: Victor.Duchovni () morganstanley ! com
Date: 2004-03-11 2:23:25
Message-ID:
[Download message RAW]
On Wed, 10 Mar 2004, Cesar Bonadio wrote:
> The perl program delivers only one message per time so increasing the
> smtpd number
> did not help.
>
message submission latency is the sum of the "smtpd" and "cleanup"
latencies. The main source of latency in "smtpd" is DNS and table lookups
against LDAP, MySQL, or other "expensive" tables. The main source of
latency in "cleanup" is table lookups, body checks and disk I/O latency.
Time the response delay to each of the protocol interactions connect
(response is 220 banner), "HELO", "MAIL FROM:", "RCPT TO", "DATA", and
".", where is the delay?
--
Viktor.
Posted by hzqbbc at 05:57 AM | Comments (0)
November 14, 2003
Maildrop的若干常见问题集锦（FAQ）
为什么maildrop的quota不起作用？
请确认在编译maildrop时，增加了--enable-maildirquota 选项。此外，也要确定是否给用户增加了Quota的限制，检查用户$HOME/Maildir下的maildirsize文件，如果第一行没有类似：
10240000S,1024C
的信息的话，那说明Quota = unlimit（无限制）。
收件人只有一个时maildrop投递正常，但多个收件人时为什么只有第一个收到？
原因是：Postfix默认会一次性的给多个收件人发邮件，而maildrop每次只处理一个收件人，所以只有第一个收件人能收到邮件，其他的则会被maildrop不留痕迹的丢弃。
解决方法：如果maildrop配置成一个叫maildrop的transport，那么在Postfix的配置文件main.cf里增加一个配置：
maildrop_destination_recipient_limit = 1
这样迫使Postfix每次只处理一个收件人，使maildrop能正常工作。
如何使maildrop支持用户快超出Quota时自动报警？
要达到这个效果，必须使用maildrop 1.5.1以上版本，该版本之后的maildrop支持-w参数，设定警报的上限百分比，例如90%就是-w 90 。在Postfix的master.cf里配置内容可以这样写：
maildrop unix - n n - 50 pipe
flags=DRhu user=vuser argv=maildrop -w 90 -d ${user}@${nexthop}
${recipient} ${user} ${extension} ${nexthop}
同时，还必须在/etc/下放置quotawarnmsg文件，具体可参考maildrop说明文档。
Posted by hzqbbc at 12:33 PM | Comments (0)
January 25, 2003
qmail与Postfix的性能对比测试
一直以来，人们普遍的观点都认为qmail很快，比sendmail快得多，甚至有人还吹嘘说qmail比postfix也快，为了来一个客观一点的评价，对qmail/Postfix进行测试看来是必要的了，至少可以给邮件系统的选择提供一个评价的基准。
在测试postfix同样配置的另外一个机器上装了由hleil提供的rpm包，并咨询了一下他关于qmail的问题，在此表示感谢。
1.环境
2台配置完全一致的PIII 933*2 + 512M + U160 SCSI 磁盘，qmail-1.03-7。另外系统做了优化，调整了fs.file-max=65535并放开一般限制，将/var目录单独分开。
先使用smtp-sink测试qmail，使用系统帐号，效果是26封/秒，觉得效果不理想。
hleil表示，我做的smtp 测试只测试了smtpd的性能而已，队列并没受到真正考验，因为smtp的速度才是bottleneck。
2.简单结果：
用系统帐号，smtp-sink 配置成发1000封信，10个并发，耗费了56秒。换成100并发，测试失败，加大了qmail的并发数后完成顺利，结果也大概是26封/秒。
然后用postal软件，50个并发连接，每个连接发一封信，qmail配置到400个并发上限。结果就大概是1150-1200封/分钟的样子，怎么提高都不行了。看了一下top，发现kjounald频繁的出现，估计是写操作耗费了很多资源，并根据postfix测试结果推断可能是multilog耗费了很多资源的原因。于是去掉multilog记录的功能，qmail不使用log系统，再测试：
[root@ns2 postal-0.61]# ./do.sh
time,messages,data(K),errors,connections,SSL connections
14:52,2029,2767,0,2078,0
14:53,2056,2684,0,2056,0
14:54,2061,2785,0,2062,0
14:55,2022,2686,0,2021,0
14:56,1998,2684,0,1998,0
14:57,2006,2666,0,2007,0
14:58,1971,2644,0,1971,0
14:59,1971,2584,0,1970,0
15:00,1942,2596,0,1943,0
15:01,1984,2655,0,1984,0
15:02,1957,2567,0,1957,0
很明显速度提升1倍左右，但部分测试出现could not creat qq(队列）的问题，看不出具体的原因。估计是qmail没足够资源去创建队列，或者速度太快队列出了问题？
然后再将/etc/passwd的信息转换成cdb提高查询速度：
qmail-pw2u < /etc/passwd > /var/qmail/users/assign
qmail-newu
然后再测试，结果如下：
[root@ns2 postal-0.61]# ./do.sh
time,messages,data(K),errors,connections,SSL connections
16:42,2118,2858,0,2167,0
16:43,2095,2788,0,2096,0
16:44,2114,2790,0,2114,0
16:45,2245,3002,0,2244,0
16:46,2233,2971,0,2233,0
16:47,2165,2891,0,2166,0
16:48,1977,2637,0,1976,0
16:49,1999,2645,0,2000,0
注入速度似乎提高了5%-10%。
hleil对上述测试结果认为：这个测试是测试smtp的注入，并没办法真正考验队列，如上所发现的那样，队列应该没有受到特别的考验，即便是fail to create queue也可能是其他原因。必须是mx-mx真实环境下才有意义，而且一个smtp过程比现在的复杂。
注意：原文我记得不是太清晰，只是转述hleil说的内容。
当时我认为这样的测试起码还是有点效果的，首先就是测试了邮件dos情况。如果出现dos的时候，不管mta用何方法，只要支持住就可以了。如果qmail真的如我测试这样容易出现队列crash的话，是比较危险的。
但似乎这个问题不是那么严重，有那么多qmail用户，如果真的那么多DOS攻击，那么多qmail出问题，一定有解决办法的。
其次就是log的问题，去掉log后速度提高很多，可以说明必须将log的开销尽量降低，但一个产品系统必须有log。这个是必须解决的问题。
另外，由于qmail的队列设计，每封处理的邮件至少要在队列中建立3个文件，因此write的i/o操作频繁很多（top的时候就看到kjounald占用的cpu资源明显高于postfix的测试环境），可能会造成一定bottleneck（尤其是log不是async写的时候更明显，通过简单的调整为非同步写的话就能提高很多速度，至少注入快了很多）
从2003年这个测试结果说明了如下的问题：
1.这个结果比较的只是smtp注入速度
2.这个结果主要是体现了若干MTA的I/O消耗及高流量下的并发处理能力
3.从结果可推断，postfix的fsync()操作最少，而qmail的fsync()较多，从体系结构及设计来说，qmail需要的是一个能立刻将数据同步(sync)到磁盘的文件系统，如果是使用了softupdates或async则在系统崩溃或掉电时，造成丢信。
也就是说，在bottleneck上，同样的文件系统及磁盘子系统，postfix使用更少资源，如果cpu足够快，能够忽略处理队列耗费cpu的时间的话，那么postfix的队列应该比qmail快而稳定。
最后，测试过程中qmail的队列出现过crash，而postfix从未发生过。
参考资料：
一个专门做测试的网站
Serverwatch
Postfix vs qmail
Posted by hzqbbc at 08:22 AM | Comments (1)
January 18, 2003
D. J. Bernstein 和Wietse Venema有关qmail的争论
两人高手过招，一个mta所需要做的工作不多，也不至于什么都干吧??限制smtp log大小是log的事.我觉得。当然,反过来看，mta如果能自己限制，也不错。类似qmail的配套工具一样。
以下是wietse的邮件，关于djb的写的slander回应.
http://groups.google.com/groups?q=artificial+limit+postfix+log+group:mailing.postfix.users&hl=zh-CN&lr=&ie=UTF-8&inlang=zh-CN&selm=9t130n%2412ha%241%40FreeBSD.csie.NCTU.edu.tw&rnum=1
DJB对venema的说法
http://cr.yp.to/qmail/venema.html
另外一个maillist上的邮件，包含了一个简单的patch
http://lists.q-linux.com/pipermail/plug-misc/2001-November/000963.html
Posted by hzqbbc at 07:18 PM | Comments (0)
在足够好的硬件条件下Postfix比qmail更快的原因分析
经过实际的测试我也发现Postfix比qmail快（在较平等的条件下比较）。究其原因，主要是因为磁盘I/O 的差异，Postfix的磁盘I/O原则上比qmail少耗用资源，仅1/3左右，所以速度原则上应该快3倍。
以下是wietse的解释。
以下是postfix作者的原话，就偶自己的小知识，对硬盘写操作越多性能越低。it's true..
===================================================================
寄件者：Wietse Venema (wietse@porcupine.org)
主旨：Re: performance tuning
View this article only
新闻群组：mailing.postfix.users
日期：2001-02-15 17:34:07 PST
jet:
> I'm trying to dump e-mails into postfix via smtp, but can only achieve
> speeds of about 15-20 mails/second (locally).
>
> I've tried spawning multiple outgoing-mail scripts, and tweaked with them,
> but always end up in the 15-20 mail/second range.
This is pretty much single disk Postfix performance. Most other
mailers are much, much, worse than this.
The bottle neck is your disk.
When I first tested Postfix against qmail, Postfix was 3x faster
on local and LAN benchmarks because Postfix uses one queue file
where qmail uses three. The create/delete operations are much,
much, more time consuming than reading and writing the content.
In order to make Postfix fast you need to avoid queue file
create/delete operations. Multiple recipients per queue file are
a big win. With one recipient per queue file Postfix is starved
because the disk is too slow.
Postfix does mostly random writes, and it actually has to wait
until a file is safe on disk. It's not allowed to wait until dirty
blocks are synced automatically.
In order to process more mails per unit time, you need faster disks.
Well, that is not possible.
You can, however, spread the load over multiple spindles. Don't
use RAID5 because it still has single-disk performance for applications
that produce random writes like Postfix does.
Another possibility is to use non-volatile RAM of the type that
used to be sold for SUN file servers. It speeds up disk access of
random writes by an order of magnitude, because writes can be sorted
for speed without loss of reliability. Next to solid-state disks
it is the best thing you can do.
Wietse
--
>My test machine is a:
> AMD Athlon Thunderbird 800 mHz
> 128 megs memory
not enough memory for sending 50K msgs/hour. You'll need 512 or more
to get 400+ SMTP processes into memory.
You will also have to increase maxfiles and maxfilesperprocess with sysctl.
I have a client with a listar announcement list machine hitting 40
msgs/hour on a P500 + 512 megs RAM and just one IDE disk. I think 50
is reachable, but 40 is more than enough for him.
> UDMA66 IDE drive
Better to have two IDE master drives on two separate IDE channels,
one for system + logging and the other for mailq, and maybe even a
third for mailbox storage, but that means master/slave disk on IDE
which is slower.
Of course, IDE is slower than SCSI, and SCSI cards with 64 or 128
megs on-board buffer would help the disk channel congestion, which
Wietse says is the limit to an MTA.
好象wietse说不要用raid5哦。。因为也是random write，这样的效能低。确实也是。因为如果是连续的有规律的写，速度会块得多的。。
Posted by hzqbbc at 01:50 PM | Comments (0)
Postfix+ldap的性能测试及分析
ChangeLog
·2005-04-16 进行内容整理 ·2005-04-10 搜集到blog的Postfix专栏 ·2003-01-18 第一版 发表于http://www.hzqbbc.com/forum/
1.测试平台
PIII 933 × 2 + 512M + SCSI 18G
Postfix 2.0.0.1 + openldap 2.0.11(默认安装) + virtual 投递
2.使用HASH格式作为查询表（lookukp table）时,SMTP 注入速度大概都有5000多封信/分钟。最高能达到87.5封/s。
这个时候采用的方式为只测试信头,100个并发连接。将postfix的默认进程限制提升到500，即：default_process_limit = 500，这也加大了并发速度。其中对syslogd采用了异步写优化(在设置的目录前加-号) 也能大幅度提高速度。
mail.* -/var/log/maillog
3.将用户信息保存在ldap里，SMTP注入速度基本都有2500-2800/分钟,再简化了lookup的方式，transport_maps采用固定文本的方式,只有virtual_mailbox_maps才用ldap，这样速度提高到3200-3500封/分钟左右。测试recipients地址固定为10个
4.在ldap里存放50万条用户记录。进行500,1000,2000地址的并发测试，结果发现SMTP注入速度和使用的用户地址多少关系不大，以下是2000地址的结果:
Postal result: time,msg/m,KB,error,connection,SSL 10:26,3779,1516,2,1930,0 10:27,3889,1556,0,1953,0 10:28,3987,1592,0,1997,0 10:29,4174,1667,0,2067,0 10:30,4308,1720,0,2166,0 10:31,4039,1613,0,1986,0 10:32,2660,1061,0,1338,0 10:33,2436,972,0,1216,0
SMTP注入速度比较快，这里信件内容只包含了信头，采用50个并发连接，每个连接只发2-3封信。感觉效果还是不错的。不过10:32之后速度开始有点问题了，我检查了/var/spool/postfix目录，发现incoming的数量大幅提高，估计是cleanup不能那么快处理incoming的信件，使队列堆积起来；但也可能是ldap反应速度开始变慢。
小结：性能优化技巧
·关闭ldap的log使之=0·将maillog等需要频繁进行write/sync的日志设置为async write，方法见上文
·调整postfix的default_process_limit > 150
·调整/var/spool/postfix目录的属性为async，使用chattr命令
5. 50个并发2000地址，邮件只包含信头，每个连接3封信的结果：
11:17,3613,1447,0,1849,0 11:18,3876,1547,0,1930,0 11:19,4088,1634,0,2080,0 11:20,3920,1566,0,1977,0 11:21,4057,1621,0,2021,0 11:22,3946,1573,0,1960,0 11:23,3316,1324,0,1695,0 11:24,2576,1030,0,1295,0 11:25,2889,1155,0,1439,0 ^^^^^^^^^^^^^^^^^^^^^^* 11:26,3984,1596,0,1990,0 11:27,3986,1588,0,1993,0 11:28,4164,1662,0,2064,0 11:29,3973,1591,0,1986,0 11:30,3991,1592,0,1982,0 11:31,3521,1406,0,1758,0 11:32,2590,1035,0,1297,0 ^^^^^^^^^^^^^^^^^^^^^^* 11:33,2655,1058,0,1322,0 11:34,2745,1097,0,1348,0
注意^^*所标记的时间段，SMTP注入性能大幅度下降。检查/var/spool/postfix目录，发现qmgr已达到处理上限了，该上限由qmgr_message_active_limit 参数设置。故此incoming开始增长很多，Postfix开始放慢处理速度。
以上所有测试都是在default_destination_concurrency_limit = 10的条件下完成的，所以在virtual进行本地mailbox投递时，并发数很少，而active目录增长得很快，没有足够多的virtual进行投递，结果是无法快速地将邮件写到硬盘。
6.提升default_destination_concurrency_limit：
default_destination_concurrency_limit = 1000
此时出现另外一个问题：active队列大概只维持在200-300百封信的样子，但defer/defered目录开始有所增长，并且SMTP注入速度下降，大概只有1850封信/分钟的速度。此时virtual的并发数目达到了400个！
可以推断，这是因为virtual投递速度和数量大幅度增加，致使磁盘写(write)操作过于频繁，使磁盘I/O性能吃紧，导致性能下降。
解决办法：使用分布式存储，利用mailswitch这一类技术，将最终的邮件投递由存储机器上的软件完成。
将这个default_destination_concurrency_limit改成100，速度提升了一些，但很多出现了很多lookup failure：
time,messages,data(K),errors,connections,SSL connections
Server error:451 <user10798@bigmail.hzqbbc.com>: Temporary lookup failure
.
Server error:451 <user10700@bigmail.hzqbbc.com>: Temporary lookup failure
.
11:57,2773,1124,2,1459,0
11:58,2302,920,0,1156,0
11:59,2492,996,0,1264,0
12:00,2204,879,0,1087,0
Server error:451 <user10380@bigmail.hzqbbc.com>: Temporary lookup failure
.
Server error:451 <user11876@bigmail.hzqbbc.com>: Temporary lookup failure
.
12:01,2311,924,2,1146,0
Server error:451 <user10559@bigmail.hzqbbc.com>: Temporary lookup failure
.
12:02,2487,993,1,1265,0
12:03,2284,912,0,1112,0
12:04,1515,602,81,808,0
^^ 81个错误
12:05,1969,787,11,1012,0
12:06,2139,858,0,1073,0
12:07,2174,868,0,1095,0
12:08,2214,884,0,1093,0
12:09,2252,900,0,1118,0
Server error:451 <user11171@bigmail.hzqbbc.com>: Temporary lookup failure
并发降至50：
default_destination_concurrency_limit = 50
测试的log：
Server error:451 <user10062@bigmail.hzqbbc.com>: Temporary lookup failure . 12:21,2645,1064,2,1379,0 12:22,3033,1211,0,1503,0 12:23,2996,1206,0,1472,0 Server error:451 <user10413@bigmail.hzqbbc.com>: Temporary lookup failure . 12:24,2911,1152,1,1455,0 12:25,2716,1093,0,1359,0 12:26,2844,1131,0,1434,0 12:27,2748,1096,0,1360,0 12:28,2738,1107,0,1396,0 12:29,2652,1051,0,1352,0 12:30,2759,1108,0,1355,0 Server error:451 <user11773@bigmail.hzqbbc.com>: Temporary lookup failure . Server error:451 <user11684@bigmail.hzqbbc.com>: Temporary lookup failure . Server error:451 <user10859@bigmail.hzqbbc.com>: Temporary lookup failure . Server error:451 <user11567@bigmail.hzqbbc.com>: Temporary lookup failure . Server error:451 <user11039@bigmail.hzqbbc.com>: Temporary lookup failure . Server error:451 <user11336@bigmail.hzqbbc.com>: Temporary lookup failure . Server error:451 <user10781@bigmail.hzqbbc.com>: Temporary lookup failure . 12:31,2472,987,13,1253,0
Postal测试时打开100个并发,每个邮件最大10k,每个连接1封,发5000封，产生了非常多的failure，不过速度还不错。
接下来Postal 使用50并发，每并发1个邮件，打开了openLDAP cache支持：
time,messages,data(K),errors,connections,SSL connections
23:03,3540,19196,46,3685,0
23:04,4246,23136,0,4247,0
23:05,4182,22873,19,4194,0
^^19个错误
23:06,3736,19922,74,3808,0
^^ 70多个错误!
23:33,3683,3359,0,3733,0
23:34,3935,3632,0,3935,0
23:35,4041,3664,0,4041,0
23:36,3990,3636,0,3989,0
23:37,4164,3798,0,4165,0
23:38,3872,3512,0,3872,0
23:39,3210,2945,0,3209,0
23:40,2624,2400,0,2625,0
Server error:451 <user10717@bigmail.hzqbbc.com>: Temporary lookup failure
.
Server error:451 <user10614@bigmail.hzqbbc.com>: Temporary lookup failure
.
Server error:451 <user10634@bigmail.hzqbbc.com>: Temporary lookup failure
.
23:41,2502,2286,3,2505,0
23:42,2760,2508,0,2759,0
Server error:451 <user11449@bigmail.hzqbbc.com>: Temporary lookup failure
.
Server error:451 <user11660@bigmail.hzqbbc.com>: Temporary lookup failure
.
23:43,2528,2305,2,2531,0
23:44,2078,1972,0,2078,0
Server error:451 <user10298@bigmail.hzqbbc.com>: Temporary lookup failure
.
Server error:451 <user10706@bigmail.hzqbbc.com>: Temporary lookup failure
.
Server error:451 <user11515@bigmail.hzqbbc.com>: Temporary lookup failure
.
Server error:451 <user10499@bigmail.hzqbbc.com>: Temporary lookup failure
.
23:45,1520,1387,4,1524,0
23:46,1715,1536,0,1715,0
23:47,1475,1362,0,1475,0
23:48,1508,1371,0,1507,0
可见active目录的上限已经达到了，所以SMTP注入速度开始下降，邮件已经开始处理不过来了。
调整Postal 参数，使用50并发，每个并发1个msg，尽最大能力发送，打开openLDAP cache模式，qmgr_messages_active_limit=50000，测试持续了近一个小时，日志如下：
[root@ns1 postal-0.61]# ./do.sh time,messages,data(K),errors,connections,SSL connections Server error:451 <user11017@bigmail.hzqbbc.com>: Temporary lookup failure . Server error:451 <user10714@bigmail.hzqbbc.com>: Temporary lookup failure . Server error:451 <user10989@bigmail.hzqbbc.com>: Temporary lookup failure . 23:52,3663,3353,3,3714,0 23:53,3784,3454,0,3786,0 23:54,3747,3409,0,3747,0 23:55,3861,3498,0,3861,0 23:56,3922,3577,0,3922,0 23:57,3822,3491,0,3822,0 23:58,3748,3429,0,3747,0 23:59,3608,3264,0,3609,0 00:00,3320,3027,0,3319,0 00:01,3410,3117,0,3411,0 00:02,3317,3042,0,3316,0 00:03,3580,3286,0,3581,0 00:04,3507,3205,0,3507,0 00:05,3475,3114,0,3475,0 00:06,3259,2995,0,3258,0 00:07,3186,2912,0,3186,0 00:08,3552,3216,0,3553,0 00:09,2887,2635,0,2887,0 00:10,2517,2292,0,2517,0 00:11,2721,2505,0,2721,0 00:12,2574,2317,0,2574,0 00:13,2587,2357,0,2587,0 00:14,2573,2342,0,2573,0 00:15,2704,2463,0,2704,0 00:16,2647,2405,0,2647,0 00:17,2585,2383,0,2585,0 00:18,2421,2235,0,2421,0 00:19,2573,2349,0,2572,0 00:20,2378,2170,0,2378,0 Server error:451 <user11351@bigmail.hzqbbc.com>: Temporary lookup failure . Server error:451 <user11458@bigmail.hzqbbc.com>: Temporary lookup failure . ...... 期间Temporary lookup failure有12个 00:21,2330,2144,15,2346,0 00:22,2610,2376,0,2610,0 00:23,2551,2317,0,2550,0 00:24,2586,2380,0,2587,0 00:25,2530,2282,0,2528,0 00:26,2558,2306,0,2560,0 00:27,2444,2232,0,2444,0 00:28,2509,2250,0,2508,0 00:29,2523,2293,0,2524,0 00:30,2445,2210,0,2445,0 00:31,2555,2345,0,2554,0 00:32,2455,2225,0,2456,0 00:33,2477,2266,0,2476,0 Server error:451 <user10490@bigmail.hzqbbc.com>: Temporary lookup failure . Server error:451 <user10822@bigmail.hzqbbc.com>: Temporary lookup failure . 00:34,2236,2038,2,2238,0 00:35,2245,2061,0,2245,0 00:36,2536,2311,0,2537,0 00:37,2210,2004,0,2210,0 00:38,2385,2150,0,2385,0 00:39,2463,2242,0,2463,0 00:40,2496,2291,0,2495,0 00:41,2471,2261,0,2471,0 00:42,2422,2208,0,2422,0 00:43,2370,2159,0,2371,0 00:44,2387,2197,0,2386,0 00:45,2203,2028,0,2203,0 00:46,2494,2301,0,2494,0 00:47,2313,2133,0,2314,0 00:48,2255,2016,0,2255,0 00:49,2428,2236,0,2428,0 00:50,2327,2113,0,2326,0 00:51,2436,2244,0,2437,0 00:52,2334,2127,0,2334,0 00:53,2399,2202,0,2398,0 00:54,2271,2078,0,2272,0 00:55,2255,2046,0,2255,0 00:56,2397,2205,0,2397,0 00:57,2392,2183,0,2392,0 00:58,2364,2161,0,2364,0 00:59,2241,2031,0,2240,0 01:00,2354,2137,0,2355,0 01:01,2223,2000,0,2223,0 01:02,2326,2118,0,2325,0 01:03,2343,2155,0,2344,0 01:04,2388,2209,0,2388,0
队列里积压了13万封信，70多分钟内注入了22万封信，正确投递并写入用户$HOME目录的有95000多封，由于人为限制了投递的并发数，因此这个结果也在意料之内了。
[root@ns2 postfix]# postsuper -d ALL postsuper: Deleted: 137410 messages [root@ns2 postfix]# du -sk /home/domains/bigmail.hzqbbc.com/ 392124 /home/domains/bigmail.hzqbbc.com [root@ns2 postfix]# find /home/domains/bigmail.hzqbbc.com/ | wc -l 95349
6. 新一轮测试
Postal 使用50/100个并发，1-3封邮件/每连接，将Postfix升级到Snapshot 2.0.2，并打开proxymap daemon，做如下的测试：
[root@ns1-bjcnc postal-0.61]# ./do.sh time,messages,data(K),errors,connections,SSL connections Server error:451 <user11097@bigmail.hzqbbc.com>: Temporary lookup failure . Server error:451 <user10170@bigmail.hzqbbc.com>: Temporary lookup failure . Server error:451 <user11333@bigmail.hzqbbc.com>: Temporary lookup failure . 14:02,3743,3366,3,1880,0 14:03,4244,3809,0,2129,0 14:04,4345,3915,0,2204,0 14:05,4187,3790,0,2114,0
[root@ns1-bjcnc postal-0.61]# ./do.sh time,messages,data(K),errors,connections,SSL connections 14:10,4380,3973,5,2289,0 14:11,4481,4065,0,2215,0 14:12,4541,4083,0,2260,0 . 14:13,2741,2455,14,1389,0 14:14,3927,3549,0,1949,0 14:15,3611,3235,0,1807,0 14:16,3634,3304,0,1796,0 14:17,3889,3475,0,1933,0
[root@ns1-bjcnc postal-0.61]# ./do.sh time,messages,data(K),errors,connections,SSL connections 14:22,1838,1644,0,961,0 Server error:451 <user11857@bigmail.hzqbbc.com>: Temporary lookup failure . Server error:451 <user11434@bigmail.hzqbbc.com>: Temporary lookup failure . 14:23,1910,1721,2,959,0 Server error:451 <user10268@bigmail.hzqbbc.com>: Temporary lookup failure . Server error:451 <user11688@bigmail.hzqbbc.com>: Temporary lookup failure . 14:24,1989,1769,2,1007,0 Server error:451 <user11264@bigmail.hzqbbc.com>: Temporary lookup failure . Server error:451 <user11066@bigmail.hzqbbc.com>: Temporary lookup failure . 14:25,1828,1652,2,915,0
[root@ns1-bjcnc postal-0.61]# ./do.sh Can't open config file "conver". Doing no expansion. time,messages,data(K),errors,connections,SSL connections Server error:451 <user11182@bigmail.hzqbbc.com>: Temporary lookup failure . 14:27,4110,3721,1,2105,0 14:28,289,221,0,149,0
[root@ns1-bjcnc postal-0.61]# cat do.sh
#!/bin/sh MAXMSGSIZE=1 PROC=50 MSGPERCONN=3 MSGPERMIN=5000 SSLPERCENT=0 SMTP_HOST=210.82.193.91 ./postal -m $MAXMSGSIZE -p $PROC -c $MSGPERCONN -r $MSGPERMIN -a \ -b netscape $SMTP_HOST myuser conver
[root@ns1-bjcnc postal-0.61]# ./do.sh time,messages,data(K),errors,connections,SSL connections . 14:38,4213,3793,10,2176,0 14:39,4308,3869,0,2174,0 14:40,4296,3835,0,2160,0 14:41,4261,3850,0,2136,0 14:42,4342,3930,0,2168,0 14:43,4258,3843,0,2139,0 14:44,4172,3769,0,2067,0 14:45,4111,3684,0,2052,0 14:46,3920,3577,0,1961,0 14:47,3987,3590,0,1987,0 14:48,3729,3365,0,1851,0 14:49,3702,3318,0,1842,0 14:50,3745,3371,0,1879,0 14:51,3961,3558,0,1969,0 14:52,3753,3415,0,1876,0 14:53,3498,3139,0,1781,0 14:54,3337,3001,0,1661,0 14:55,2671,2398,0,1341,0 14:56,2722,2453,0,1365,0 14:57,2648,2388,0,1342,0 14:58,2466,2199,7,1227,0 14:59,2601,2344,0,1299,0
通过上述多次测试表明，在打开proxymap daemon后，SMTP注入速度明显提高，结果是提升了10%-15%。
接下来将virtual投递并发限制
[root@ns1-bjcnc postal-0.61]# ./do.sh
time,messages,data(K),errors,connections,SSL connections
20:44,4289,3902,0,2166,0
20:45,4329,3874,0,2169,0
Server error:451
.
期间产生12个Temporary lookup failure
20:46,3993,3594,13,1983,0
20:47,4422,4018,0,2216,0
20:48,4639,4163,0,2323,0
以上部分是final_destination 限制在20个并发virtual
========================================
接着改成10个，速度提升。说明bottleneck仍然在disk i/o（投递多了就明显影响i/o）ldap仍然有潜力可挖掘。
[root@ns1-bjcnc postal-0.61]# ./do.sh
Can't open config file "conver". Doing no expansion.
time,messages,data(K),errors,connections,SSL connections
20:53,5172,4690,0,2636,0
20:54,5074,4530,0,2529,0
20:55,4988,4529,0,2494,0
20:56,4856,4376,0,2456,0
20:57,5114,4602,0,2530,0
20:58,4720,4251,0,2345,0
20:59,4490,4038,0,2230,0
21:00,4716,4234,0,2363,0
21:01,4355,3931,0,2188,0
21:02,4488,4046,0,2241,0
21:03,4495,4062,0,2271,0
21:04,4680,4224,0,2363,0
21:05,4295,3882,0,2141,0
21:06,3822,3436,0,1921,0
21:07,2899,2597,0,1425,0
21:08,2863,2587,0,1447,0
完整结果如下：
[root@ns1-bjcnc postal-0.61]# ./do.sh
Can't open config file "conver". Doing no expansion.
time,messages,data(K),errors,connections,SSL connections
20:53,5172,4690,0,2636,0
20:54,5074,4530,0,2529,0
20:55,4988,4529,0,2494,0
20:56,4856,4376,0,2456,0
20:57,5114,4602,0,2530,0
20:58,4720,4251,0,2345,0
20:59,4490,4038,0,2230,0
21:00,4716,4234,0,2363,0
21:01,4355,3931,0,2188,0
21:02,4488,4046,0,2241,0
21:03,4495,4062,0,2271,0
21:04,4680,4224,0,2363,0
21:05,4295,3882,0,2141,0
21:06,3822,3436,0,1921,0
21:07,2899,2597,0,1425,0
21:08,2863,2587,0,1447,0
21:09,2887,2600,0,1473,0
21:10,2772,2494,0,1372,0
21:11,2106,1896,0,1033,0
21:12,1188,1079,0,584,0
21:13,1493,1355,0,742,0
21:14,1696,1540,0,828,0
21:15,1860,1677,0,920,0
21:16,1581,1428,0,784,0
21:17,2022,1837,0,1018,0
21:18,1477,1317,0,735,0
21:19,1511,1375,0,760,0
21:20,1877,1691,0,950,0
21:21,1459,1314,0,725,0
21:22,1494,1321,0,740,0
21:23,1526,1386,0,756,0
21:24,1309,1188,0,643,0
21:25,1446,1288,0,751,0
21:26,1396,1252,0,722,0
21:27,1431,1260,0,713,0
21:28,1312,1199,0,663,0
21:29,1290,1149,0,658,0
21:30,1190,1078,0,608,0
21:31,949,864,0,464,0
21:32,969,875,0,484,0
21:33,1210,1095,0,625,0
21:34,1384,1255,0,681,0
21:35,1332,1207,0,645,0
21:36,1018,928,0,503,0
21:37,882,795,0,442,0
21:38,1222,1095,0,613,0
21:39,1171,1051,0,581,0
21:40,1222,1123,0,609,0
21:41,1223,1088,0,606,0
21:42,1220,1086,0,607,0
21:43,1328,1208,0,665,0
21:44,1409,1263,0,719,0
21:45,1488,1349,0,723,0
21:46,1237,1115,0,628,0
21:47,1160,1035,0,586,0
21:48,1270,1146,0,626,0
21:49,1400,1261,0,708,0
到了21:10开始，actvie已接近上限，处理速度开始来不及了。此外，postal里限制了一分钟只发5000封信，所以本次测试的峰值51xx信/分未必可信，应能提高更多。
提高注入速度的新测试
笔者根据在IBM developerWorks上得到的灵感，将Postfix queue的所在的磁盘分区挂接参数(mount)改为noatime，这样实际减少了磁盘的写入操作（noatime表示对于访问操作，不更新文件的atime记录，这可减少磁盘的write操作），提高了5%的性能。通过elvtune调整了磁盘write操作的延时，可明显提高速度。
注意使用linux的iostat及相关的I/O监视工具。。以下是调整后的数值：
time,messages,data(K),errors,connections,SSL connections 23:57,5325,4804,0,2703,0 23:58,5480,4905,0,2702,0 23:59,5160,4624,0,2573,0 00:00,5211,4689,0,2609,0 00:01,4918,4441,0,2453,0 00:02,4872,4377,0,2395,0 00:03,5039,4564,0,2492,0 00:04,4694,4223,0,2308,0
效果有了明显进展。而再进行多次测试发现，有些时段可达到峰值63xx封/分，估计这因为是Postfix的Multiplex技术及磁盘I/O的cache所带来的收益，multiplex可复用已打开的smtp通道，减少了进程的创建带来的开销。
[root@ns1-bjcnc postal-0.61]# ./do.sh time,messages,data(K),errors,connections,SSL connections 22:23,6330,5721,0,3213,0 22:24,5592,5038,0,2783,0 22:25,5069,4555,0,2509,0 22:26,4624,4203,0,2333,0 22:27,4778,4311,0,2438,0 22:28,4396,3943,0,2215,0
[root@ns1-bjcnc postal-0.61]# ./do.sh Can't open config file "conver". Doing no expansion. time,messages,data(K),errors,connections,SSL connections 22:32,5121,4615,0,2611,0 22:33,4696,4263,0,2352,0 22:34,4532,4094,0,2277,0 22:35,4622,4138,0,2297,0 22:36,5006,4515,0,2493,0 22:37,4487,4052,0,2284,0 22:38,4595,4121,0,2300,0 22:39,4669,4221,0,2300,0 22:40,4381,3961,0,2149,0 22:41,4217,3781,0,2100,0
[root@ns1-bjcnc postal-0.61]# ./do.sh time,messages,data(K),errors,connections,SSL connections 22:44,5148,4642,0,2629,0 22:45,4868,4390,0,2414,0 22:46,4890,4392,0,2451,0 22:47,4803,4306,0,2419,0 22:48,5005,4552,0,2493,0 22:49,4435,3976,0,2221,0 22:50,4515,4055,0,2246,0 22:51,4666,4187,0,2336,0 22:52,4556,4105,0,2283,0 22:53,4472,3979,0,2239,0 22:54,4136,3749,0,2057,0 22:55,4649,4172,0,2355,0 22:56,4304,3895,0,2142,0 22:57,4055,3607,0,2008,0 22:58,2692,2425,0,1335,0 22:59,2690,2402,0,1346,0 23:00,2796,2533,0,1379,0 23:01,2738,2448,0,1396,0 23:02,2522,2287,0,1252,0 23:03,2574,2337,0,1286,0
[root@ns1-bjcnc postal-0.61]# ./do.sh time,messages,data(K),errors,connections,SSL connections 23:08,6084,5525,0,3068,0 23:09,5834,5254,0,2912,0 23:10,4683,4205,0,2303,0 23:11,4859,4376,0,2453,0 23:12,4265,3814,0,2142,0 23:13,4595,4179,0,2282,0 23:14,4466,4002,0,2217,0 23:15,4438,3982,0,2201,0 23:16,4463,4033,0,2242,0 23:17,4592,4123,0,2327,0 23:18,4234,3802,0,2097,0 23:19,4545,4049,0,2265,0 23:20,3853,3437,0,1925,0 23:21,2708,2417,0,1349,0 23:22,2730,2452,0,1363,0
[root@ns1-bjcnc postal-0.61]# ./do.sh Can't open config file "conver". Doing no expansion. time,messages,data(K),errors,connections,SSL connections 23:42,5327,4852,0,2721,0 23:43,5201,4653,0,2631,0 23:44,5377,4815,0,2705,0 23:45,4899,4431,0,2398,0 23:46,4863,4378,0,2421,0 23:47,4885,4382,0,2460,0 23:48,5012,4523,0,2496,0 23:49,5038,4501,0,2561,0 23:50,4898,4427,0,2431,0 23:51,4888,4400,0,2494,0 23:52,4856,4350,0,2398,0 Server error:451 <user10945@bigmail.hzqbbc.com>: Temporary lookup failure . Server error:451 <user11208@bigmail.hzqbbc.com>: Temporary lookup failure . Server error:451 <user10766@bigmail.hzqbbc.com>: Temporary lookup failure . Server error:451 <user11461@bigmail.hzqbbc.com>: Temporary lookup failure . 23:53,4584,4115,4,2306,0 23:54,4693,4243,0,2336,0
接下来，将保存对列的磁盘sda3进行调整:
(queue) r_d w_d from 4096/8192 -> 4096/2048
并清理队列，重新测试，注入速度快了不少。
没调整磁盘R/W延迟之前的结果（r/w -> 8192/4096) ：
time,messages,data(K),errors,connections,SSL connections 00:08,5280,4790,0,2704,0 00:09,5135,4570,0,2546,0 00:10,5280,4810,0,2635,0 00:11,5082,4572,0,2547,0 00:12,5195,4648,0,2596,0 00:13,5053,4591,0,2549,0 00:14,5089,4568,0,2583,0 00:15,4937,4438,0,2485,0 00:16,4727,4312,0,2340,0 00:17,4396,3958,0,2224,0 00:18,4860,4353,0,2413,0 00:19,4451,3994,0,2227,0 00:20,4582,4132,0,2300,0 00:21,2809,2486,0,1439,0
调整为r/w -> 4096/2048，结果连续出现了6次6xxx/分:
[root@ns1-bjcnc postal-0.61]# ./do.sh time,messages,data(K),errors,connections,SSL connections 00:23,6247,5597,0,3174,0 00:24,6581,5891,0,3280,0 00:25,6544,5898,0,3236,0 00:26,6457,5737,0,3235,0 00:27,6283,5690,0,3137,0
最后结论
LDAP的速度不容质疑，在硬件足够好的前提下，系统的瓶颈一般在磁盘的I/O上，LDAP一般较磁盘I/O要快。而如何调整Postfix的注入并发数、系统总进程数、投递的并发数等，需要大量的测试，找到在具体平台、具体硬件下所能达到的最高效能，继而定出合理的数值。作为一个繁忙的邮件系统，磁盘的I/O能力，尤其是队列所在的磁盘I/O能力要求非常高。在不增加硬件投入的前提下，善于利用iostat, vmstat及相关I/O分析工具，可以找到磁盘的限制在哪里，利用elvtune调整磁盘的读/写时延(delay)来减少这种限制所带来的性能劣化。
如果有足够的经济实力，可以使用带NVRAM的磁盘系统，使用Ultra 320的磁盘，甚至组成RAID0+1的系统来构成队列。这样可有效的提高速度。有报道称，国外有公司使用Sun的存储设备，在Solaris下实现了sendmail 287封邮件/秒 的处理速度。换言之相当于17220封/分的速度。这得益于它们使用了Sun的高级存储设备，配备了NVRAM的Cache，非常高速的硬盘，相当好的性能调整和优化。
通过这个国外的实例可以说明一点，Sendmail其实性能也可以相当好，关键是看系统的设计和部署人员的能力和经验。
而本文所实现的结果，也反映了Postfix在低端硬件（U160 SCSI磁盘，PIII 933的普通CPU）下效能相当不俗。如果按正常的4000封/分钟的处理速度，那么一天相当于处理576万封普通大小的电子邮件。
原文地址：http://www.hzqbbc.com/forum/read.php?forumid=2&filename=f_73
Posted by hzqbbc at 10:56 AM | Comments (0)
January 01, 2003
配置per user transport实现用户分布存储
高级Postfix配置
大容量邮件系统的关键之一就在于存储，而存储无外乎集中存储和分布存储。
O 集中存储主要是通过光纤磁盘阵列，SAN或iSCSI等统一存储技术，将邮件数据集中或较为集中的存储在一个地方，应用系统共享使用这些数据，所带来的问题主要是如何解决同时请求一个文件的缩定问题
O 分布存储则将用户数据分布在多个存储设备/机器上，主要的手段是通过一定的算法，将邮件分发到存储的设备或主机上。原理上有点象路由器，邮件被路由到了最终的目的地。所带来的问题是管理比较麻烦，但较少共享缩定的问题。
如何利用Postfix实现分布的用户数据存储呢？这里通过per user transport来实现。
在postfix没有release 2.0.0之前这个per user transport的功能只出现在snapshot里，现在好了可以在正式版里使用了。本文所描述的方法适合于：postfix stable > 2.0.0.0或snapshot version > 1.1.11-200205xx
1.描述
什么是per user transport？有什么用？简单来说就是可以为每个用户设置单独的transport，类似原来1.1.x里的domain transport一样，只不过现在的key可以是user+extension@domain.tld了。好处是什么？好处就是可针对不同用户设置特定的transport，例如有些用户使用默认的local transport，而有些则可以使用病毒过滤或maildrop功能，甚至将用户送到nexthop继续进行处理等。可能的应用：病毒过滤、关键字过滤、垃圾邮件过滤、邮件的物理分布或转发等。
自己的感觉：这个功能就类似qmail里的qmqp（中央队列服务器<-->qmtpd/qmqpd机器）或者类似qmail-ldap里的cluster功能。而且要强大得多！！qmail的qmqp/qmtp协议设计得很好，思路也很不错，不过具体的implemenation就不够好。使用postfix可享受最大的灵活性。
如果postfix能考虑完整的实现qmail的qmqp/qmtp等不错的特性，并吸取qmail的优点话，那么qmail的价值则更加大打折扣了。
2.配置思路
# # +------+ # local | | # handling | MTA1 | # layer | | # +------+ # ^ foo@lvs.hzqbbc.com # | # +--------+ +------+ # +------+ | | | | # inbound |mail | ----->|switch |-----> | MTA2 | bar@lvs.hzqbbc.com # email +------+ | | | | # +--------+ +------+ # # 邮件交换 local handling layer
将不同用户的邮件switch到不同的机器，达到分布的目的。配置方法见下文。
3.配置方法：
在mail switch机器上：
postconf -n
alias_database = hash:/etc/postfix/aliases alias_maps = hash:/etc/postfix/aliases command_directory = /usr/sbin config_directory = /etc/postfix daemon_directory = /usr/lib/postfix debug_peer_level = 2 inet_interfaces = all local_recipient_maps = unix:passwd.byname $alias_maps $virtual_mailbox_maps mail_owner = postfix mailq_path = /usr/bin/mailq.postfix manpage_directory = /usr/share/man mydestination = $myhostname,$mydomain,$transport_maps mydomain = LVS.hzqbbc.com myhostname = LVS.hzqbbc.com mynetworks = 192.168.0.0/16, 127.0.0.0/8 myorigin = $mydomain newaliases_path = /usr/bin/newaliases.postfix queue_directory = /var/spool/postfix readme_directory = /etc/postfix/README_FILES sample_directory = /etc/postfix/samples sendmail_path = /usr/sbin/sendmail.postfix setgid_group = postdrop smtpd_banner = $myhostname ESMTP $mail_name Mail-Switch ($mail_version) smtpd_recipient_restrictions = check_relay_domains,reject transport_maps = hash:/mnt/disk/vmail/maps/transport unknown_local_recipient_reject_code = 450
在MTA1和MTA2（这里我定义的机器名字是srv1, srv2)上配置完全一样，除了机器名不同，因此只列出一个srv2的example：
postconf -n
alias_database = hash:/etc/postfix/aliases alias_maps = hash:/etc/postfix/aliases command_directory = /usr/sbin config_directory = /etc/postfix daemon_directory = /usr/lib/postfix debug_peer_level = 2 inet_interfaces = all local_recipient_maps = unix:passwd.byname $alias_maps $virtual_mailbox_maps local_transport = virtual mail_owner = postfix mailq_path = /usr/bin/mailq.postfix manpage_directory = /usr/share/man mydestination = $myhostname,$mydomain, $virtual_mailbox_domains mydomain = LVS.hzqbbc.com myhostname = SRV2.LVS.hzqbbc.com mynetworks = 192.168.0.0/16, 127.0.0.0/8 myorigin = $mydomain newaliases_path = /usr/bin/newaliases.postfix queue_directory = /var/spool/postfix readme_directory = /etc/postfix/README_FILES sample_directory = /etc/postfix/samples sendmail_path = /usr/sbin/sendmail.postfix setgid_group = postdrop smtpd_banner = $myhostname ESMTP $mail_name ($mail_version) transport_maps = hash:/mnt/disk/vmail/maps/vuser_transport unknown_local_recipient_reject_code = 450 virtual_gid_maps = static:250 virtual_mailbox_base = /mnt/disk/vmail virtual_mailbox_domains = lvs.hzqbbc.com virtual_mailbox_maps = hash:/mnt/disk/vmail/maps/mailbox virtual_uid_maps = static:250
之后就是配置了。
/mnt/disk/vmail/maps下有这些文件：
cat mailbox
LVS.hzqbbc.com anything hzq@LVS.hzqbbc.com hzq/Maildir/ ben@LVS.hzqbbc.com ben/Maildir/ foo@LVS.hzqbbc.com foo/Maildir/ bar@LVS.hzqbbc.com bar/Maildir/
cat transport
foo@LVS.hzqbbc.com smtp:[SRV1.LVS.hzqbbc.com] bar@LVS.hzqbbc.com smtp:[SRV2.LVS.hzqbbc.com]
cat vuser_transport
foo@LVS.hzqbbc.com virtual: bar@LVS.hzqbbc.com virtual:
这样，对外部用户只有一个postfix mail-switch/hub。投递给该switch的邮件会转到合适的后端机器（srv1 or srv2）并进行本地投递，整个系统扩容时也特别简单，增加1-n个switch的机器，并增加相应mx记录即可。负载可以比较平均，而不需要使用LVS设备
后端扩充用户及存储也不难，不断增加srvn(n>2)就可以了。这样的体系缺点是效率不够高，耗费机器比较多，但TOC相当低，不需要太多的开发，直接利用postfix的功能并进行合理扩充即可。
Posted by hzqbbc at 09:27 PM | Comments (0)
December 18, 2002
产生mail loop的原因及解决方案
Mail loop是一个很常见，但有时却又让人很难发现故障所在的问题。如何去发现和解决呢？这里以我自己遇到的mail loop问题为例子，来分析一下个中原因。
日志里关于mail loop的内容如下：
Dec 18 18:35:20 smtp postfix/cleanup[9526]: 4846417D5E:
message-id=<20021218223122.C5B6C680A@smtp.hzqbbc.com>
Dec 18 18:35:20 smtp postfix/qmgr[9342]: 4846417D5E:
from=<hzqbbc@hzqbbc.com>, size=1431, nrcpt=4 (queue active)
Dec 18 18:35:20 smtp postfix/local[9585]: C4F5117D5C:
to=<postmaster@192.168.0.27>, relay=local, delay=33,
status=sent (forwarded as 4846417D5E)
Dec 18 18:35:20 smtp postfix/local[9585]: 4846417D5E:
to=<postmaster@192.168.0.27>, relay=local, delay=22,
status=bounced (mail forwarding loop for postmaster@192.168.0.27)
这里ldap的entry是：
dn: mail=postmaster@overtheland.com,virtualDomain=overtheland.com,o=myldap
objectClass: top
objectClass: MailAccount
objectClass: MailAlias
accountActive: TRUE
mailacceptAction: RELAY
mail: postmaster@overtheland.com
homeDirectory: overtheland.com/postmaster
mailbox: overtheland.com/postmaster/Maildir/
quota: 20971520S,1000C
maildrop: postmaster@smtp.overtheland.com
maildrop: root@overtheland.com
maildrop: postmaster@192.168.0.27
maildrop: hzqbbc@redhut.net
相应的ldap设置是：
=================================================================
ldapstaraliasfixup_query_filter = (&(maildrop=%s)(accountActive=TRUE))
ldapstaraliasfixup_result_attribute = mail
~~~~~~~~~~~~
ldapforwardings_query_filter =(&(mail=%s)(accountActive=TRUE))
ldapforwardings_result_attribute = maildrop
~~~~~~~~~~~~~
这里为什么有问题呢？很简单的解释：我发信给root@192.168.0.27。按照alias部分的ldap
设定，信笺应该投递给postmaster@overtheland.com。。确实正常的投递了。不过问题也就
来了。因为还设置了forward部分，而它expect的是maildrop属性的值，这里maildrop有多个值
其中包括了root@192.168.0.27，这样forward时除了给那些非root@192.168.0.27的地址发信外
还会给root@192.168.0.27自己发信。。。这个就产生loop了（看log可看到）
之后。由于root@overtheland.com并不存在机器上（只是一个postmaster@overtheland.com）
的一个alias）。而这时pipe将信笺给了maildrop，maildrop查ldap时并没有办法查得出
root@overtheland.com就是postmaster@overtheland.com（这个是maildrop的ldap部分缺陷，
有待改进）。因此返回user unknow。这个时候自然得给发信者。。。
之后就是很混乱的loop及bounce 信笺了。所以注意一点。forward及alias必须分开来
设置（用不同属性）否则就麻烦了。
而有关类似的mail loop问题的讨论，在Postfix邮件列表上非常多，这里仅举一例：
寄件者：Dirk Datzert (Dirk.Datzert@rasselstein-hoesch.de)
主旨：postfix and ldap
View this article only
新闻群组：mailing.postfix.users
日期：2001-04-10 14:54:04 PST
Hi all,
I have a postfix question with ldap using:
How can I configure aliases_maps and virtual_maps for the following
issue:
a user has a local 'mail'/'mailalternateaddress' with 'uid' and an
'mailforwardingaddress'
I could configure for local delivery:
alias_maps: ldap:aliases
aliases_query_filter = (│(mail=%s)(mailalternateaddress=%s))
aliases_result_attribute = uid
If a user has no local address I can forward:
virtual_maps: ldap:forwarders
forwarders_query_filter = (│(mail=%s)(mailalternateaddress=%s))
forwarders_result_attribute = mailforwardingaddress
Now my question:
How can I configure postfix to do both for one user: delivery local and
forward ?
Thanx,
Dirk
-
To unsubscribe, send mail to majordomo@postfix.org with content
(not subject): unsubscribe postfix-users
第 4 条留言
寄件者：LaMont Jones (lamont@hp.com)
主旨：Re: postfix and ldap
View this article only
新闻群组：mailing.postfix.users
日期：2001-04-10 16:04:07 PST
> aliases_query_filter = (│(mail=%s)(mailalternateaddress=%s))
> aliases_result_attribute = uid
> virtual_maps: ldap:forwarders
> forwarders_query_filter = (│(mail=%s)(mailalternateaddress=%s))
> forwarders_result_attribute = mailforwardingaddress
> Now my question:
>
> How can I configure postfix to do both for one user: delivery local and
> forward ?
result_attribute is a _LIST of attributes, so if you can get the changes
down to one map (merge alias and virtual), then you can just say:
_result_attribute = uid mailforwardingaddress
Untested, your mileage may vary, but we have similar things working in
production.
lamont
Wietse关于mail loop的解释
m 如何防止mail loop？看看wietse如何说的 ( 2003年1月2日21:28 )http://groups.google.com/groups?hl=zh-CN&lr=&ie=UTF-8&inlang=zh-CN&threadm=a3f25j%24urn%241%40FreeBSD.csie.NCTU.edu.tw&rnum=6&prev=/groups%3Fq%3Dmail%2Bloop%2Blist%2Bgroup:mailing.postfix.users%26hl%3Dzh-CN%26lr%3D%26ie%3DUTF-8%26inlang%3Dzh-CN%26selm%3Da3f25j%2524urn%25241%2540FreeBSD.csie.NCTU.edu.tw%26rnum%3D6
You must list myhostname under mydestination:
mydestination = $myhostname localhost.$mydomain ...
Otherwise you will have a mail loop.
Wietse
寄件者：Wietse Venema (wietse@porcupine.org)
主旨：Re: mail loops back to myself
View this article only
新闻群组：mailing.postfix.users
日期：2003-01-18 17:29:08 PST
Is clanetsys.homeunix.net listed in mydestination? See the
main.cf file comments for guidance.
Is clanetsys.homeunix.net listed in virtual_alias_domains?
See the virtual(5) manual page for guidance.
Is clanetsys.homeunix.net listed in virtual_mailbox_domains?
See the VIRTUAL_README file for guidance.
If the domain is not listed in any of the above, then mail
will loop.
Wietse
Posted by hzqbbc at 05:58 PM | Comments (0)
December 09, 2002
Postfix更换Logo了
一直以来，Postfix的logo都没有，现在，终于有一个了！不过不知道是谁设计的呢？感觉看上去还以为是mailman呢，但不管如何，有了就是有了！
请看这里：
具体地址在：
Posted by hzqbbc at 09:58 PM | Comments (1)
December 05, 2002
限制特定用户收发邮件的高级访问控制方法
一般邮件应用稍微有一电规模的企业，大多都会涉及到这样的应用需求：
某销售A只能收到来自163.com及sina.com的邮件 某技术员B只能给内部用户发邮件，不允许向外网发邮件，但可以收到外网邮件 某主管可以收发内、外网邮件 ......
如何实现呢？Postfix 1.x就已经可以通过配置实现这些功能了。
以下是邮件列表里Noel Jones 提出的解决方案：
Hello!
I have the same problem:
1. I need to permit some internal users (not all) to send mail to any
external user (Internet)
2. I need to permite any external user to send mail to some my internal
users (not all)
In fact, the problem is: just "some" of my internal users have permission to
receive mail from and send mail to the Internet.
How can I solve this?
寄件者：Noel Jones (njones@megan.vbhcs.org)
主旨：Re: RES: User Restriction.....
View this article only
新闻群组：mailing.postfix.users
日期：2002-12-04 15:18:04 PST
At 05:39 PM 12/4/02 -0200, raissa.medeiros@caixa.gov.br wrote:
>1. I need to permit some internal users (not all) to send mail to any
>external user (Internet)
>2. I need to permite any external user to send mail to some my internal
>users (not all)
>
>In fact, the problem is: just "some" of my internal users have permission
>to receive mail from and send mail to the Internet.
>
>How can I solve this?
>
>Thanks,
>Raissa
Restricting also who can receive mail from the internet changes the example
I posted a few minutes ago.
In the example below, the same list of restricted_users is used for
controlling both who can send and who can receive internet mail. If you
don't require the local_plus feature, just leave that part out.
- in main.cf:
- use restriction classes to make restricted_users file more readable.
smtpd_restriction_classes = local_only, local_plus
local_only =
reject_unauth_destination
permit_mynetworks
reject
local_plus =
check_recipient_access hash:/etc/postfix/local_plus
check_sender_access hash:/etc/postfix/local_plus
reject_unauth_destination
permit_mynetworks
reject
- this is the default setting, required for this setup.
smtpd_delay_reject = yes
- we'll do this in sender restrictions to avoid open relay problems.
smtpd_sender_restrictions =
check_sender_access hash:/etc/postfix/restricted_users
check_recipient_access hash:/etc/postfix/restricted_users
and in /etc/postfix/restricted_users
# /etc/postfix/restricted_users
# this file contains a list of users only allowed to send and receive local
mail
# postmap this file after changes
# local users not listed here have no restrictions
user1@miodemi.com local_only
user2@miodemi.com local_plus
and in /etc/postfix/local_plus:
# /etc/postfix/local_plus
# this file contains allowed destinations and senders
# for users restricted to local_plus
# postmap this file after changes
miproveedor.com OK
Remember to "postmap local_plus" and "postmap restricted_users" after
making changes to them.
Remember to run "postfix reload" after changing main.cf
我自己常用的限制方法
有几个方法：
1) smtpd_restriction_classes = local_only
设置一个限制类别叫local_only，然后参考access(5)的格式做一个访问控制：
local_only = check_recipient_access hash:/etc/postfix/maps/my_rcpt
文件my_rcpt内容：
163.com RELAY 21cn.com RELAY hzqbbc.com RELAY
然后，设置：
smtpd_sender_restrictions = check_sender_access hash:/etc/postfix/maps/mysender
mysender内容：
hzqbbc@local.hzqbbc.com local_only
这样凡是Mail from:
按这个方法，还可以设置更多的类别，例如remote_only以及不限制的帐号等。但这些都只对from:限制。而且不管是否SASL后的。所以有一定缺陷。不过，已经达到目的了。
2)使用snapshot版的policy策略
根据某个hash表或配置文件，判断对应的sender和recipient是否匹配，匹配就返回OK或者DUNNO或者RELAY等（可能RELAY已经过时，这个是postfix 1.1.x的）如果不匹配就返回错误代码
按postfix所带的smtpd-policy.pl模式，修改一下就可以使用了。详细参考POLICY_README等。
3)使用APF for postfix
详细地址：http://apf.org.cn/addon/
其实和2）方法是基本一致的。同样利用postfix内置的policy功能。
Last Update: 2005-06-06
Posted by hzqbbc at 07:34 PM | Comments (0)
December 02, 2002
维护Postfix的一些脚本、程序集锦
作为一名Postfix的管理员，维护Postfix的正常运行，并随时排除故障，这是最基本的任务。本文提供了一些我自己写及平时搜集到的脚本和程序，以便实现自动化的维护和操作。包括自动清理日志，分析日志，自动增加用户等等。
分析拒收邮件的日志分析程序
作者：Wietse Venema
Message 6 in thread
寄件者：Wietse Venema (wietse@porcupine.org)
主旨：Re: Freebsd Postifx Daily messages
View this article only
新闻群组：mailing.postfix.users
日期：2002-12-01 10:48:47 PST
Here's my own /etc/periodic/daily/460.status-mail-rejects script,
and yes it needs updating when logging formats change.
This script will not show rejected mail when the client never
returned after a soft (450) error code from the server. I use 450
for non-FQDN HELO hostnames, so that I can whitelist mis-configured
sites that aren't spammers.
Wietse
#!/bin/sh
#
echo ""; echo "Scanning maillog for rejections:"
zcat -fc /var/log/maillog.0* /var/log/maillog | egrep reject: | \
cut -d : -f 5- | sort | uniq -c | sort -nr | \
grep -v '^ *[1-2] *[^ ]* [^ ]* from [^ ]*: 450 '
echo "";
echo "Scanning maillog for warnings:"
zcat -fc /var/log/maillog.0* /var/log/maillog | egrep warning: | \
cut -d : -f 5- | sort | uniq -c
print "";
echo "Scanning maillog for trouble:"
zcat -fc /var/log/maillog.0* /var/log/maillog | egrep '(fatal|panic):'
exit
Post a follow-up to this message
一个简单的分析log的脚本
Author: Michael Tokarev
#!/bin/sh
# Parses postfix style logs
if [ $# -ne 2 ]; then
echo "Usage: `basename $0` logfile search-criteria"
exit 1
fi
TMPLOG="/tmp/`basename $0`.work.$$"
MATCHES="/tmp/`basename $0`.matches.$$"
LOGFILE="$1"
if [ "${LOGFILE}" != "-" -a ! -f ${LOGFILE} ]; then
echo "File not found (${LOGFILE})"
exit 2
fi
if echo ${LOGFILE} │ egrep '\.bz2$' >/dev/null 2>&1; then
CAT="bunzip2 -c ${LOGFILE}"
elif echo ${LOGFILE} │ egrep '\.(Z│gz)$' >/dev/null 2>&1; then
CAT="gunzip -c ${LOGFILE}"
elif [ "${LOGFILE}" = "-" ]; then
cat > ${TMPLOG}.orig
CAT="cat ${TMPLOG}.orig"
else
CAT="cat ${LOGFILE}"
fi
shift
${CAT} │ egrep -i "$1" │ \
awk '$9 ~ /^[A-Z0-9][A-Z0-9]*:$/ {print $1 " *" $2 ".*" $9}' > ${MATCHES}
[ -s ${MATCHES} ] && ${CAT} │ egrep -i -f ${MATCHES}
[ -f ${TMPLOG}.orig ] && rm ${TMPLOG}.orig
[ -f ${TMPLOG} ] && rm ${TMPLOG}
[ -f ${MATCHES} ] && rm ${MATCHES}
exit 0
我写的一个简单邮件IN/OUT分析脚本
Author: hzqbbc
是参考了Wietse的reject 分析脚本后，得到启发写的，希望对用Postfix的朋友有点帮助。
#!/bin/bash
# mail-statistic.sh :
# Simple script for analysis Postfix maillog
# Version: 0.03
# Author: hzqbbc@hzqbbc.com
#
RAN=`echo $RANDOM`
echo "Mail log analysis ID is $RAN "
echo "Checking ...................."
echo ""
echo ""
# make a direcotry to contain temp log file
mkdir $RAN
cp /var/log/maillog $RAN/maillog
zcat /var/log/maillog.*.gz >> $RAN/maillog
# statistic recipient
echo "Part one : - Top 10 Recipient domain "
fgrep to= $RAN/maillog | cut -d\< -f2 | cut -d\> -f1 | \
cut -d@ -f2 | sort | uniq -c | sort -nr |head
echo ""
echo ""
# statistic sender
echo "Part two: - Top 10 Sender domain "
fgrep from= $RAN/maillog | cut -d\< -f2 | cut -d\> -f1 | \
cut -d@ -f2 | sort | uniq -c | sort -nr | head
echo ""
echo ""
# statistic destination mail total count
echo "Part three: Total lines and sizes of log, And total mail sent"
echo "Log_lines = `cat $RAN/maillog | wc -l | awk '{ print $1 }'`"
echo "Log_sizes = `du -s $RAN/maillog | awk '{ print $1}'`"
echo "Mail_sent = `cat $RAN/maillog | fgrep status=sent | wc -l | \
awk '{ print $1 }'`"
rm -rf $RAN
Posted by hzqbbc at 10:07 AM | Comments (0)
November 28, 2002
关于Postfix虚拟域的相关释疑
在Postfix 1.1.x中，virtual_maps及virtual_mailbox_maps容易搞混，而在未来的Postfix 2.x中，virtual_*相关的设置也会有所变动。因此有必要搞清楚两者之间的区别和联系，以及虚拟域在Postfix中的实现和相关含义。
寄件者："James T. Richardson, Jr." (james.richardson@wickander.com)
主旨：Postifix & Virtual Domains
View this article only
新闻群组：mailing.postfix.users
日期：2002-11-26 15:12:06 PST
After setting up Postfix + MySQL I'm having a hard time getting "virtual
aliases" working.
I'm wanting to use Virtual Mailbox Domains (not Postfix-Style, nor
Sendmail-Style).
Mail is delivered to both my local domain and to my virtual domains. At
least for my users that have Maildir's defined.
Relevant sectionof the config:
---
myhostname = mail-gateway.mydomain.com
myorigin = $mydomain
mydestination = $myhostname, localhost.$mydomain $mydomain
local_recipient_maps = $alias_maps unix:passwd.byname
alias_maps = hash:/etc/postfix/aliases
alias_database = $alias_maps
transport_maps = mysql:/etc/postfix/transport.cf
virtual_mailbox_maps = mysql:/etc/postfix/mysql_virt.cf
hash:/etc/postfix/virtual
virtual_uid_maps = mysql:/etc/postfix/uids.cf
virtual_gid_maps = mysql:/etc/postfix/gids.cf
virtual_mailbox_base = /var/mail/vhosts
---
The "mysql_virt.cf" file looks like:
user=postfix
password=sekret
dbname=maildb
table=users
select_field_maildir
where_field=address
hosts=localhost
Following instructions by Victor Duchovni
(http://groups.google.com/groups?hl=en&lr=&ie=UTF-8&oe=UTF-8&selm=ajh5mv
$h93$1@FreeBSD.csie.NCTU.edu.tw&rnum=1):
> I call such domains *virtual-mailbox* domains. To define a virtual-mailbox
> domain, just a "domain whatever" entry to one of the tables listed in
> $virtual_mailbox_maps. For now it is also necessary to add the domain to
> the transport table in order to arrange for mail to the domain to be
> handled by a suitable virtual delivery agent (presumably but not
> necessarily the Postfix "virtual" delivery agent).
I created a simple hash (for now) files of "domain whatever" (and ran
postmap on it) and stuck it on the end of virtual_mailbox_maps.
Now, I'm not sure where to create these aliases or where to tell Postfix
about them. I tried putting an alias to postmaster@domain underneath the
"domain whatever" line in the virtual file, but that didn't work. I got
a
To=
(recipient postmaster@domain: uid not found in virtual_uid_maps)
Surely my aliases don't need assciated UID/GID's? do they?
---
James T. Richardson, Jr.
james.richardson@wickander.com
Post a follow-up to this message
Message 2 in thread
寄件者：Russell Mosemann (mose@ns.cune.edu)
主旨：Re: Postifix & Virtual Domains
View this article only
新闻群组：mailing.postfix.users
日期：2002-11-26 15:33:51 PST
On Tue, 26 Nov 2002, James T. Richardson, Jr. wrote:
> After setting up Postfix + MySQL I'm having a hard time getting "virtual
> aliases" working.
>
> I'm wanting to use Virtual Mailbox Domains (not Postfix-Style, nor
> Sendmail-Style).
>
> Mail is delivered to both my local domain and to my virtual domains. At
> least for my users that have Maildir's defined.
>
>
> Relevant sectionof the config:
> ---
> myhostname = mail-gateway.mydomain.com
> myorigin = $mydomain
> mydestination = $myhostname, localhost.$mydomain $mydomain
> local_recipient_maps = $alias_maps unix:passwd.byname
> alias_maps = hash:/etc/postfix/aliases
> alias_database = $alias_maps
> transport_maps = mysql:/etc/postfix/transport.cf
> virtual_mailbox_maps = mysql:/etc/postfix/mysql_virt.cf
> hash:/etc/postfix/virtual
> virtual_uid_maps = mysql:/etc/postfix/uids.cf
> virtual_gid_maps = mysql:/etc/postfix/gids.cf
> virtual_mailbox_base = /var/mail/vhosts
> ---
>
> The "mysql_virt.cf" file looks like:
> user=postfix
> password=sekret
> dbname=maildb
> table=users
> select_field_maildir
> where_field=address
> hosts=localhost
>
> Following instructions by Victor Duchovni
> (http://groups.google.com/groups?hl=en&lr=&ie=UTF-8&oe=UTF-8&selm=ajh5mv
> $h93$1@FreeBSD.csie.NCTU.edu.tw&rnum=1):
>
> > I call such domains *virtual-mailbox* domains. To define a virtual-mailbox
> > domain, just a "domain whatever" entry to one of the tables listed in
> > $virtual_mailbox_maps. For now it is also necessary to add the domain to
> > the transport table in order to arrange for mail to the domain to be
> > handled by a suitable virtual delivery agent (presumably but not
> > necessarily the Postfix "virtual" delivery agent).
>
> I created a simple hash (for now) files of "domain whatever" (and ran
> postmap on it) and stuck it on the end of virtual_mailbox_maps.
>
> Now, I'm not sure where to create these aliases or where to tell Postfix
> about them. I tried putting an alias to postmaster@domain underneath the
> "domain whatever" line in the virtual file, but that didn't work. I got
> a
> To=
> (recipient postmaster@domain: uid not found in virtual_uid_maps)
>
> Surely my aliases don't need assciated UID/GID's? do they?
I don't think you read Victor's message very carefully. A virtual domain
is different from a virtual mailbox domain. virtual_maps are used not
only for redirecting email from one address to one or more other
addresses, it is also used for virtual domains. virtual_mailbox_maps are
used for virtual mailbox domains and for mailboxes on the local computer
that do not have an associated Unix account. You are mixing the two, and
that is the cause of your problems.
----
Russell Mosemann, Ph.D. * Computing Services * Concordia University, Nebraska
"A penny saved is a penny earned." "Penny wise, pound foolish."
Post a follow-up to this message
Message 3 in thread
寄件者：Victor.Duchovni@morganstanley.com (Victor.Duchovni@morganstanley.com)
主旨：Re: Postifix & Virtual Domains
View this article only
新闻群组：mailing.postfix.users
日期：2002-11-26 19:12:51 PST
On Tue, 26 Nov 2002, James T. Richardson, Jr. wrote:
> Following instructions by Victor Duchovni
> (http://groups.google.com/groups?hl=en&lr=&ie=UTF-8&oe=UTF-8&selm=ajh5mv
> $h93$1@FreeBSD.csie.NCTU.edu.tw&rnum=1):
>
> > I call such domains *virtual-mailbox* domains. To define a virtual-mailbox
> > domain, just a "domain whatever" entry to one of the tables listed in
> > $virtual_mailbox_maps. For now it is also necessary to add the domain to
> > the transport table in order to arrange for mail to the domain to be
> > handled by a suitable virtual delivery agent (presumably but not
> > necessarily the Postfix "virtual" delivery agent).
>
> I created a simple hash (for now) files of "domain whatever" (and ran
> postmap on it) and stuck it on the end of virtual_mailbox_maps.
>
> Now, I'm not sure where to create these aliases or where to tell Postfix
> about them. I tried putting an alias to postmaster@domain underneath the
> "domain whatever" line in the virtual file, but that didn't work. I got
> a
> To=
> (recipient postmaster@domain: uid not found in virtual_uid_maps)
>
You actually searched for and found some documentation and mostly
followed it correctly! You deserve a break.
Your issue is that you are trying to specify address rewriting rules
("aliases" of sorts) for virtual mailbox domains in the
virtual_mailbox_maps tables. These tables define the pathname of the
virtual user's mailbox, not an alternate address.
For virtual mailbox users the only available address rewriting mechanism
is $virtual_maps. Don't put the "domain anything" keys for virtual
mailbox domainsthere, but do put address rewriting rules there.
So
virtual_mailbox_maps = hash:/etc/postfix/vmboxdomains, mysql:mumble
virtual_maps = mysql:fumble
/etc/postfix/vmboxdomains will have all the "domain anything" keys. List
it first, as these lookups will be very fast in comparison to mysql.
mysql:mumble will map virtual users to their mailbox path.
mysql:fumble (or hash:fumble) will map "aliased" virtual users to the
correct email address(es).
--
Viktor.
Posted by hzqbbc at 01:13 AM | Comments (0)
November 13, 2002
有关于mjt所写的tinycdb的技术内幕讨论
TinyCDB是一个对djb提出的CDB全新的实现，体积非常小，速度非常非常快。有俄罗斯人mjt开发。但它有一个致命的缺点，那就是每次更新/删除/添加都必须重建数据库（也因为不需要维护太多东西，所以才快）。
原文：
About TinyCDB
Message 1 in thread
寄件者：Michael Tokarev (mjt@tls.msk.ru)
主旨：[ot] tinyCDB is available in packaged form
View this article only
新闻群组：mailing.postfix.users
日期：2002-10-12 16:28:04 PST
After recent comparisions and mention of CDB map type, I received
several emails from several people about my tinycdb package. I've
made cosmetic changes in packaging scripts and uploaded binaries for
RedHat-like (rpm) and Debian Linux systems. It is available at
ftp://ftp.corpit.ru/pub/tinycdb/ :
21985 tinycdb-0.72.tar.gz source tarball incl. build scripts
26572 tinycdb-0.72-1.i386.rpm RedHat i386 rpm (rpm -tb)
24064 tinycdb_0.72_i386.deb Debian i386 .deb (native build)
This includes both the -lcdb library and simple cdb command-line
utility to work with .cdb files.
CDB map for postfix is available at ftp://ftp.corpit.ru/pub/postfix/,
see README.CDB in this directory.
/mjt
Message 2 in thread
寄件者：Wietse Venema (wietse@porcupine.org)
主旨：Re: [ot] tinyCDB is available in packaged form
View this article only
新闻群组：mailing.postfix.users
日期：2002-10-12 17:22:01 PST
Why does mkmap_cdb_open() lock the .cdb file?
Wietse
Michael Tokarev:
> After recent comparisions and mention of CDB map type, I received
> several emails from several people about my tinycdb package. I've
> made cosmetic changes in packaging scripts and uploaded binaries for
> RedHat-like (rpm) and Debian Linux systems. It is available at
> ftp://ftp.corpit.ru/pub/tinycdb/ :
>
> 21985 tinycdb-0.72.tar.gz source tarball incl. build scripts
> 26572 tinycdb-0.72-1.i386.rpm RedHat i386 rpm (rpm -tb)
> 24064 tinycdb_0.72_i386.deb Debian i386 .deb (native build)
>
> This includes both the -lcdb library and simple cdb command-line
> utility to work with .cdb files.
>
> CDB map for postfix is available at ftp://ftp.corpit.ru/pub/postfix/,
> see README.CDB in this directory.
>
> /mjt
>
> -
> To unsubscribe, send mail to majordomo@postfix.org with content
> (not subject): unsubscribe postfix-users
>
>
-
To unsubscribe, send mail to majordomo@postfix.org with content
(not subject): unsubscribe postfix-users
Message 3 in thread
寄件者：Michael Tokarev (mjt@tls.msk.ru)
主旨：Re: [ot] tinyCDB is available in packaged form
View this article only
新闻群组：mailing.postfix.users
日期：2002-10-12 18:32:02 PST
Wietse Venema wrote:
> Why does mkmap_cdb_open() lock the .cdb file?
Hmm, this is an interesting question - now, after more than a
year since that discussions, I actually see what did you mean
when asked similar question.
There are two things to protect by a lock: creating a map file
updating of a map - this last one in case of cdb consists of
two operations, namely creating and renaming. I.e., new (temp)
file may be protected during it's creation/writing so no two
processes will write to it at once, or the whole operation
may be protected, so no two proceses will (create+rename) at
once.
Or, in the other words, lock on .cdb file should mean "wait,
I'm working on this map", while a lock on .tmp file should
mean "wait, I'm working with this file".
There is a minor and somewhat unclean difference, but it
exists.
I don't know which is better - in fact, there is no real
difference in a result, it seems. Why? Because if the
only program who'll update a map will be postfix, there is
really no difference once all postfix's components are
consistent. But if there will be other programs, things
will be more "interesting": there is no "standard" choice
for a name of a temp file, one program may use .cdb.tmp
(as my mkmap_cdb), another may use .tmp, yet another may
use random-unique-name etc. No other tool I know of uses
any locking at all when creating cdb (there are only 2:
DJB's cdbmake and tinycdb's cdb -c).
Well, if some other program will actually do soome locking,
it's a good idea to choose the same locking scheme, and the
right one (from the CDB's ideology) seems to be to lock .tmp
file ("I'm working on this file").
But implementation is more difficult in this case (when locking
.tmp) - it's more tricky to avoid a race here and not to start
updating .cdb file instead of .tmp -- if memory serves me right;
it was long ago when I thought about this. This was the only
reason why I choosed to lock .cdb instead of .tmp when first
wrote the map. Anyway, it shouldn't be impossible ;)
BTW, the whole cdb thing, just like maildir format, seems to
be designed to avoid any and all locking altogether. For this,
simplest solution is to choose a random (unique) name of temp
file and forget about concurrency issues. This solution has
one implication (already noted by you) - it's possible to have
leftover files after crashes in this case.
/mjt
-
To unsubscribe, send mail to majordomo@postfix.org with content
(not subject): unsubscribe postfix-users
Message 4 in thread
寄件者：Matthias Andree (ma@dt.e-technik.uni-dortmund.de)
主旨：Re: [ot] tinyCDB is available in packaged form
View this article only
新闻群组：mailing.postfix.users
日期：2002-10-14 09:00:06 PST
Michael Tokarev
> There are two things to protect by a lock: creating a map file
> updating of a map - this last one in case of cdb consists of
> two operations, namely creating and renaming. I.e., new (temp)
> file may be protected during it's creation/writing so no two
> processes will write to it at once, or the whole operation
> may be protected, so no two proceses will (create+rename) at
> once. ...
> BTW, the whole cdb thing, just like maildir format, seems to
> be designed to avoid any and all locking altogether.
Not really, but that's why you pass DJB's applications the name of a
temp file. Whether you let the user choose one or use mkstemp, is a
matter of convenience and portability.
> For this, simplest solution is to choose a random (unique) name of
> temp file and forget about concurrency issues. This solution has one
> implication (already noted by you) - it's possible to have leftover
> files after crashes in this case.
Do you think it would be possible to use mkstemp or something for the
new map file and rename(2) that into place after it is complete, and
dropping all locking code?
I'd not really worry about "two processes rename to the same .cdb file"
at all: let the last one to rename win. Usually, these .cdb files are
under administrator control and will actually be built from some input
by a Makefile and make or something.
You could just document the template you passed to mkstemp() and tell
users they can safely delete these files. Should this happen while your
makecdb is running at the same time, rename() will fail without touching
the destination file.
No need to be paranoid here.
--
Matthias Andree
-
To unsubscribe, send mail to majordomo@postfix.org with content
(not subject): unsubscribe postfix-users
Post a follow-up to this message
Message 5 in thread
寄件者：Victor.Duchovni@morganstanley.com (Victor.Duchovni@morganstanley.com)
主旨：Re: [ot] tinyCDB is available in packaged form
View this article only
新闻群组：mailing.postfix.users
日期：2002-10-14 10:12:01 PST
On Mon, 14 Oct 2002, Matthias Andree wrote:
> I'd not really worry about "two processes rename to the same .cdb file"
> at all: let the last one to rename win. Usually, these .cdb files are
> under administrator control and will actually be built from some input
> by a Makefile and make or something.
>
I don't think this is right. When I run two concurrent copies of "postmap
-i" I expect both sets of key/value pairs to be added to the map.
Does the cdb "mkmap" code support incremental inserts or only full map
rebuilds?
--
Viktor.
-
To unsubscribe, send mail to majordomo@postfix.org with content
(not subject): unsubscribe postfix-users
Post a follow-up to this message
Message 6 in thread
寄件者：Michael Tokarev (mjt@tls.msk.ru)
主旨：Re: [ot] tinyCDB is available in packaged form
View this article only
新闻群组：mailing.postfix.users
日期：2002-10-14 12:28:07 PST
Victor.Duchovni@morganstanley.com wrote:
[]
> I don't think this is right. When I run two concurrent copies of "postmap
> -i" I expect both sets of key/value pairs to be added to the map.
>
> Does the cdb "mkmap" code support incremental inserts or only full map
> rebuilds?
No - this is a principal limitation of a cdb, it's a Constant database.
I.e. any and all manipulation with content requires complete rebuild.
Well, not strictly this - updates are possible by rebuilding the whole
hash table (inplace) or by recreating a file using old file as a source,
but that's not worth the effort to program the whole thing IMHO. For
this very reason, cdb isn't sutable for pop-before-smtp and the like
dynamic maps.
Mkmap code in my cdb patch does not support incremental updates, again,
for the same reason (it will be complete rebuild anyway).
/mjt
-
To unsubscribe, send mail to majordomo@postfix.org with content
(not subject): unsubscribe postfix-users
Post a follow-up to this message
Message 7 in thread
寄件者：Bennett Todd (bet@rahul.net)
主旨：Re: [ot] tinyCDB is available in packaged form
View this article only
新闻群组：mailing.postfix.users
日期：2002-10-15 12:24:05 PST
--n8g4imXOkfNTN/H1
Content-Type: text/plain; charset=us-ascii
Content-Disposition: inline
2002-10-14-15:24:50 Michael Tokarev:
> No - this is a principal limitation of a cdb, it's a Constant
> database. I.e. any and all manipulation with content requires
> complete rebuild. [...] For this very reason, cdb isn't sutable
> for pop-before-smtp and the like dynamic maps.
Not to dispute any of the other stuff you said, but I'm really not
sure about this bit.
I'd have to do some experimenting, but I really suspect
pop-before-smtp would work very well indeed with cdb.
It might be worth taking the trouble to have a version of cdb -c
inlined into the daemon, perhaps make a dynamically loadable perl
module if I were doing this on my pop-before-smtp daemon. That'd
save the need for fork-n-exec for each db update; if you ain't on
Linux, fork-n-exec is overpriced:-).
But with an in-memory cdb create, the update would be
open-tmp-file/write/close/rename, and for reasonable size maps ---
many folks don't have that many concurrent poppers from outside
mynetworks --- I'm sure this would be cheaper than
open-db/lock/read/write/flush/unlock/close. In fact, I wouldn't be
surprised if the break-even point for an optimized cdb-writer
weren't actually pretty darned large. cdb is awfully quick.
Hmm. Now I'm gonna have to find time to play with CDB_File.
-Bennett
--n8g4imXOkfNTN/H1
Content-Type: application/pgp-signature
Content-Disposition: inline
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.0.7 (GNU/Linux)
iD8DBQE9rF82HZWg9mCTffwRAuGhAJ9×4iqOrAmlmsXdy0QpPHFFFFaDCgCgjumd
Dpbm/WiCJre7+sXb9TCRJ/g=
=AMc7
-----END PGP SIGNATURE-----
n8g4imXOkfNTN/H1
-
To unsubscribe, send mail to majordomo@postfix.org with content
(not subject): unsubscribe postfix-users
Post a follow-up to this message
Message 8 in thread
寄件者：Michael Tokarev (mjt@tls.msk.ru)
主旨：Re: [ot] tinyCDB is available in packaged form
View this article only
新闻群组：mailing.postfix.users
日期：2002-10-15 12:46:02 PST
Bennett Todd wrote:
> 2002-10-14-15:24:50 Michael Tokarev:
>
>>No - this is a principal limitation of a cdb, it's a Constant
>>database. I.e. any and all manipulation with content requires
>>complete rebuild. [...] For this very reason, cdb isn't sutable
>>for pop-before-smtp and the like dynamic maps.
>
>
> Not to dispute any of the other stuff you said, but I'm really not
> sure about this bit.
>
> I'd have to do some experimenting, but I really suspect
> pop-before-smtp would work very well indeed with cdb.
>
> It might be worth taking the trouble to have a version of cdb -c
> inlined into the daemon, perhaps make a dynamically loadable perl
> module if I were doing this on my pop-before-smtp daemon. That'd
> save the need for fork-n-exec for each db update; if you ain't on
> Linux, fork-n-exec is overpriced:-).
>
> But with an in-memory cdb create, the update would be
> open-tmp-file/write/close/rename, and for reasonable size maps ---
> many folks don't have that many concurrent poppers from outside
> mynetworks --- I'm sure this would be cheaper than
> open-db/lock/read/write/flush/unlock/close. In fact, I wouldn't be
> surprised if the break-even point for an optimized cdb-writer
> weren't actually pretty darned large. cdb is awfully quick.
One word: scalability. With a dynamically updateable map, a
time needed to insert one record does not depend on the number
of records already present in a map (well, almost). Cdb
requires complete rewrite. It will work for small sites.
Also, create/rename requires filesystem metadata updates,
while read/write does not - thus, depending on a filesystem,
situation with cdb will be worse. But it has one advantage:
it will work well over NFS, so it will be possible to use
PBS when pop daemon isn't on the same machine as smtp.
In fact, the best way to implement PBS is to keep all records
in memory (8 bytes for each (IP and time) isn't that much)
and use tcp map in postfix (and this too will work good
on a large site). Crashes/reboots aren't that important
here, it's almost ok to lose data since it isn't keept
for a long time anyway (just "rePOP" will be required).
With such a method, far more efficient search algorithm
may be implemented (i.e. sorted array of IP addresses,
maybe hashed).
> Hmm. Now I'm gonna have to find time to play with CDB_File.
I do NOT recommend this, at least w/o fixing things first.
Cdbmake code has at least one (huge) memory leak and thus
isn't sutable for a long-living daemon. Also, cdbmake code
requires much more memory than tinycdb, and is less
efficient (this all is true for cdb-0.75, I not looked
if new version was released).
What I want to is to find a time to write perl and especially
nss module for cdb - this is on a todo list for more than a
year now... ;)
/mjt
-
To unsubscribe, send mail to majordomo@postfix.org with content
(not subject): unsubscribe postfix-users
寄件者：Bennett Todd (bet@rahul.net)
主旨：Re: [ot] tinyCDB is available in packaged form
View this article only
新闻群组：mailing.postfix.users
日期：2002-10-16 09:40:03 PST
--kXdP64Ggrk/fb43R
Content-Type: text/plain; charset=us-ascii
Content-Disposition: inline
Content-Transfer-Encoding: quoted-printable
2002-10-15-15:43:50 Michael Tokarev:
> Bennett Todd wrote:
> >I'd have to do some experimenting, but I really suspect
> >pop-before-smtp would work very well indeed with cdb.
>=20
> One word: scalability. With a dynamically updateable map, a
> time needed to insert one record does not depend on the number
> of records already present in a map (well, almost). Cdb
> requires complete rewrite. It will work for small sites.
Define "small".
I seriously believe that a large majority of pop-before-smtp users
rarely have more than a few records in their database; and that
sites with as many as a hundred such records much of the time are
liable to be a teensy minority, if not a non-existent limit case.
Where's the crossover point? I.e. where does the cost of the CDB
generation grow to exceed the cost of doing the DB manipulation?
This undoubtedly varies from platform to platform. I'll try and
produce an answer for at least one platform:-).
> Also, create/rename requires filesystem metadata updates,
> while read/write does not - thus, depending on a filesystem,
> situation with cdb will be worse.
I'll try it with ext3, and with tmpfs. But while read/write may not
require metadata updates, it does require locking. I don't know
what sort of difference that will make, if any.
> In fact, the best way to implement PBS is to keep all records
> in memory (8 bytes for each (IP and time) isn't that much)
> and use tcp map in postfix (and this too will work good
> on a large site).
Sounds like we're converging on DRAC:-).
> >Hmm. Now I'm gonna have to find time to play with CDB_File.
>=20
> I do NOT recommend this, at least w/o fixing things first.
> Cdbmake code has at least one (huge) memory leak and thus
> isn't sutable for a long-living daemon.
观看文件全部内容 （仍有 22 行）
Post a follow-up to this message
Message 10 in thread
寄件者：Matthias Andree (ma@dt.e-technik.uni-dortmund.de)
主旨：Re: [ot] tinyCDB is available in packaged form
View this article only
新闻群组：mailing.postfix.users
日期：2002-10-21 06:25:57 PST
Bennett Todd
> I'd have to do some experimenting, but I really suspect
> pop-before-smtp would work very well indeed with cdb.
pop-before-smtp is inherently insecure, no matter HOW you implement
it. The problem is that you give relay permission to a host that may
well be "replaced" by some other (1st hosts disconnects, 2nd is assigned
the previous IP from 1st host from the dialup pool), without your SMTP
server noticing the disconnect. SMTP AUTH does not suffer from these
problems and does not need all these expire cronjobs and tossing data
from POP server to SMTP server and the like, worrying about how you
update a cdb... A cdb does not cut incremental updates. It would cope
well with a passwd like file though.
--
Matthias Andree
-
To unsubscribe, send mail to majordomo@postfix.org with content
(not subject): unsubscribe postfix-users
寄件者：Bennett Todd (bet@rahul.net)
主旨：Re: [ot] tinyCDB is available in packaged form
View this article only
新闻群组：mailing.postfix.users
日期：2002-10-21 07:44:59 PST
--gTtJ75FAzB1T2CN6
Content-Type: text/plain; charset=us-ascii
Content-Disposition: inline
2002-10-21-08:59:40 Matthias Andree:
> pop-before-smtp is inherently insecure, no matter HOW you implement
> it.
Lightly so, yes.
> The problem is that you give relay permission to a host that may
> well be "replaced" by some other (1st hosts disconnects, 2nd is assigned
> the previous IP from 1st host from the dialup pool), without your SMTP
> server noticing the disconnect.
That's why there's a timeout. A default of 1/2 hour seems to be
fine. Yes, there's a window during which someone else who got the IP
addr of a recently-pop-authed machine could relay. No, it doesn't
appear to be a problem in practice; I've never heard of such a setup
actually being exploited through this timing window.
> SMTP AUTH does not suffer from these problems [...]
If SMTP AUTH didn't require SASL, or if there were an implementation
of SASL besides Cyrus, I might be more interested. Is it possible to
implement enough SASL for SMTP AUTH without bringing in mountains of
excruciatingly painful goo, like e.g. GSSAPI with it's ASN.1 and so
forth?
> [...] and does not need all these expire cronjobs and tossing data
> from POP server to SMTP server and the like, [...]
I don't have any expire cronjobs, I just have a single persistent
daemon that manages the data file. Works great for a single server
that's both the pop/imap mailbox server and the smtp relay server.
For server farms, something more like DRAC or whoson would probably
be in order.
> worrying about how you update a cdb... A cdb does not cut
> incremental updates.
Sure it does, it does so beautifully. cdb file rebuild is so fast
that doing a full rebuild for each update is practical in many
settings --- as DNS hosting sites have discovered when using
tinydns.
-Bennett
--gTtJ75FAzB1T2CN6
Content-Type: application/pgp-signature
Content-Disposition: inline
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.0.7 (GNU/Linux)
iD8DBQE9tA7UHZWg9mCTffwRAhs7AKDEw1uALoCxjHAhlSJA0xic+2hfTACgm6a2
0BJdYgRUKo90nwQlSFdkK24=
=XNg1
-----END PGP SIGNATURE-----
gTtJ75FAzB1T2CN6
-
To unsubscribe, send mail to majordomo@postfix.org with content
(not subject): unsubscribe postfix-users
Post a follow-up to this message
Message 12 in thread
寄件者："Schmehl, Paul L" (pauls@utdallas.edu)
主旨：RE: [ot] tinyCDB is available in packaged form
View this article only
新闻群组：mailing.postfix.users
日期：2002-10-21 08:30:04 PST
> -----Original Message-----
> From: Bennett Todd [mailto:bet@rahul.net]
> Sent: Monday, October 21, 2002 9:28 AM
> To: Matthias Andree
> Cc: postfix-users@postfix.org
> Subject: Re: [ot] tinyCDB is available in packaged form
>
>
> 2002-10-21-08:59:40 Matthias Andree:
> > pop-before-smtp is inherently insecure, no matter HOW you implement
> > it.
>
> Lightly so, yes.
>
> > The problem is that you give relay permission to a host that may well
> > be "replaced" by some other (1st hosts disconnects, 2nd is assigned
> > the previous IP from 1st host from the dialup pool), without your SMTP
> > server noticing the disconnect.
>
> That's why there's a timeout. A default of 1/2 hour seems to
> be fine. Yes, there's a window during which someone else who
> got the IP addr of a recently-pop-authed machine could relay.
> No, it doesn't appear to be a problem in practice; I've never
> heard of such a setup actually being exploited through this
> timing window.
There's quite an assumption behind this claim of insecurity. The
assumption is:
1) User one connects using pop before SMTP (or DRAC), reads mail and
disconnects before the timeout.
2) User two knows he a) has user one's IP address and knows b) the
IP/ hostname of the mail server and knows c) the server is open for
relay and intends d) to send spam through the server.
This assumption requires some incredible leaps in logic. Is it
possible? Of course. Just as breaking a 4096 AES key is possible. Is
it likely? Not in this lifetime. Even if I were a spammer, and even if
I knew that a particular server used pop before SMTP or DRAC and even
if I knew someone who used that server was online and using the server
and even if I knew precisely when they disconnected, I would still
have to obtain the exact same IP address to exploit the server.
I have real problems thinking of a scenario where this is even remotely
possible.
Paul Schmehl (pauls@utdallas.edu)
TCS Department Coordinator
The University of Texas at Dallas
AVIEN Founding Member
http://www.utdallas.edu/~pauls/
-
To unsubscribe, send mail to majordomo@postfix.org with content
(not subject): unsubscribe postfix-users
Post a follow-up to this message
Message 13 in thread
寄件者：Simon White (simon@mtds.com)
主旨：Re: [ot] tinyCDB is available in packaged form
View this article only
新闻群组：mailing.postfix.users
日期：2002-10-21 08:32:05 PST
21-Oct-02 at 10:19, Schmehl, Paul L (pauls@utdallas.edu) wrote :
> 1) User one connects using pop before SMTP (or DRAC), reads mail and
> disconnects before the timeout.
> 2) User two knows he a) has user one's IP address and knows b) the
> IP/ hostname of the mail server and knows c) the server is open for
> relay and <