Zhiqiang::He: November 2002 Archives

« October 2002 | Main | December 2002 »

November 28, 2002

关于Postfix虚拟域的相关释疑

在Postfix 1.1.x中，virtual_maps及virtual_mailbox_maps容易搞混，而在未来的Postfix 2.x中，virtual_*相关的设置也会有所变动。因此有必要搞清楚两者之间的区别和联系，以及虚拟域在Postfix中的实现和相关含义。

寄件者："James T. Richardson, Jr." (james.richardson@wickander.com)
主旨：Postifix & Virtual Domains

View this article only
新闻群组：mailing.postfix.users
日期：2002-11-26 15:12:06 PST

After setting up Postfix + MySQL I'm having a hard time getting "virtual
aliases" working.

I'm wanting to use Virtual Mailbox Domains (not Postfix-Style, nor
Sendmail-Style).

Mail is delivered to both my local domain and to my virtual domains. At
least for my users that have Maildir's defined.

Relevant sectionof the config:
---
myhostname = mail-gateway.mydomain.com
myorigin = $mydomain
mydestination = $myhostname, localhost.$mydomain $mydomain
local_recipient_maps = $alias_maps unix:passwd.byname
alias_maps = hash:/etc/postfix/aliases
alias_database = $alias_maps
transport_maps = mysql:/etc/postfix/transport.cf
virtual_mailbox_maps = mysql:/etc/postfix/mysql_virt.cf
hash:/etc/postfix/virtual
virtual_uid_maps = mysql:/etc/postfix/uids.cf
virtual_gid_maps = mysql:/etc/postfix/gids.cf
virtual_mailbox_base = /var/mail/vhosts
---

The "mysql_virt.cf" file looks like:
user=postfix
password=sekret
dbname=maildb
table=users
select_field_maildir
where_field=address
hosts=localhost

Following instructions by Victor Duchovni
(http://groups.google.com/groups?hl=en&lr=&ie=UTF-8&oe=UTF-8&selm=ajh5mv
$h93$1@FreeBSD.csie.NCTU.edu.tw&rnum=1):

> I call such domains *virtual-mailbox* domains. To define a virtual-mailbox
> domain, just a "domain whatever" entry to one of the tables listed in
> $virtual_mailbox_maps. For now it is also necessary to add the domain to
> the transport table in order to arrange for mail to the domain to be
> handled by a suitable virtual delivery agent (presumably but not
> necessarily the Postfix "virtual" delivery agent).

I created a simple hash (for now) files of "domain whatever" (and ran
postmap on it) and stuck it on the end of virtual_mailbox_maps.

Now, I'm not sure where to create these aliases or where to tell Postfix
about them. I tried putting an alias to postmaster@domain underneath the
"domain whatever" line in the virtual file, but that didn't work. I got
a
To=, relay=virtual, delay=0, status=deferred
(recipient postmaster@domain: uid not found in virtual_uid_maps)

Surely my aliases don't need assciated UID/GID's? do they?

---

James T. Richardson, Jr.
james.richardson@wickander.com
Post a follow-up to this message

Message 2 in thread
寄件者：Russell Mosemann (mose@ns.cune.edu)
主旨：Re: Postifix & Virtual Domains

View this article only
新闻群组：mailing.postfix.users
日期：2002-11-26 15:33:51 PST

On Tue, 26 Nov 2002, James T. Richardson, Jr. wrote:

> After setting up Postfix + MySQL I'm having a hard time getting "virtual
> aliases" working.
>
> I'm wanting to use Virtual Mailbox Domains (not Postfix-Style, nor
> Sendmail-Style).
>
> Mail is delivered to both my local domain and to my virtual domains. At
> least for my users that have Maildir's defined.
>
>
> Relevant sectionof the config:
> ---
> myhostname = mail-gateway.mydomain.com
> myorigin = $mydomain
> mydestination = $myhostname, localhost.$mydomain $mydomain
> local_recipient_maps = $alias_maps unix:passwd.byname
> alias_maps = hash:/etc/postfix/aliases
> alias_database = $alias_maps
> transport_maps = mysql:/etc/postfix/transport.cf
> virtual_mailbox_maps = mysql:/etc/postfix/mysql_virt.cf
> hash:/etc/postfix/virtual
> virtual_uid_maps = mysql:/etc/postfix/uids.cf
> virtual_gid_maps = mysql:/etc/postfix/gids.cf
> virtual_mailbox_base = /var/mail/vhosts
> ---
>
> The "mysql_virt.cf" file looks like:
> user=postfix
> password=sekret
> dbname=maildb
> table=users
> select_field_maildir
> where_field=address
> hosts=localhost
>
> Following instructions by Victor Duchovni
> (http://groups.google.com/groups?hl=en&lr=&ie=UTF-8&oe=UTF-8&selm=ajh5mv
> $h93$1@FreeBSD.csie.NCTU.edu.tw&rnum=1):
>
> > I call such domains *virtual-mailbox* domains. To define a virtual-mailbox
> > domain, just a "domain whatever" entry to one of the tables listed in
> > $virtual_mailbox_maps. For now it is also necessary to add the domain to
> > the transport table in order to arrange for mail to the domain to be
> > handled by a suitable virtual delivery agent (presumably but not
> > necessarily the Postfix "virtual" delivery agent).
>
> I created a simple hash (for now) files of "domain whatever" (and ran
> postmap on it) and stuck it on the end of virtual_mailbox_maps.
>
> Now, I'm not sure where to create these aliases or where to tell Postfix
> about them. I tried putting an alias to postmaster@domain underneath the
> "domain whatever" line in the virtual file, but that didn't work. I got
> a
> To=, relay=virtual, delay=0, status=deferred
> (recipient postmaster@domain: uid not found in virtual_uid_maps)
>
> Surely my aliases don't need assciated UID/GID's? do they?

I don't think you read Victor's message very carefully. A virtual domain
is different from a virtual mailbox domain. virtual_maps are used not
only for redirecting email from one address to one or more other
addresses, it is also used for virtual domains. virtual_mailbox_maps are
used for virtual mailbox domains and for mailboxes on the local computer
that do not have an associated Unix account. You are mixing the two, and
that is the cause of your problems.

----
Russell Mosemann, Ph.D. * Computing Services * Concordia University, Nebraska
"A penny saved is a penny earned." "Penny wise, pound foolish."
Post a follow-up to this message

Message 3 in thread
寄件者：Victor.Duchovni@morganstanley.com (Victor.Duchovni@morganstanley.com)
主旨：Re: Postifix & Virtual Domains

View this article only
新闻群组：mailing.postfix.users
日期：2002-11-26 19:12:51 PST

On Tue, 26 Nov 2002, James T. Richardson, Jr. wrote:

> Following instructions by Victor Duchovni
> (http://groups.google.com/groups?hl=en&lr=&ie=UTF-8&oe=UTF-8&selm=ajh5mv
> $h93$1@FreeBSD.csie.NCTU.edu.tw&rnum=1):
>
> > I call such domains *virtual-mailbox* domains. To define a virtual-mailbox
> > domain, just a "domain whatever" entry to one of the tables listed in
> > $virtual_mailbox_maps. For now it is also necessary to add the domain to
> > the transport table in order to arrange for mail to the domain to be
> > handled by a suitable virtual delivery agent (presumably but not
> > necessarily the Postfix "virtual" delivery agent).
>
> I created a simple hash (for now) files of "domain whatever" (and ran
> postmap on it) and stuck it on the end of virtual_mailbox_maps.
>
> Now, I'm not sure where to create these aliases or where to tell Postfix
> about them. I tried putting an alias to postmaster@domain underneath the
> "domain whatever" line in the virtual file, but that didn't work. I got
> a
> To=, relay=virtual, delay=0, status=deferred
> (recipient postmaster@domain: uid not found in virtual_uid_maps)
>

You actually searched for and found some documentation and mostly
followed it correctly! You deserve a break.

Your issue is that you are trying to specify address rewriting rules
("aliases" of sorts) for virtual mailbox domains in the
virtual_mailbox_maps tables. These tables define the pathname of the
virtual user's mailbox, not an alternate address.

For virtual mailbox users the only available address rewriting mechanism
is $virtual_maps. Don't put the "domain anything" keys for virtual
mailbox domainsthere, but do put address rewriting rules there.

virtual_mailbox_maps = hash:/etc/postfix/vmboxdomains, mysql:mumble
virtual_maps = mysql:fumble

/etc/postfix/vmboxdomains will have all the "domain anything" keys. List
it first, as these lookups will be very fast in comparison to mysql.

mysql:mumble will map virtual users to their mailbox path.
mysql:fumble (or hash:fumble) will map "aliased" virtual users to the
correct email address(es).

--
Viktor.

Posted by hzqbbc at 01:13 AM | Comments (0)

November 13, 2002

有关于mjt所写的tinycdb的技术内幕讨论

TinyCDB是一个对djb提出的CDB全新的实现，体积非常小，速度非常非常快。有俄罗斯人mjt开发。但它有一个致命的缺点，那就是每次更新/删除/添加都必须重建数据库（也因为不需要维护太多东西，所以才快）。

原文：
About TinyCDB

Message 1 in thread
寄件者：Michael Tokarev (mjt@tls.msk.ru)
主旨：[ot] tinyCDB is available in packaged form

View this article only
新闻群组：mailing.postfix.users
日期：2002-10-12 16:28:04 PST

After recent comparisions and mention of CDB map type, I received
several emails from several people about my tinycdb package. I've
made cosmetic changes in packaging scripts and uploaded binaries for
RedHat-like (rpm) and Debian Linux systems. It is available at
ftp://ftp.corpit.ru/pub/tinycdb/ :

21985 tinycdb-0.72.tar.gz source tarball incl. build scripts
26572 tinycdb-0.72-1.i386.rpm RedHat i386 rpm (rpm -tb)
24064 tinycdb_0.72_i386.deb Debian i386 .deb (native build)

This includes both the -lcdb library and simple cdb command-line
utility to work with .cdb files.

CDB map for postfix is available at ftp://ftp.corpit.ru/pub/postfix/,
see README.CDB in this directory.

/mjt

Message 2 in thread
寄件者：Wietse Venema (wietse@porcupine.org)
主旨：Re: [ot] tinyCDB is available in packaged form

View this article only
新闻群组：mailing.postfix.users
日期：2002-10-12 17:22:01 PST

Why does mkmap_cdb_open() lock the .cdb file?

Wietse

Michael Tokarev:
> After recent comparisions and mention of CDB map type, I received
> several emails from several people about my tinycdb package. I've
> made cosmetic changes in packaging scripts and uploaded binaries for
> RedHat-like (rpm) and Debian Linux systems. It is available at
> ftp://ftp.corpit.ru/pub/tinycdb/ :
>
> 21985 tinycdb-0.72.tar.gz source tarball incl. build scripts
> 26572 tinycdb-0.72-1.i386.rpm RedHat i386 rpm (rpm -tb)
> 24064 tinycdb_0.72_i386.deb Debian i386 .deb (native build)
>
> This includes both the -lcdb library and simple cdb command-line
> utility to work with .cdb files.
>
> CDB map for postfix is available at ftp://ftp.corpit.ru/pub/postfix/,
> see README.CDB in this directory.
>
> /mjt
>
> -
> To unsubscribe, send mail to majordomo@postfix.org with content
> (not subject): unsubscribe postfix-users
>
>

-
To unsubscribe, send mail to majordomo@postfix.org with content
(not subject): unsubscribe postfix-users
Message 3 in thread
寄件者：Michael Tokarev (mjt@tls.msk.ru)
主旨：Re: [ot] tinyCDB is available in packaged form

View this article only
新闻群组：mailing.postfix.users
日期：2002-10-12 18:32:02 PST

Wietse Venema wrote:
> Why does mkmap_cdb_open() lock the .cdb file?

Hmm, this is an interesting question - now, after more than a
year since that discussions, I actually see what did you mean
when asked similar question.

There are two things to protect by a lock: creating a map file
updating of a map - this last one in case of cdb consists of
two operations, namely creating and renaming. I.e., new (temp)
file may be protected during it's creation/writing so no two
processes will write to it at once, or the whole operation
may be protected, so no two proceses will (create+rename) at
once.

Or, in the other words, lock on .cdb file should mean "wait,
I'm working on this map", while a lock on .tmp file should
mean "wait, I'm working with this file".

There is a minor and somewhat unclean difference, but it
exists.

I don't know which is better - in fact, there is no real
difference in a result, it seems. Why? Because if the
only program who'll update a map will be postfix, there is
really no difference once all postfix's components are
consistent. But if there will be other programs, things
will be more "interesting": there is no "standard" choice
for a name of a temp file, one program may use .cdb.tmp
(as my mkmap_cdb), another may use .tmp, yet another may
use random-unique-name etc. No other tool I know of uses
any locking at all when creating cdb (there are only 2:
DJB's cdbmake and tinycdb's cdb -c).

Well, if some other program will actually do soome locking,
it's a good idea to choose the same locking scheme, and the
right one (from the CDB's ideology) seems to be to lock .tmp
file ("I'm working on this file").

But implementation is more difficult in this case (when locking
.tmp) - it's more tricky to avoid a race here and not to start
updating .cdb file instead of .tmp -- if memory serves me right;
it was long ago when I thought about this. This was the only
reason why I choosed to lock .cdb instead of .tmp when first
wrote the map. Anyway, it shouldn't be impossible ;)

BTW, the whole cdb thing, just like maildir format, seems to
be designed to avoid any and all locking altogether. For this,
simplest solution is to choose a random (unique) name of temp
file and forget about concurrency issues. This solution has
one implication (already noted by you) - it's possible to have
leftover files after crashes in this case.

/mjt

-
To unsubscribe, send mail to majordomo@postfix.org with content
(not subject): unsubscribe postfix-users
Message 4 in thread
寄件者：Matthias Andree (ma@dt.e-technik.uni-dortmund.de)
主旨：Re: [ot] tinyCDB is available in packaged form

View this article only
新闻群组：mailing.postfix.users
日期：2002-10-14 09:00:06 PST

Michael Tokarev writes:

> There are two things to protect by a lock: creating a map file
> updating of a map - this last one in case of cdb consists of
> two operations, namely creating and renaming. I.e., new (temp)
> file may be protected during it's creation/writing so no two
> processes will write to it at once, or the whole operation
> may be protected, so no two proceses will (create+rename) at
> once. ...
> BTW, the whole cdb thing, just like maildir format, seems to
> be designed to avoid any and all locking altogether.

Not really, but that's why you pass DJB's applications the name of a
temp file. Whether you let the user choose one or use mkstemp, is a
matter of convenience and portability.

> For this, simplest solution is to choose a random (unique) name of
> temp file and forget about concurrency issues. This solution has one
> implication (already noted by you) - it's possible to have leftover
> files after crashes in this case.

Do you think it would be possible to use mkstemp or something for the
new map file and rename(2) that into place after it is complete, and
dropping all locking code?

I'd not really worry about "two processes rename to the same .cdb file"
at all: let the last one to rename win. Usually, these .cdb files are
under administrator control and will actually be built from some input
by a Makefile and make or something.

You could just document the template you passed to mkstemp() and tell
users they can safely delete these files. Should this happen while your
makecdb is running at the same time, rename() will fail without touching
the destination file.

No need to be paranoid here.

--
Matthias Andree
-
To unsubscribe, send mail to majordomo@postfix.org with content
(not subject): unsubscribe postfix-users
Post a follow-up to this message

Message 5 in thread
寄件者：Victor.Duchovni@morganstanley.com (Victor.Duchovni@morganstanley.com)
主旨：Re: [ot] tinyCDB is available in packaged form

View this article only
新闻群组：mailing.postfix.users
日期：2002-10-14 10:12:01 PST

On Mon, 14 Oct 2002, Matthias Andree wrote:

> I'd not really worry about "two processes rename to the same .cdb file"
> at all: let the last one to rename win. Usually, these .cdb files are
> under administrator control and will actually be built from some input
> by a Makefile and make or something.
>

I don't think this is right. When I run two concurrent copies of "postmap
-i" I expect both sets of key/value pairs to be added to the map.

Does the cdb "mkmap" code support incremental inserts or only full map
rebuilds?

--
Viktor.

-
To unsubscribe, send mail to majordomo@postfix.org with content
(not subject): unsubscribe postfix-users
Post a follow-up to this message

Message 6 in thread
寄件者：Michael Tokarev (mjt@tls.msk.ru)
主旨：Re: [ot] tinyCDB is available in packaged form

View this article only
新闻群组：mailing.postfix.users
日期：2002-10-14 12:28:07 PST

Victor.Duchovni@morganstanley.com wrote:
[]
> I don't think this is right. When I run two concurrent copies of "postmap
> -i" I expect both sets of key/value pairs to be added to the map.
>
> Does the cdb "mkmap" code support incremental inserts or only full map
> rebuilds?

No - this is a principal limitation of a cdb, it's a Constant database.
I.e. any and all manipulation with content requires complete rebuild.
Well, not strictly this - updates are possible by rebuilding the whole
hash table (inplace) or by recreating a file using old file as a source,
but that's not worth the effort to program the whole thing IMHO. For
this very reason, cdb isn't sutable for pop-before-smtp and the like
dynamic maps.

Mkmap code in my cdb patch does not support incremental updates, again,
for the same reason (it will be complete rebuild anyway).

/mjt

-
To unsubscribe, send mail to majordomo@postfix.org with content
(not subject): unsubscribe postfix-users
Post a follow-up to this message

Message 7 in thread
寄件者：Bennett Todd (bet@rahul.net)
主旨：Re: [ot] tinyCDB is available in packaged form

View this article only
新闻群组：mailing.postfix.users
日期：2002-10-15 12:24:05 PST

--n8g4imXOkfNTN/H1
Content-Type: text/plain; charset=us-ascii
Content-Disposition: inline

2002-10-14-15:24:50 Michael Tokarev:
> No - this is a principal limitation of a cdb, it's a Constant
> database. I.e. any and all manipulation with content requires
> complete rebuild. [...] For this very reason, cdb isn't sutable
> for pop-before-smtp and the like dynamic maps.

Not to dispute any of the other stuff you said, but I'm really not
sure about this bit.

I'd have to do some experimenting, but I really suspect
pop-before-smtp would work very well indeed with cdb.

It might be worth taking the trouble to have a version of cdb -c
inlined into the daemon, perhaps make a dynamically loadable perl
module if I were doing this on my pop-before-smtp daemon. That'd
save the need for fork-n-exec for each db update; if you ain't on
Linux, fork-n-exec is overpriced:-).

But with an in-memory cdb create, the update would be
open-tmp-file/write/close/rename, and for reasonable size maps ---
many folks don't have that many concurrent poppers from outside
mynetworks --- I'm sure this would be cheaper than
open-db/lock/read/write/flush/unlock/close. In fact, I wouldn't be
surprised if the break-even point for an optimized cdb-writer
weren't actually pretty darned large. cdb is awfully quick.

Hmm. Now I'm gonna have to find time to play with CDB_File.

-Bennett

--n8g4imXOkfNTN/H1
Content-Type: application/pgp-signature
Content-Disposition: inline

-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.0.7 (GNU/Linux)

iD8DBQE9rF82HZWg9mCTffwRAuGhAJ9×4iqOrAmlmsXdy0QpPHFFFFaDCgCgjumd
Dpbm/WiCJre7+sXb9TCRJ/g=
=AMc7
-----END PGP SIGNATURE-----

n8g4imXOkfNTN/H1
-
To unsubscribe, send mail to majordomo@postfix.org with content
(not subject): unsubscribe postfix-users
Post a follow-up to this message

Message 8 in thread
寄件者：Michael Tokarev (mjt@tls.msk.ru)
主旨：Re: [ot] tinyCDB is available in packaged form

View this article only
新闻群组：mailing.postfix.users
日期：2002-10-15 12:46:02 PST

Bennett Todd wrote:
> 2002-10-14-15:24:50 Michael Tokarev:
>
>>No - this is a principal limitation of a cdb, it's a Constant
>>database. I.e. any and all manipulation with content requires
>>complete rebuild. [...] For this very reason, cdb isn't sutable
>>for pop-before-smtp and the like dynamic maps.
>
>
> Not to dispute any of the other stuff you said, but I'm really not
> sure about this bit.
>
> I'd have to do some experimenting, but I really suspect
> pop-before-smtp would work very well indeed with cdb.
>
> It might be worth taking the trouble to have a version of cdb -c
> inlined into the daemon, perhaps make a dynamically loadable perl
> module if I were doing this on my pop-before-smtp daemon. That'd
> save the need for fork-n-exec for each db update; if you ain't on
> Linux, fork-n-exec is overpriced:-).
>
> But with an in-memory cdb create, the update would be
> open-tmp-file/write/close/rename, and for reasonable size maps ---
> many folks don't have that many concurrent poppers from outside
> mynetworks --- I'm sure this would be cheaper than
> open-db/lock/read/write/flush/unlock/close. In fact, I wouldn't be
> surprised if the break-even point for an optimized cdb-writer
> weren't actually pretty darned large. cdb is awfully quick.

One word: scalability. With a dynamically updateable map, a
time needed to insert one record does not depend on the number
of records already present in a map (well, almost). Cdb
requires complete rewrite. It will work for small sites.
Also, create/rename requires filesystem metadata updates,
while read/write does not - thus, depending on a filesystem,
situation with cdb will be worse. But it has one advantage:
it will work well over NFS, so it will be possible to use
PBS when pop daemon isn't on the same machine as smtp.

In fact, the best way to implement PBS is to keep all records
in memory (8 bytes for each (IP and time) isn't that much)
and use tcp map in postfix (and this too will work good
on a large site). Crashes/reboots aren't that important
here, it's almost ok to lose data since it isn't keept
for a long time anyway (just "rePOP" will be required).
With such a method, far more efficient search algorithm
may be implemented (i.e. sorted array of IP addresses,
maybe hashed).

> Hmm. Now I'm gonna have to find time to play with CDB_File.

I do NOT recommend this, at least w/o fixing things first.
Cdbmake code has at least one (huge) memory leak and thus
isn't sutable for a long-living daemon. Also, cdbmake code
requires much more memory than tinycdb, and is less
efficient (this all is true for cdb-0.75, I not looked
if new version was released).

What I want to is to find a time to write perl and especially
nss module for cdb - this is on a todo list for more than a
year now... ;)

/mjt

-
To unsubscribe, send mail to majordomo@postfix.org with content
(not subject): unsubscribe postfix-users
寄件者：Bennett Todd (bet@rahul.net)
主旨：Re: [ot] tinyCDB is available in packaged form

View this article only
新闻群组：mailing.postfix.users
日期：2002-10-16 09:40:03 PST

--kXdP64Ggrk/fb43R
Content-Type: text/plain; charset=us-ascii
Content-Disposition: inline
Content-Transfer-Encoding: quoted-printable

2002-10-15-15:43:50 Michael Tokarev:
> Bennett Todd wrote:
> >I'd have to do some experimenting, but I really suspect
> >pop-before-smtp would work very well indeed with cdb.
>=20
> One word: scalability. With a dynamically updateable map, a
> time needed to insert one record does not depend on the number
> of records already present in a map (well, almost). Cdb
> requires complete rewrite. It will work for small sites.

Define "small".

I seriously believe that a large majority of pop-before-smtp users
rarely have more than a few records in their database; and that
sites with as many as a hundred such records much of the time are
liable to be a teensy minority, if not a non-existent limit case.

Where's the crossover point? I.e. where does the cost of the CDB
generation grow to exceed the cost of doing the DB manipulation?

This undoubtedly varies from platform to platform. I'll try and
produce an answer for at least one platform:-).

> Also, create/rename requires filesystem metadata updates,
> while read/write does not - thus, depending on a filesystem,
> situation with cdb will be worse.

I'll try it with ext3, and with tmpfs. But while read/write may not
require metadata updates, it does require locking. I don't know
what sort of difference that will make, if any.

> In fact, the best way to implement PBS is to keep all records
> in memory (8 bytes for each (IP and time) isn't that much)
> and use tcp map in postfix (and this too will work good
> on a large site).

Sounds like we're converging on DRAC:-).

> >Hmm. Now I'm gonna have to find time to play with CDB_File.
>=20
> I do NOT recommend this, at least w/o fixing things first.
> Cdbmake code has at least one (huge) memory leak and thus
> isn't sutable for a long-living daemon.
观看文件全部内容（仍有 22 行）

Post a follow-up to this message

Message 10 in thread
寄件者：Matthias Andree (ma@dt.e-technik.uni-dortmund.de)
主旨：Re: [ot] tinyCDB is available in packaged form

View this article only
新闻群组：mailing.postfix.users
日期：2002-10-21 06:25:57 PST

Bennett Todd writes:

> I'd have to do some experimenting, but I really suspect
> pop-before-smtp would work very well indeed with cdb.

pop-before-smtp is inherently insecure, no matter HOW you implement
it. The problem is that you give relay permission to a host that may
well be "replaced" by some other (1st hosts disconnects, 2nd is assigned
the previous IP from 1st host from the dialup pool), without your SMTP
server noticing the disconnect. SMTP AUTH does not suffer from these
problems and does not need all these expire cronjobs and tossing data
from POP server to SMTP server and the like, worrying about how you
update a cdb... A cdb does not cut incremental updates. It would cope
well with a passwd like file though.

--
Matthias Andree
-
To unsubscribe, send mail to majordomo@postfix.org with content
(not subject): unsubscribe postfix-users

寄件者：Bennett Todd (bet@rahul.net)
主旨：Re: [ot] tinyCDB is available in packaged form

View this article only
新闻群组：mailing.postfix.users
日期：2002-10-21 07:44:59 PST

--gTtJ75FAzB1T2CN6
Content-Type: text/plain; charset=us-ascii
Content-Disposition: inline

2002-10-21-08:59:40 Matthias Andree:
> pop-before-smtp is inherently insecure, no matter HOW you implement
> it.

Lightly so, yes.

> The problem is that you give relay permission to a host that may
> well be "replaced" by some other (1st hosts disconnects, 2nd is assigned
> the previous IP from 1st host from the dialup pool), without your SMTP
> server noticing the disconnect.

That's why there's a timeout. A default of 1/2 hour seems to be
fine. Yes, there's a window during which someone else who got the IP
addr of a recently-pop-authed machine could relay. No, it doesn't
appear to be a problem in practice; I've never heard of such a setup
actually being exploited through this timing window.

> SMTP AUTH does not suffer from these problems [...]

If SMTP AUTH didn't require SASL, or if there were an implementation
of SASL besides Cyrus, I might be more interested. Is it possible to
implement enough SASL for SMTP AUTH without bringing in mountains of
excruciatingly painful goo, like e.g. GSSAPI with it's ASN.1 and so
forth?

> [...] and does not need all these expire cronjobs and tossing data
> from POP server to SMTP server and the like, [...]

I don't have any expire cronjobs, I just have a single persistent
daemon that manages the data file. Works great for a single server
that's both the pop/imap mailbox server and the smtp relay server.
For server farms, something more like DRAC or whoson would probably
be in order.

> worrying about how you update a cdb... A cdb does not cut
> incremental updates.

Sure it does, it does so beautifully. cdb file rebuild is so fast
that doing a full rebuild for each update is practical in many
settings --- as DNS hosting sites have discovered when using
tinydns.

-Bennett

--gTtJ75FAzB1T2CN6
Content-Type: application/pgp-signature
Content-Disposition: inline

-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.0.7 (GNU/Linux)

iD8DBQE9tA7UHZWg9mCTffwRAhs7AKDEw1uALoCxjHAhlSJA0xic+2hfTACgm6a2
0BJdYgRUKo90nwQlSFdkK24=
=XNg1
-----END PGP SIGNATURE-----

gTtJ75FAzB1T2CN6
-
To unsubscribe, send mail to majordomo@postfix.org with content
(not subject): unsubscribe postfix-users
Post a follow-up to this message

Message 12 in thread
寄件者："Schmehl, Paul L" (pauls@utdallas.edu)
主旨：RE: [ot] tinyCDB is available in packaged form

View this article only
新闻群组：mailing.postfix.users
日期：2002-10-21 08:30:04 PST

> -----Original Message-----
> From: Bennett Todd [mailto:bet@rahul.net]
> Sent: Monday, October 21, 2002 9:28 AM
> To: Matthias Andree
> Cc: postfix-users@postfix.org
> Subject: Re: [ot] tinyCDB is available in packaged form
>
>
> 2002-10-21-08:59:40 Matthias Andree:
> > pop-before-smtp is inherently insecure, no matter HOW you implement
> > it.
>
> Lightly so, yes.
>
> > The problem is that you give relay permission to a host that may well
> > be "replaced" by some other (1st hosts disconnects, 2nd is assigned
> > the previous IP from 1st host from the dialup pool), without your SMTP
> > server noticing the disconnect.
>
> That's why there's a timeout. A default of 1/2 hour seems to
> be fine. Yes, there's a window during which someone else who
> got the IP addr of a recently-pop-authed machine could relay.
> No, it doesn't appear to be a problem in practice; I've never
> heard of such a setup actually being exploited through this
> timing window.

There's quite an assumption behind this claim of insecurity. The
assumption is:

1) User one connects using pop before SMTP (or DRAC), reads mail and
disconnects before the timeout.
2) User two knows he a) has user one's IP address and knows b) the
IP/ hostname of the mail server and knows c) the server is open for
relay and intends d) to send spam through the server.

This assumption requires some incredible leaps in logic. Is it
possible? Of course. Just as breaking a 4096 AES key is possible. Is
it likely? Not in this lifetime. Even if I were a spammer, and even if
I knew that a particular server used pop before SMTP or DRAC and even
if I knew someone who used that server was online and using the server
and even if I knew precisely when they disconnected, I would still
have to obtain the exact same IP address to exploit the server.

I have real problems thinking of a scenario where this is even remotely
possible.

Paul Schmehl (pauls@utdallas.edu)
TCS Department Coordinator
The University of Texas at Dallas
AVIEN Founding Member
http://www.utdallas.edu/~pauls/
-
To unsubscribe, send mail to majordomo@postfix.org with content
(not subject): unsubscribe postfix-users
Post a follow-up to this message

Message 13 in thread
寄件者：Simon White (simon@mtds.com)
主旨：Re: [ot] tinyCDB is available in packaged form

View this article only
新闻群组：mailing.postfix.users
日期：2002-10-21 08:32:05 PST

21-Oct-02 at 10:19, Schmehl, Paul L (pauls@utdallas.edu) wrote :

> 1) User one connects using pop before SMTP (or DRAC), reads mail and
> disconnects before the timeout.
> 2) User two knows he a) has user one's IP address and knows b) the
> IP/ hostname of the mail server and knows c) the server is open for
> relay and intends d) to send spam through the server.
>
> This assumption requires some incredible leaps in logic. Is it
> possible? Of course. Just as breaking a 4096 AES key is possible. Is
> it likely? Not in this lifetime. Even if I were a spammer, and even if
> I knew that a particular server used pop before SMTP or DRAC and even
> if I knew someone who used that server was online and using the server
> and even if I knew precisely when they disconnected, I would still
> have to obtain the exact same IP address to exploit the server.
>
> I have real problems thinking of a scenario where this is even remotely
> possible.

Unless you have ISDN users with a fixed IP address : but then you'd have
to sniff their dialup username and password into the bargain.

--
[Simon White. vim/mutt. simon@mtds.com. GIMPS:41.94% see www.mersenne.org]
/"\ ASCII Ribbon Campaign
\ / Respect for open standards
X No HTML/RTF in email
/ \ No M$ Word docs in email
-
To unsubscribe, send mail to majordomo@postfix.org with content
(not subject): unsubscribe postfix-users
Post a follow-up to this message

Message 14 in thread
寄件者：Michael Tokarev (mjt@tls.msk.ru)
主旨：Re: [ot] tinyCDB is available in packaged form

View this article only
新闻群组：mailing.postfix.users
日期：2002-10-21 09:44:04 PST

Matthias Andree wrote:
[]
> pop-before-smtp is inherently insecure, no matter HOW you implement
> it. The problem is that you give relay permission to a host that may
> well be "replaced" by some other (1st hosts disconnects, 2nd is assigned
> the previous IP from 1st host from the dialup pool), without your SMTP
> server noticing the disconnect.

Mattias, this is not an issue at all with current world. Any given
(dialup) host has several minutes (hours? maybe days?) to find out
which mailservers was used by a host that has this IP previously,
from about 2^32 other hosts on the 'net. This is not possible in
that timeframe, and there are way far simpler ways exists to find
other open relays. Granted, if such PBS records will be permanent,
at some time a spammer eventually will find a relay, but that's a)
not the case (records aren't permanent), and b) chances are very
low anyway. Having a timeout of about an hour makes zero chances
for spammers to reuse the "relay". Again, in some cases, PBS is
somewhat insecure - think of common well-known public mail
systems like e.g. mail.com (a dialup spammer may try to send out
spam via mail.com just after dialing in to see if some previous
user used mail.com from this IP address recently), but again,
such cases are rare, and public mail systems should use other
protection methods as well (i.e. limiting number of recipients
in a time period, have smaller timeout for PBS etc).

/mjt

-
To unsubscribe, send mail to majordomo@postfix.org with content
(not subject): unsubscribe postfix-users
Post a follow-up to this message

Message 15 in thread
寄件者："Devin L. Ganger" (devin@thecabal.org)
主旨：Re: [ot] tinyCDB is available in packaged form

View this article only
新闻群组：mailing.postfix.users
日期：2002-10-21 13:20:05 PST

On Mon, Oct 21, 2002 at 10:19:04AM -0500, Schmehl, Paul L wrote:

> There's quite an assumption behind this claim of insecurity. The
> assumption is:
>
> 1) User one connects using pop before SMTP (or DRAC), reads mail and
> disconnects before the timeout.

True.

> 2) User two knows he a) has user one's IP address and knows b) the
> IP/ hostname of the mail server and knows c) the server is open for
> relay and intends d) to send spam through the server.

False. All user two has to do is attempt to relay. I wouldn't be at
all surprised to find that as more and more ISPs start protecting
their servers from relay, we're going to see more and more examples
of this from whackamole accounts.

--
Devin L. Ganger
Co-Admin, The cabalSASL Project ( http://sasl.thecabal.org/ )
A man, a miss, a car -- a curve,
He kissed the miss and missed the curve -- Burma Shave (1948)
-
To unsubscribe, send mail to majordomo@postfix.org with content
(not subject): unsubscribe postfix-users
Post a follow-up to this message

Message 16 in thread
寄件者："Schmehl, Paul L" (pauls@utdallas.edu)
主旨：RE: [ot] tinyCDB is available in packaged form

View this article only
新闻群组：mailing.postfix.users
日期：2002-10-21 14:16:04 PST

Why do you say this? If I attempt to relay through a server that uses
pbfs, it will fail - unless I am using the correct IP. So I not only
have to have the mail server's address, but I also must "own" the IP
address that's authorized to relay - or do you know something no one
else does about pbfs?

You can try relaying all day long, but unless you have the IP, you have
nothing. If you have the IP, then you must also be trying to relay
through that mail server. No other server will do. You make it sound
as though it's trivial to do. I don't see that at all.

Perhaps you could elaborate further?

Paul Schmehl (pauls@utdallas.edu)
TCS Department Coordinator
The University of Texas at Dallas
AVIEN Founding Member
http://www.utdallas.edu/~pauls/

> -----Original Message-----
> From: Devin L. Ganger [mailto:devin@thecabal.org]
> Sent: Monday, October 21, 2002 3:11 PM
> To: postfix-users@postfix.org
> Subject: Re: [ot] tinyCDB is available in packaged form
>
>
> On Mon, Oct 21, 2002 at 10:19:04AM -0500, Schmehl, Paul L wrote:
>
> > There's quite an assumption behind this claim of insecurity. The
> > assumption is:
> >
> > 1) User one connects using pop before SMTP (or DRAC), reads mail and
> > disconnects before the timeout.
>
> True.
>
> > 2) User two knows he a) has user one's IP address and knows b) the
> > IP/ hostname of the mail server and knows c) the server is open for
> > relay and intends d) to send spam through the server.
>
> False. All user two has to do is attempt to relay. I
> wouldn't be at all surprised to find that as more and more
> ISPs start protecting their servers from relay, we're going
> to see more and more examples of this from whackamole accounts.
-
To unsubscribe, send mail to majordomo@postfix.org with content
(not subject): unsubscribe postfix-users
Post a follow-up to this message

Message 17 in thread
寄件者：Simon White (simon@mtds.com)
主旨：Re: [ot] tinyCDB is available in packaged form

View this article only
新闻群组：mailing.postfix.users
日期：2002-10-21 15:46:02 PST

21-Oct-02 at 13:10, Devin L. Ganger (devin@thecabal.org) wrote :
> On Mon, Oct 21, 2002 at 10:19:04AM -0500, Schmehl, Paul L wrote:
>
> > There's quite an assumption behind this claim of insecurity. The
> > assumption is:
> >
> > 1) User one connects using pop before SMTP (or DRAC), reads mail and
> > disconnects before the timeout.
>
> True.
>
> > 2) User two knows he a) has user one's IP address and knows b) the
> > IP/ hostname of the mail server and knows c) the server is open for
> > relay and intends d) to send spam through the server.
>
> False. All user two has to do is attempt to relay. I wouldn't be at
> all surprised to find that as more and more ISPs start protecting
> their servers from relay, we're going to see more and more examples
> of this from whackamole accounts.
>

Indeed, if a spammer already has an inkling of a dialup subnet range
with the same domain name, it is quite likely that he will already have
been able to send spam through an open proxy which is purely a poor SME
connection sharing thing with some Exchange administrator who wouldn't
even pass an MCSE... although Microsoft have made some progress in this
direction there's a cost to upgrade unlike Open Source which offers free
upgrades in return for a bit of clue and your time.

Then he can regularly attempt to find a connection with an IP in the
POP-before-SMTP database but this is of little use to him: he might more
easily forge something and get past a poor server rather than look for
30 minute windows in which to inject as much spam as possible. With a
solid mailserver setup as any self-respecting mail admin should have,
this is less /and more!/ of a problem.

Sadly, there are more people in this world interested in finding mugs,
and they do find them, so spam is here to stay. As much as you'd have
many avenues to explore before feeling obliged to hack enough just to
get a maximum of one half hour of time, an admin proud of his Postfix
server which could (and has done, on my site, once...) deliver something
like 15,000 messages in a short period of time, on a reasonably fat pipe
to the Internet, might think that he is indeed prime spam territory. Of
course this is all relative because here fat is 2mbps!

Anyway anecdotes aside we're into philosophical ground really. The truly
paranoid may never run POP-before-SMTP but a good admin can feel solid
with that kind of system installed. Heck, it's probably reasonably
secure.

--
[Simon White. vim/mutt. simon@mtds.com. GIMPS:42.88% see www.mersenne.org]
Recognizing disagreements in belief requires having enough agreements in
belief to translate or understand the words and deeds of my opponent.
-- Anthony O'Hear (combining, somewhat, several modern philosophers).
-
To unsubscribe, send mail to majordomo@postfix.org with content
(not subject): unsubscribe postfix-users
Post a follow-up to this message

Message 18 in thread
寄件者：Victor.Duchovni@morganstanley.com (Victor.Duchovni@morganstanley.com)
主旨：Re: [ot] tinyCDB is available in packaged form

View this article only
新闻群组：mailing.postfix.users
日期：2002-10-21 16:12:05 PST

On Mon, 21 Oct 2002, Simon White wrote:

> Indeed, if a spammer already has an inkling of a dialup subnet range
> with the same domain name, it is quite likely that he will already have
> been able to send spam through an open proxy which is purely a poor SME
> connection sharing thing with some Exchange administrator who wouldn't
> even pass an MCSE... although Microsoft have made some progress in this
> direction there's a cost to upgrade unlike Open Source which offers free
> upgrades in return for a bit of clue and your time.
>

Pop before SMTP is designed to allow a user using ISP A to relay with the
mailhub of ISP B. ISP A does not need POP before SMTP to relay for their
own users, that's what permit_mynetworks is for.

The probability claims are plausible, one needs to know which third-party
dialup pool the user used to relay mail via their ISP.

Even if one can entice the user to send the attacker a message and hang
up, one then still needs to dial-in into the same IP in the dialup-pool.

If has a user account for the dialup pool, presumably one automatically
gets to relay through the mail relays of the pool provider.

--
Viktor.

-
To unsubscribe, send mail to majordomo@postfix.org with content
(not subject): unsubscribe postfix-users
Post a follow-up to this message

Message 19 in thread
寄件者："Devin L. Ganger" (devin@thecabal.org)
主旨：Re: [ot] tinyCDB is available in packaged form

View this article only
新闻群组：mailing.postfix.users
日期：2002-10-21 18:18:03 PST

On Mon, Oct 21, 2002 at 10:42:32PM +0000, Simon White wrote:

> Anyway anecdotes aside we're into philosophical ground really. The truly
> paranoid may never run POP-before-SMTP but a good admin can feel solid
> with that kind of system installed. Heck, it's probably reasonably
> secure.

Reasonably, yes. However, security is not a magic formula. You have
to understand the strengths and weaknesses of each of the various
pieces you use so that you use them intelligently.

I didn't want statements that could have appeared to place pbfs as some
sort of magic sovereign specific to go without also pointing out that
it does, under certain conditions (likely to be found in smaller ISPs
with limited address space) have drawbacks you need to be aware of.

Message 20 in thread
寄件者："Devin L. Ganger" (devin@thecabal.org)
主旨：Re: [ot] tinyCDB is available in packaged form

View this article only
新闻群组：mailing.postfix.users
日期：2002-10-21 18:18:03 PST

On Mon, Oct 21, 2002 at 07:05:21PM -0400,
Victor.Duchovni@morganstanley.com wrote:

> Pop before SMTP is designed to allow a user using ISP A to relay with the
> mailhub of ISP B. ISP A does not need POP before SMTP to relay for their
> own users, that's what permit_mynetworks is for.

Yes, but there are ISPs that are using it for precisely that purpose,
despite the design goals because they think it's somehow more secure.

If you are one of them, be aware of the consequences.

寄件者：Victor.Duchovni@morganstanley.com (Victor.Duchovni@morganstanley.com)
主旨：Re: [ot] tinyCDB is available in packaged form

View this article only
新闻群组：mailing.postfix.users
日期：2002-10-21 19:59:40 PST

On Mon, 21 Oct 2002, Devin L. Ganger wrote:

> On Mon, Oct 21, 2002 at 07:05:21PM -0400,
> Victor.Duchovni@morganstanley.com wrote:
>
> > Pop before SMTP is designed to allow a user using ISP A to relay with the
> > mailhub of ISP B. ISP A does not need POP before SMTP to relay for their
> > own users, that's what permit_mynetworks is for.
>
> Yes, but there are ISPs that are using it for precisely that purpose,
> despite the design goals because they think it's somehow more secure.
>
> If you are one of them, be aware of the consequences.
>

Yes, but presumably they require passwords for establishing PPP sessions
with their own dialup pool. So if the user is dialing into their dialup
pool, POP before SMTP is at least as strong as (though not much stonger
than) simply letting the user relay, and if the user dials up into someone
else's dialup pool the previous analysis holds, the spammer does not know
whose relay to try.

Either way POP before SMTP is safe.

--
Viktor.

-
To unsubscribe, send mail to majordomo@postfix.org with content
(not subject): unsubscribe postfix-users
Post a follow-up to this message

Message 22 in thread
寄件者：lst_hoe (lst_hoe@kwsoft.de)
主旨：Re: [ot] tinyCDB is available in packaged form

View this article only
新闻群组：mailing.postfix.users
日期：2002-10-22 02:08:06 PST

At 22:45 21.10.2002 -0400, Victor.Duchovni@morganstanley.com wrote:
>On Mon, 21 Oct 2002, Devin L. Ganger wrote:
>
>> On Mon, Oct 21, 2002 at 07:05:21PM -0400,
>> Victor.Duchovni@morganstanley.com wrote:
>>
>> > Pop before SMTP is designed to allow a user using ISP A to relay with the
>> > mailhub of ISP B. ISP A does not need POP before SMTP to relay for their
>> > own users, that's what permit_mynetworks is for.
>>
>> Yes, but there are ISPs that are using it for precisely that purpose,
>> despite the design goals because they think it's somehow more secure.
>>
>> If you are one of them, be aware of the consequences.
>>
>
>Yes, but presumably they require passwords for establishing PPP sessions
>with their own dialup pool. So if the user is dialing into their dialup
>pool, POP before SMTP is at least as strong as (though not much stonger
>than) simply letting the user relay, and if the user dials up into someone
>else's dialup pool the previous analysis holds, the spammer does not know
>whose relay to try.
>
>Either way POP before SMTP is safe.

The only real hole i see is with (bigger) NATed Networks. If one user
fetches his mail by POP3 all others are able to relay. With a little bit of
glue they will know this and getting the provider by personal talk should
not be a problem.
But this is a clear fact of unsecure (Company-) Network, although this is
common with DSL LANs today.

Regards

Andreas H?dle

Kühn & Weyh Software GmbH

WWW.KWSOFT.DE

-
To unsubscribe, send mail to majordomo@postfix.org with content
(not subject): unsubscribe postfix-users
Post a follow-up to this message

Message 23 in thread
寄件者：Simon White (simon@mtds.com)
主旨：Re: [ot] tinyCDB is available in packaged form

View this article only
新闻群组：mailing.postfix.users
日期：2002-10-22 02:28:39 PST

21-Oct-02 at 19:05, Victor.Duchovni@morganstanley.com (Victor.Duchovni@morganstanley.com) wrote :
> On Mon, 21 Oct 2002, Simon White wrote:
>
> > Indeed, if a spammer already has an inkling of a dialup subnet range
> > with the same domain name, it is quite likely that he will already have
> > been able to send spam through an open proxy which is purely a poor SME
> > connection sharing thing with some Exchange administrator who wouldn't
> > even pass an MCSE... although Microsoft have made some progress in this
> > direction there's a cost to upgrade unlike Open Source which offers free
> > upgrades in return for a bit of clue and your time.
> >

I am hinting, here, at the probability that a large dialup pool IP
address range is probably more interesting to scan for open relays /
proxies than for a posisble POP-before-SMTP exploit.

I was thinking of IP spoofing attacks in the first place; if you have to
call the ISP to get access to the right IP, you're probably in
permit_mynetworks anyway. But what if you are NATted on a large private
IP dialup subnet - with one public address which is in the
POP-before-SMTP database?

> The probability claims are plausible, one needs to know which third-party
> dialup pool the user used to relay mail via their ISP.

Exactly. This is what is reasonably hard.

> Even if one can entice the user to send the attacker a message and hang
> up, one then still needs to dial-in into the same IP in the dialup-pool.

Unless the system allows you to spoof the connections, or if you are
behind a NAT IP that is also the NAT IP of many others.

> If has a user account for the dialup pool, presumably one automatically
> gets to relay through the mail relays of the pool provider.

Precisely. That's why mynetworks should never make it into the pbsmtp
database.

--
[Simon White. vim/mutt. simon@mtds.com. GIMPS:44.31% see www.mersenne.org]
The only reason I'm burning my candle at both ends, is because I haven't
figured out how to light the middle yet.
[Linux user #170823 http://counter.li.org. Home cooked signature rotator.]
-
To unsubscribe, send mail to majordomo@postfix.org with content
(not subject): unsubscribe postfix-users
Post a follow-up to this message

Message 24 in thread
寄件者：Ralf Hildebrandt (Ralf.Hildebrandt@charite.de)
主旨：Re: [ot] tinyCDB is available in packaged form

View this article only
新闻群组：mailing.postfix.users
日期：2002-10-23 07:40:19 PST

On Wed, Oct 23, 2002 at 03:58:05PM +0200, Matthias Andree wrote:

> 30 minutes is way too much. 3 might be an option. SMTP AUTH is
> implemented in most clients with GUI, so there is no compelling reason
> to use RELAY-after-POP3 instead of SMTP AUTH. Cyrus-SASL might be one.

I'd go or 10, because that's the default in Mozilla/Netscape

--
Ralf Hildebrandt Ralf.Hildebrandt@charite.de
Postfix Tips: http://www.arschkrebs.de/postfix/ Tel. +49 (0)30-450 570-155
Microsoft: A Proven Danger to National Security
http://www.infowarrior.org/articles/msdanger.pdf
-
To unsubscribe, send mail to majordomo@postfix.org with content
(not subject): unsubscribe postfix-users
Post a follow-up to this message

Message 25 in thread
寄件者：Matthias Andree (ma@dt.e-technik.uni-dortmund.de)
主旨：Re: [ot] tinyCDB is available in packaged form

View this article only
新闻群组：mailing.postfix.users
日期：2002-10-23 08:20:11 PST

Bennett Todd writes:

> 2002-10-21-08:59:40 Matthias Andree:
>> pop-before-smtp is inherently insecure, no matter HOW you implement
>> it.
>
> Lightly so, yes.

Is there "lightly pregnant" as well?

> That's why there's a timeout. A default of 1/2 hour seems to be
> fine. Yes, there's a window during which someone else who got the IP
> addr of a recently-pop-authed machine could relay. No, it doesn't
> appear to be a problem in practice; I've never heard of such a setup
> actually being exploited through this timing window.
> If SMTP AUTH didn't require SASL, or if there were an implementation
> of SASL besides Cyrus, I might be more interested. Is it possible to
> implement enough SASL for SMTP AUTH without bringing in mountains of
> excruciatingly painful goo, like e.g. GSSAPI with it's ASN.1 and so
> forth?

I must amend my statement: if there is a reliable way to detect when the
client has gone offline (session termination notification,
expiry of relay permission via radius), then these points don't hold;
OTOH, if you have radius, you can omit the "POP3-before-" part.

30 minutes is way too much. 3 might be an option. SMTP AUTH is
implemented in most clients with GUI, so there is no compelling reason
to use RELAY-after-POP3 instead of SMTP AUTH. Cyrus-SASL might be one.

I pinged Brian Stafford who has a partial SASL library (LGPL) as part of
his libesmtp to see if he'd work together with Wietse to make a more
secure replacement for Cyrus-SASL, for Postfix authentication
purposes. I've not seen code yet though.

> I don't have any expire cronjobs, I just have a single persistent
> daemon that manages the data file. Works great for a single server
> that's both the pop/imap mailbox server and the smtp relay server.

OK.

> Sure it does, it does so beautifully. cdb file rebuild is so fast
> that doing a full rebuild for each update is practical in many
> settings --- as DNS hosting sites have discovered when using
> tinydns.

Well, it's ok for you, but if you have thousands (because you have a
long timeout) of clients, then it won't scale well and BerkeleyDB will
be faster.

寄件者：Bennett Todd (bet@rahul.net)
主旨：Re: [ot] tinyCDB is available in packaged form
View: Complete Thread (31 articles)
Original Format
新闻群组：mailing.postfix.users
日期：2002-10-23 13:38:03 PST

--DBIVS5p969aUjpLe
Content-Type: multipart/mixed; boundary="uAKRQypu60I7Lcqm"
Content-Disposition: inline

--uAKRQypu60I7Lcqm
Content-Type: text/plain; charset=us-ascii
Content-Disposition: inline
Content-Transfer-Encoding: quoted-printable

2002-10-16-12:18:58 Bennett Todd:
> 2002-10-15-15:43:50 Michael Tokarev:
> > Bennett Todd wrote:
> > >I'd have to do some experimenting, but I really suspect
> > >pop-before-smtp would work very well indeed with cdb.
> >=20
> > One word: scalability. [...] It will work for small sites.
>=20
> Define "small".
> [...]
> Where's the crossover point? I.e. where does the cost of the CDB
> generation grow to exceed the cost of doing the DB manipulation?

Ok, in the interests of fairness, I've gotta come back and confess
that my intuition was way, way off here, and Michael Tokarev's was
right on.

While cdb can work for small sites, where the load of doing things
inefficiently doesn't make any difference anyway --- just what he
suggested --- there's no "crossover point"; no matter how small
the database, DB_Hash is faster than CDB_File for "updates" in all
circumstances.

On ext3fs:

CDBDB
107341797
1006621894
10005811372
10000133772

and on tmpfs:

CDBDB
10103406240060
10032839198009
10003513121343
1000034892536

The first column of these tables is the number of entries in the
table, where an entry is a key that looks like an IP address in
ASCII, four dotted quads randomly generated in [0..255], while the
value is "ok".

The numbers under the columns labelled CDB and DB are the number
of add/remove pair transactions completed in 30 seconds; for CDB
there's one full rebuild for each such transaction, while for DB
each such add/remove pair is surrounded by flocking, with a sync
right before the LOCK_UN. This mimics the traffic patterns seen with
pop-before-smtp, with some simplifying assumptions:-).

I used the attached program to generate that output, killing it
when the bad news was painfully obvious; rather than hassling with
emptying /tmp to overmount, I just edited the script to use /mnt for
the tmpfile for the tmpfs run.

Oh, and I ran these under Red Hat 7.3, kernel 2.4.19, on a Vaio
Picturebook C1VPK, TM5600 Crusoe CPU, 128MB RAM.

-Bennett

--uAKRQypu60I7Lcqm
Content-Type: text/plain; charset=us-ascii
Content-Disposition: attachment; filename=cdbbench

#!/usr/bin/perl -w
use strict;

=head1 NAME

cdbbench --- benchmark CDB, specifically to attempt to determine its appropriateness for pop-before-smtp

=head1 SYNOPSIS

cdbbench

=head1 DESCRIPTION

cdbbench performs alternating runs, while stepping up the chosen
setsize. The setsize is intended to reflect the number of
currently-active users in the pop-before-smtp database; it's the
number of records in the database.

Each record is a random IP-address-like key (four dotted decimals,
each in [0-255]), with the value "ok".

For each setsize, starting with 10 and growing by factors of 10 (10,
100, 1000, 10000, etc). For each setsize, cdbbench generates a load
of 10 times that many random keys. These are then cycled, first
through a fresh new cdb, then through a fresh new db, as fast as
possible, for 30 seconds; at the end of 30 seconds the total number
cycled through is reported. The DB clock doesn't start until the first
setsize items have been loaded (to prime the pump); the equivalent
step means nothing with CDB since the whole CDB file must be
rewritten each pass.

Hit control-C when you get tired of waiting.

=cut

use File::Basename;

my $tmpfile = "/tmp/" . basename($0) . '.' . $$;
$│ = 1;

print "\tCDB\tDB\n";

my $setsize = 1;
while (1) {
$setsize *= 10;
print $setsize, "\t";

my @load; # in-memory cache
for (my $i = 0; $i < 10*$setsize; $i++) {
push @load, join(".", int(rand(256)), int(rand(256)), int(rand(256)), int(rand(256)));
}

# CDB
use CDB_File;
my $count = 0;
my $end = time + 30;
while (time < $end) {
my $cdb = CDB_File->new($tmpfile, "$tmpfile.tmp") or die;
for (my $i = 0; $i < $setsize; $i++) {
$cdb->insert($load[$i], "ok");
}
$cdb->finish;
my $tmp = shift(@load);
push @load, $tmp;
$count++;
}
print $count, "\t";
unlink $tmpfile;

# DB
use DB_File;
use Fcntl qw(:flock);
use vars qw(%h);
my $dbh = tie %h, 'DB_File', $tmpfile;
my $fd = $dbh->fd;
open(DB_FH, "+<&=$fd") or die "$0: cannot open $tmpfile filehandle: $!\n";
for (my $i = 0; $i < $setsize; $i++) {
$h{$load[$i]} = "ok";
}
$count = 0;
$end = time + 30;
while (time < $end) {
flock(DB_FH, LOCK_EX) or die "$0: flock LOCK_EX failed: $!\n";
delete $h{$load⁰};
$h{$load[$setsize]} = "ok";
$dbh->sync;
flock(DB_FH, LOCK_UN) or die "$0: flock LOCK_UN failed: $!\n";
my $tmp = shift(@load);
push @load, $tmp;
$count++;
}
print $count, "\n";
close DB_FH;
undef $dbh;
untie %h;
unlink $tmpfile;
}

uAKRQypu60I7Lcqm

--DBIVS5p969aUjpLe
Content-Type: application/pgp-signature
Content-Disposition: inline

-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.0.7 (GNU/Linux)

iD8DBQE9twcGHZWg9mCTffwRAu0UAJ0QSJKYqBm6HYE3/ypbvOlGEfyJyACgtLa0
H9svGNdTkcm/XNn/PrLOgSA=
=LtOG
-----END PGP SIGNATURE-----

DBIVS5p969aUjpLe

寄件者：Wietse Venema (wietse@porcupine.org)
主旨：Re: [ot] tinyCDB is available in packaged form

View this article only
新闻群组：mailing.postfix.users
日期：2002-10-23 14:06:04 PST

Bennett Todd:
> 2002-10-16-12:18:58 Bennett Todd:
> > 2002-10-15-15:43:50 Michael Tokarev:
> > > Bennett Todd wrote:
> > > >I'd have to do some experimenting, but I really suspect
> > > >pop-before-smtp would work very well indeed with cdb.
> > >
> > > One word: scalability. [...] It will work for small sites.
> >
> > Define "small".
> > [...]
> > Where's the crossover point? I.e. where does the cost of the CDB
> > generation grow to exceed the cost of doing the DB manipulation?
>
> Ok, in the interests of fairness, I've gotta come back and confess
> that my intuition was way, way off here, and Michael Tokarev's was
> right on.
>
> While cdb can work for small sites, where the load of doing things
> inefficiently doesn't make any difference anyway --- just what he
> suggested --- there's no "crossover point"; no matter how small
> the database, DB_Hash is faster than CDB_File for "updates" in all
> circumstances.

It may still be fast enough for small sites. With up to 1000
pop-before-smtp entries in a table, the cost of an update is only
1-2 milliseconds. On a system that is maintained by a private
person the impact would be negligible.

Thanks for measuring this.

Wietse
-
To unsubscribe, send mail to majordomo@postfix.org with content
(not subject): unsubscribe postfix-users
Post a follow-up to this message

Message 28 in thread
寄件者：Rahul Dhesi (dhesi@rahul.net)
主旨：Re: [ot] tinyCDB is available in packaged form

View this article only
新闻群组：mailing.postfix.users
日期：2002-10-23 15:30:07 PST

On Wed, Oct 23, 2002 at 04:31:02PM -0400, Bennett Todd wrote:

> While cdb can work for small sites, where the load of doing things
> inefficiently doesn't make any difference anyway --- just what he
> suggested --- there's no "crossover point"; no matter how small
> the database, DB_Hash is faster than CDB_File for "updates" in all
> circumstances.

If I were designing a pop-before-smtp system from scratch, I would do
it like this:

1. A function that creates (or updates the timestamp of) a file
representing an IP address. The filename is generated by replacing the
first N dots in the IP address with slashes. N is in the range 0 .. 3.
Number of system calls needed: two (an open and a close).

2. A function that returns true or false according as the file for a
given IP address exists or not. Number of system calls needed: one (a
stat).

3. Every half hour a cron job does this:

cd somedir; find . -type f -cmin +30 -print │ xargs rm -f

This is how I first implemented pop-before-smtp in 1997. I used N = 0.

As a possible optimization, the daemon that creates the files can
consult an in-memory hash table and avoid recreating/retouching the same
file more than once every 20 minutes.

As another optimization, we can run the cron job every 24 hours instead,
and have the lookup function (in item 2) return true only if the file is
no older than 30 minutes. This increases the number of system calls to
two: a gettimeofday and a stat.

I'm guessing that the reason we are discussing the efficiency of
different types of databases, instead of using the simple and efficient
scheme I have described above, is because Postfix's smtp server does not
have a built-in facility to look for a filename representing an IP
address. Thus I believe the cart is dragging the horse here.

Rahul
-
To unsubscribe, send mail to majordomo@postfix.org with content
(not subject): unsubscribe postfix-users
Post a follow-up to this message

Message 29 in thread
寄件者：Wietse Venema (wietse@porcupine.org)
主旨：Re: [ot] tinyCDB is available in packaged form

View this article only
新闻群组：mailing.postfix.users
日期：2002-10-23 15:34:28 PST

Wietse Venema:
> Bennett Todd:
> > 2002-10-16-12:18:58 Bennett Todd:
> > > 2002-10-15-15:43:50 Michael Tokarev:
> > > > Bennett Todd wrote:
> > > > >I'd have to do some experimenting, but I really suspect
> > > > >pop-before-smtp would work very well indeed with cdb.
> > > >
> > > > One word: scalability. [...] It will work for small sites.
> > >
> > > Define "small".
> > > [...]
> > > Where's the crossover point? I.e. where does the cost of the CDB
> > > generation grow to exceed the cost of doing the DB manipulation?
> >
> > Ok, in the interests of fairness, I've gotta come back and confess
> > that my intuition was way, way off here, and Michael Tokarev's was
> > right on.
> >
> > While cdb can work for small sites, where the load of doing things
> > inefficiently doesn't make any difference anyway --- just what he
> > suggested --- there's no "crossover point"; no matter how small
> > the database, DB_Hash is faster than CDB_File for "updates" in all
> > circumstances.
>
> It may still be fast enough for small sites. With up to 1000
> pop-before-smtp entries in a table, the cost of an update is only
> 1-2 milliseconds. On a system that is maintained by a private
> person the impact would be negligible.

Argh, that should be 30-60 milliseconds.

> Thanks for measuring this.
>
> Wietse
> -
> To unsubscribe, send mail to majordomo@postfix.org with content
> (not subject): unsubscribe postfix-users
>
>

-
To unsubscribe, send mail to majordomo@postfix.org with content
(not subject): unsubscribe postfix-users
Post a follow-up to this message

Message 30 in thread
寄件者：Matthias Andree (ma@dt.e-technik.uni-dortmund.de)
主旨：Re: [ot] tinyCDB is available in packaged form

View this article only
新闻群组：mailing.postfix.users
日期：2002-10-24 07:10:04 PST

Rahul Dhesi writes:

> I'm guessing that the reason we are discussing the efficiency of
> different types of databases, instead of using the simple and efficient
> scheme I have described above, is because Postfix's smtp server does not
> have a built-in facility to look for a filename representing an IP
> address. Thus I believe the cart is dragging the horse here.

You can get all this for free with SMTP AUTH. No database required
except that one which holds the password and has a few writes a day
only, even at big sites. No file system sweeps necessary to expire old
relay permit files, and no cache to avoid touching the file system too
often.

--
Matthias Andree
-
To unsubscribe, send mail to majordomo@postfix.org with content
(not subject): unsubscribe postfix-users
Post a follow-up to this message

寄件者：Matthias Andree (ma@dt.e-technik.uni-dortmund.de)
主旨：Re: [ot] tinyCDB is available in packaged form

View this article only
新闻群组：mailing.postfix.users
日期：2002-10-28 02:54:08 PST

"Peter H. Coffin" writes:

>> You can get all this for free with SMTP AUTH. No database required
>> except that one which holds the password and has a few writes a day
>> only, even at big sites. No file system sweeps necessary to expire old
>> relay permit files, and no cache to avoid touching the file system too
>> often.
>
> Unless setup and tuning time costs nothing, based on the number of
> difficulties people have setting SASL up, you have a different
> definition of "free" than I do.

"free" here means "free of additional system load".

I've been using SASL1 with the pwcheck daemon to authenticate users
against /etc/shadow or with sasldb without difficulties. When using the
pwcheck daemon, make sure the permissions to /var/pwcheck are right, I'm
using

drwx--x--- 2 root postfix 58 Sep 26 23:30 /var/pwcheck/

Note group ownership and execute permission (That's chown root:postfix
and then chmod 0710.)

I'm aware that this scheme has been superseded in the meanwhile, but it
still works with the latest Cyrus-SASL v1 version. It is disputable
whether this is safe enough to the local machine -- you get to balance
"local security" (use POP3-before-SMTP) against "network security" (use
SASL for SMTP AUTH).

--
Matthias Andree

Posted by hzqbbc at 03:31 PM | Comments (0)

November 11, 2002

对Postfix进行性能调整的高级策略

注意看viktor的发言

注意Wietse的性能优化 （强烈建议）

注意使用TRAM提高性能的方法

所有留言及回应
Message 1 in thread
寄件者：Simon White (simon@mtds.com)
主旨：Re: Postfix Choking?

View this article only
新闻群组：mailing.postfix.users
日期：2002-10-07 04:10:07 PST

07-Oct-02 at 00:54, Brian Wizard (brian@xtnd.net) wrote :
> >From what I have heard, postfix is supposedly a robust and scalable MTA.
> In the past 24 hours I have loaded postfix on a server running FreeBSD
> 4.7-RC with dual 2 gig Xeon procs with 1 gig of RAM and an SMP kernel.
> However, Postfix seems to be running out of resources as time goes on.

Did you see the post this morning re:

http://www.stahl.bau.tu-bs.de/~hildeb/postfix/bottleneck.shtml

--
[Simon White. vim/mutt. simon@mtds.com. GIMPS:1.930% see www.mersenne.org]
If it dies, it's biology. If it blows up, it's chemistry, and if it
doesn't work, it's physics.
[Linux user #170823 http://counter.li.org. Home cooked signature rotator.]
-
To unsubscribe, send mail to majordomo@postfix.org with content
(not subject): unsubscribe postfix-users
Message 2 in thread
寄件者：Brian Wizard (brian@xtnd.net)
主旨：Postfix Choking?

View this article only
新闻群组：mailing.postfix.users
日期：2002-10-07 04:20:25 PST

Greetings,

From what I have heard, postfix is supposedly a robust and scalable MTA.
In the past 24 hours I have loaded postfix on a server running FreeBSD
4.7-RC with dual 2 gig Xeon procs with 1 gig of RAM and an SMP kernel.
However, Postfix seems to be running out of resources as time goes on. I
am very new to Postfix and the only tuning options I have set in main.cf
are the following:

qmgr_message_active_limit = 7000
local_destination_concurrency_limit = 600
default_destination_concurrency_limit = 600
default_process_limit=600
process_limit=600

When I raise the process_limits/concurrency_limits to more than 600 the
server starts swapping for memory? And outputs the following message in
/var/log/messages:

Oct 7 08:38:39 server1 postfix/trivial-rewrite⁴⁸⁸¹⁰: fatal: accept
connection: Too many open files

If I leave the settings the way they are, postfix takes a long
time to respond on port 25 (more than 1 minute) after it has been running
for a few hours.

I have played with many kernel settings and nothing seems to help, could
this just be a limitation on Postfix? This server is processing close to 2
million emails/day.

I don't think postfix should need more than this...

root@server1:~# su - postfix
postfix@server1:~% ulimit -n
39218

I have maxfiles set to the maximum amount in the kernel as well.

root@server1:~# sysctl kern.maxfiles
kern.maxfiles: 65536

Has anyone run into a situation such as this before? Or can anyone make a
suggestions as far what optimization settings should be incorporated into
postfix for email systems with high loads? BTW, I have been running qmail
with no problems, but I would really like to switch to postfix if at all
possible. Thanks for your help.

Brian Wizard

-
To unsubscribe, send mail to majordomo@postfix.org with content
(not subject): unsubscribe postfix-users
Message 3 in thread
寄件者：Ralf Hildebrandt (Ralf.Hildebrandt@charite.de)
主旨：Re: Postfix Choking?

View this article only
新闻群组：mailing.postfix.users
日期：2002-10-07 04:26:03 PST

On Mon, Oct 07, 2002 at 12:54:49AM -1000, Brian Wizard wrote:

> When I raise the process_limits/concurrency_limits to more than 600 the
> server starts swapping for memory?

Shit happens. I guess you can never have enough RAM, can you?

> Oct 7 08:38:39 server1 postfix/trivial-rewrite⁴⁸⁸¹⁰: fatal: accept
> connection: Too many open files

The kernel cannot handle that many open files.

--
Ralf Hildebrandt (Im Auftrag des Referat V a) Ralf.Hildebrandt@charite.de
Charite Campus Mitte Tel. +49 (0)30-450 570-155
Referat V a - Kommunikationsnetze - Fax. +49 (0)30-450 570-916
Why you can't find your system administrators:
(S)he's out buying some caffeine.

-
To unsubscribe, send mail to majordomo@postfix.org with content
(not subject): unsubscribe postfix-users

寄件者：Chris Laco (claco@summitracing.com)
主旨：RE: Postfix Choking?

View this article only
新闻群组：mailing.postfix.users
日期：2002-10-07 06:36:16 PST

Have you done this stuff yet?
http://www.daemonnews.org/200108/benchmark.html

寄件者：lst_hoe (lst_hoe@kwsoft.de)
主旨：Re: Postfix Choking?

View this article only
新闻群组：mailing.postfix.users
日期：2002-10-07 07:34:01 PST

At 10:10 07.10.2002 -0400, Victor.Duchovni@morganstanley.com wrote:
>Two million messages a day is a high load: average of 23 messages per
>second, with presumably much higher peaks during the day. How high are
>the peaks? Are you sure your I/O subsystem is up to the task. What maps
>are you using for your transport table? Figure out the expected query rate
>(at least 3 per recipient, more if you are using the snapshot release) and
>query latency. Make sure that queries against the transport table can be
>completed in ~5ms (1000/23/10) perhaps less if the intra business day mail
>flow is substantially higher than 23 msgs/sec.
>
>Consider mjt's CDB patch for as many maps as possible. Show output of
>"postconf -n".

May i raise some questions :

Where do this mails go to?
Who has that great volume in the pacific provider area (xtnd.net)?
What kind of people who don′t even read the FAQ try to process 2 million
mails a day?
Strange sender-name, maybe related to this one http://www.brianwizard.com/ :-)

Regards

Andreas H?dle

Kühn & Weyh Software GmbH

WWW.KWSOFT.DE

-
To unsubscribe, send mail to majordomo@postfix.org with content
(not subject): unsubscribe postfix-users
Message 6 in thread
寄件者：Victor.Duchovni@morganstanley.com (Victor.Duchovni@morganstanley.com)
主旨：Re: Postfix Choking?

View this article only
新闻群组：mailing.postfix.users
日期：2002-10-07 07:52:03 PST

On Mon, 7 Oct 2002, Brian Wizard wrote:

> qmgr_message_active_limit = 7000

Don't lower this. If your machine has enough memory raise it to 20000 or
more, but not to solve this problem. It is unrelated to throughput.

> local_destination_concurrency_limit = 600
> default_destination_concurrency_limit = 600

Don't raise these.

> default_process_limit=600

Try 200-400.

> process_limit=600
>

This is not a postfix configuration parameter.

>
> When I raise the process_limits/concurrency_limits to more than 600 the
> server starts swapping for memory? And outputs the following message in
> /var/log/messages:
>
> Oct 7 08:38:39 server1 postfix/trivial-rewrite⁴⁸⁸¹⁰: fatal: accept
> connection: Too many open files
>

The trivial-rewrite daemon cannot handle that many concurrent connections.
You must be using very slow maps. Is LDAP or MySQL used for the transport
table by any chance? Keep the transport table fast, don't use slow
transport maps.

> If I leave the settings the way they are, postfix takes a long
> time to respond on port 25 (more than 1 minute) after it has been running
> for a few hours.

See above.

>
> I have played with many kernel settings and nothing seems to help, could
> this just be a limitation on Postfix? This server is processing close to 2
> million emails/day.
>

Two million messages a day is a high load: average of 23 messages per
second, with presumably much higher peaks during the day. How high are
the peaks? Are you sure your I/O subsystem is up to the task. What maps
are you using for your transport table? Figure out the expected query rate
(at least 3 per recipient, more if you are using the snapshot release) and
query latency. Make sure that queries against the transport table can be
completed in ~5ms (1000/23/10) perhaps less if the intra business day mail
flow is substantially higher than 23 msgs/sec.

Consider mjt's CDB patch for as many maps as possible. Show output of
"postconf -n".

--
Viktor.

-
To unsubscribe, send mail to majordomo@postfix.org with content
(not subject): unsubscribe postfix-users
Message 7 in thread
寄件者：Victor.Duchovni@morganstanley.com (Victor.Duchovni@morganstanley.com)
主旨：Re: Postfix Choking?

View this article only
新闻群组：mailing.postfix.users
日期：2002-10-07 11:50:03 PST

On Mon, 7 Oct 2002, Brian Wizard wrote:

>
>
> Hi Victor,
>
> Thanks for the helpful response. What you said sounds right, we have over
> 10,000 entries in a flat shadow file and trivial rewrite is probably
> choking there. So hashing the passwd file into a CDB should help you
> think? Thanks

No. Trivial rewrite does not consult the password file. It uses
"transport_maps" if set, but you do not set up transport_maps, so
trivial-rewrite is being starved for other reasons, see below, but do you
have enough disk I/O to handle the load? What is the nature of the
traffic, why are you processing 2 million messages a day. Are they inbound
or outbound?

> default_destination_concurrency_limit = 600

Lose this, leave set to default.

> default_process_limit = 600

Lower this (Try 400 or 200).

> local_destination_concurrency_limit = 600

Defitely don't do this! This is 600 concurrent connections to a
single mailbox! Leave at the default value of 2.

> local_recipient_maps = $alias_maps unix:passwd.byname

If your password file is not hashed (nsswitch.conf) and has 10,000
entries, this may cost you a lot here by substantially increasing
"smtpd" concurrency. Make sure to configure a fast random access path to
your user database. If you can't do this system-wide, create a separate
hashed database for use by Postfix and replace unix:passwd.byname with
hash:... (or CDB for even better performance).

> maps_rbl_domains = relays.visi.com

Is this providing timely responses? If not dump it and use someone who
does.

> qmgr_message_active_limit = 7000

Raise this to 20000 or leave the default alone at 10000

> smtpd_banner = $myhostname postfix

Why not leave the default alone. Putting "ESMTP" in the banner increases
the probability that the sender will use ESMTP and hence
predeclared size limits with pipelining.

--
Viktor.

Wietse关于Postfix优化的看法

寄件者：Wietse Venema (wietse@porcupine.org)
主旨：Re: 1.000.000 email posted in a single delivery process
View: Complete Thread （共有 46 条留言）
Original Format
新闻群组：mailing.postfix.users
日期：2001-09-26 14:14:54 PST

If you want to deliver 1000000 queue files in 3 hours, that is 10
milliseconds per file. That seems a bit optimistic to me, considering
the latencies of real disks (every file needs to be read from disk,
the cache is cold) and of real networks (having a local DNS server
should really make a difference).

Neither qmail nor Postfix currently has connection caching, which
can make a difference sending mail to sites that have slow servers.

0 - In the /etc/syslog.conf file, put a - in front of the maillog
filename, so that the stupid syslogd does not hammer the disk
after writing each logfile record.

1 - The default settings of two levels of directory hashing in the
queue should suffice. If you disagree, change this with:

postconf -e hash_queue_depth=3
postfix reload

2 - Configure an adequate number of delivery agents in the master.cf
file: 500+. On your typical Linux box, this will run the kernel
out of resources and Postfix will slam on the brakes until you
fix it.

3 - Test the setup with the smtp-source and smtp-sink programs.
For the test, specify a relayhost name that points to the
smtp-sink program, and send messages with thousands of recipients
into Postfix, and have Postfix perform one-recipient deliveries.

On the Postfix host:

postconf -e relayhost=smtp-sink.host:9999 \
smtp_destination_recipient_limit=1

On the mail sink host (can be same as postfix host, if you have
enough memory and process slots and space for open files):

./smtp-sink -c :9999 1000

On the smtp-source host (can be same as postfix host):

./smtp-source -c -m 1000 -s 10 -r 10000 -t foo@postfix.host postfix.host

Postfix breaks up the deliveries into transactions of 1 recipient
each, so one message with 10000 recipients will keep lots of
delivery agents busy.

4 - Before submitting the real mail:

postconf -e defer_transports=smtp relayhost= \
smtp_destination_recipient_limit=\$default_destination_recipient_limit
postfix reload

(and don't let anyone execute a "sendmail -q" command).

5 - SUBMIT THE MAIL VIA SMTP, not via /usr/sbin/sendmail. Otherwise
you are throwing away all the performance.

6 - On Doomsday,

postconf -e defer_transports=
postfix reload
sendmail -q

7 - From here on, do not touch the box. Each time you stop or reload
Postfix, or do a sendmail -q, it slows down dramatically.

Wietse

Alexander:
> (I've posted this message in QMAIL and EXIM mailing list.
> Many people say me that POSTFIX can do a good work)
>
> This is a case study and it's NOT for create a spamming system or other
> terrible mail bombing system.
> I HATE them !
>
> Title: the best mode to deliver 1.000.000 email messages (recipient are ALL
> different and email are ALL different) in few hours during a single delivery
> session.
>
> You suppose to have a SMP Linux server (i.e. dual P-III 800 Mhz, 512 MB Ram,
> SCSI Raid 5, kernel 2.4.9 and RaiserFS) and a good internet connection (4
> Mbit/sec full bandwidth) and a fast DNS server near me.
>
> During the night a program running on the same server creates 1 million of
> email (getting user data and infos data from a SQL DB).
> This process need about 3 hours. Meanwhile email are created, they are
> "injected" in MTA queue (i.e. using qmail-inject or
> exim -odq -t) and they are NOT delivered: they stay in queue.
> The email are all different but short: about 800-1000 bytes each and the
> recipients are all different and are all remote.
> Yes, we can have a vary big number of different recipient domain but
> statistically we also have a large number of email directed on mass email
> system (Hotmail, Yahoo...)
>
> So 1.000.000 × 1.000 bytes = 1 GBytes data. Yes, like a technical
> information email system with personalized info for each recipient.
>
> Then every day at 8.00 AM the remote delivery needs to start (exim -q ,
> qmail-remote) and delivery is done once per day.
>
> Target: send 1 million of email in a very short time, shortest as possibile,
> it must end as soon as possibile. Bandwidth occupation is not a problem: we
> can use all 4 Mbit/sec (about 1 GB / 500 Kbytes/sec = about 2000 seconds , 1
> hour and few minutes. 2 or 3 hours are ok anyway).
>
> And now the questions:
>
> 1) QMAIL vs EXIM vs POSTFIX: which is the best MTA to manage 1.000.000 email
> in queue ?
> (EXIM with option "split_spool_directory" and QMAIL patched to have a more
> "deep" in queue tree, POSTIFIX ??)
>
> 2) EXIM can deliver all messages during a single SMTP session to the same
> email system (useful for mass system like HOTMAIL,YAHOO, ....), QMAIL no.
> Is it faster to have either for example 300 (more or less) qmail-remote
> process running and sending 1 messages each or for example 20 (more or
> less) "exim -q" process running sending more messages in a single SMTP
> connection ? Remember: the target is to use less time than possible.
>
> Thanks for any suggestion about.
> POSTFIX ,I've never used it, can be an alternative ?
>
>
>

另外一个人的性能调整建议，不过是产品化的建议。不错！虽然有些我不同意。

寄件者：Liviu Daia (Liviu.Daia@imar.ro)
主旨：Re: 1.000.000 email posted in a single delivery process

View this article only
新闻群组：mailing.postfix.users
日期：2001-09-27 02:11:08 PST

On 27 September 2001, Liviu Daia wrote:
[...]
> For a today's PC, something like 40-50 messages / second would still
> be an optimistic estimate;

That is, over a LAN. Expect a much lower rate over Intermet,
because of dead or slow destinations.

> that's about 6 times slower that what you want.

Regards,

Liviu Daia

--
Dr. Liviu Daia e-mail: Liviu.Daia@imar.ro
Institute of Mathematics web page: http://www.imar.ro/~daia
of the Romanian Academy PGP key: http://www.imar.ro/~daia/daia.asc
-
To unsubscribe, send mail to majordomo@postfix.org with content
(not subject): unsubscribe postfix-users
第 9 条留言
寄件者：Liviu Daia (Liviu.Daia@imar.ro)
主旨：Re: 1.000.000 email posted in a single delivery process

View this article only
新闻群组：mailing.postfix.users
日期：2001-09-27 02:11:20 PST

On 26 September 2001, Alexander wrote:
[...]
> Title: the best mode to deliver 1.000.000 email messages (recipient
> are ALL different and email are ALL different) in few hours during a
> single delivery session.
>
> You suppose to have a SMP Linux server (i.e. dual P-III 800 Mhz, 512
> MB Ram, SCSI Raid 5, kernel 2.4.9 and RaiserFS) and a good internet
> connection (4 Mbit/sec full bandwidth) and a fast DNS server near me.

Wietse already explained how to setup Postfix for best performance.
Here's an additional checklist, mainly for the rest of the system:

- if possible, use OpenBSD or FreeBSD with softupdates rather than
Linux;
- if you insist on using Linux, use a 2.2.x kernel, not a 2.4.x one, and
turn OFF the swap; buy more RAM if you feel you're running out of it;
- use RAID 0+1, not 5, and (if you have that choice) mirror the stripes
instead of striping the mirrors;
- use ext2; Xfs and Jfs are not yet ready for prime time, and everything
else is slower than ext2 for mail handling;
- install a caching DNS on the SAME machine as the mail server, a nearby
one is not good enough; use djbdns for that instead of bind; turn off
logging for lookups, and give it plenty of memory to keep the cache;
- if you trust your UPS, turn off sync writes to the queue
("chattr -R -S /var/spool/postfix")
- if possible, send the logs to another machine instead of writing them
to a local file, or at least put the logs on different disk spindles
than the queue;
- apply Michael Tokarev's patch for CDB maps, and use CDB maps instead
of hash.

(I'm sure I forgot a few things too.)

[...]
> Target: send 1 million of email in a very short time, shortest as
> possibile, it must end as soon as possibile. Bandwidth occupation is
> not a problem: we can use all 4 Mbit/sec (about 1 GB / 500 Kbytes/sec
> = about 2000 seconds , 1 hour and few minutes. 2 or 3 hours are ok
> anyway).
[...]

As Wietse said, that's overly optimistic. With careful tuning,
Postfix would perform better than Exim and Qmail, but IMHO, no MTA would
be able to send 1 million messages in an hour, even over a LAN. No
offense intended, but people who set themselves such a target have never
managed a mail systems in the real world. For a today's PC, something
like 40-50 messages / second would still be an optimistic estimate;
that's about 6 times slower that what you want.

Regards,

Liviu Daia

寄件者：Wietse Venema (wietse@porcupine.org)
主旨：Re: 1.000.000 email posted in a single delivery process

View this article only
新闻群组：mailing.postfix.users
日期：2001-09-27 06:45:59 PST

Matthias Andree:
> Well, in a LAN, it might actually be possible to get half a million
> mails through per hour without extensive tuning - however, that's
> without VERP and it assumes your destination machines are FAST.

To give you a data point: running smtp-source against smtp-sink
on a 750MHz pentium FreeBSD 4.3 machine without going over a LAN,
with 100 parallel connections:

tail% /usr/bin/time ./smtp-source -d -m 1000 -s 100 tail:9999
1.21 real 0.06 user 0.50 sys

That's 1 million messages in 20 minutes.

Same email load, two hosts connected by a 100BaseT ethernet switch,
target is P550 FreeBSD 4.1, with 100 parallel connections:

tail% /usr/bin/time ./smtp-source -d -m 1000 -s 100 spike:9999
1.05 real 0.06 user 0.23 sys

The times are pretty much comparable.

Do not expect anything near this performance with a queue on rotating
disk drives, though.

Wietse

使用内存RAM来提高Postfix的性能

如何使用TRAM提高你计算机的速度？英文的哦。不过太恐怖了。用ram。。
TRAM

以下是邮件列表的一个人说的话，确实是“化整为零”将负载分布到廉价的*nix box
============
寄件者：Craig Sanders (cas@taz.net.au)
主旨：Re: Making postfix faster?

View this article only
新闻群组：mailing.postfix.users
日期：2001-07-05 21:54:15 PST

On Thu, Jul 05, 2001 at 08:08:28PM -0400, Myke Olson wrote:
> Does anyone have any suggestions on making Postfix faster at sending mail?
> [...]
> We're running the latest version on a Sun Ultra1/140 and are only able
> to send about 160 emails/minute.

for this application, you may be better of with lots of cheap little
linux or freebsd boxes running postfix than one big Sun box.

one of the best ways is to have multiple cheap boxes rather than one big
expensive box.

this may require modifying your mailing list software so that it sends
part of each list to different machines. e.g. if you have 5 postfix smtp
sender boxes, then split the list in 5.

if you send individual messages to each subscriber rather than one message
bcc-ed to everyone then you could use a load balancer to spread the mail
load across multiple machine. e.g. linux virtual server (LVS) makes an
excellent load balancer - http://www.linuxvirtualserver.org/

alternatively, use a large RAID array of as many disks as you can
afford. the more drives in the array, the better. e.g. if you want 72GB
of total space, it is much faster to have 8 × 9GB drives than 2 × 36GB,
or 4 × 18GB.

a solid-state disk (SSD) will be even faster than a raid array.

the way to scale up a disk I/O-bound application (such as sending to
enormous mailing lists) is to speed up the disk by adding more physical
disks and/or more machines or by using faster disks (e.g. SSDs).

combining the alternatives above will get even better results - e.g. an
array of 5 linux + postfix boxes, each with a 1GB SSD queue drive.

> Some ideas that I had were to somehow make it give up on the first
> try if the DNS record for a particular host didn't exist (i.e. a user
> signed up with jdflkl@kjdjdflk.com as their email address) or somehow
> moving the spool directory into a RAM disk. But I couldn't figure out
> how to do that....

you could do that if you didn't care whether the messages got lost if
the system crashed or lost power.

a safer alternative is to use a non-volatile solid-state disk for the
postfix queue. in essence, they're a RAM disk with a battery backup
and a SCSI interface, in a standard disk form factor. they cost a
fortune, though, for any reasonable size....several thousand dollars per
gigabyte, last time i looked into them.

if 1 or 2 GB is enough for your postfix queue, then an SSD may be your
best option.

craig

第 5 条留言
寄件者：Len Conrad (LConrad@Go2France.com)
主旨：Re: Multiple frontend Postfix - One RAID5 storage

View this article only
新闻群组：mailing.postfix.users
日期：2003-01-14 06:47:20 PST

>mailserver-front--│
>mailserver-front--│--> store spool on 1 raid5 dev, seperate spools
>mailserver-front--│

It sounds like your are getting ready for very high volumes, so why have
you chosen the RAID that is slowest for writing? RAID1 has faster writing.

And of course going across a network to the spool dir is horribly slower
than local disk i/o.

RAID5 is fine for database work where read:write it 90:10, but mail is
closer to 50:50.

构筑高可靠的app，这个东西可以做个随时监测smtp 机器的mon。。不错不错。
Wackamole

Posted by hzqbbc at 10:07 PM | Comments (0)

为实现大容量邮件系统所设计LDAP ldif 的构思

在如何构Postfix(MTA)+Maildrop(MDA)+SqWebMail(web-MUA)+IMAP/POP3+OpenLDAP 组成的邮件系统时，自己考虑了不少问题，现在并没完成。

良好的postfix配置及优化好的ldap结构设计能大幅提高性能。例如：
◎适当的进行压力测试（例如postal）可以估算出系统能承受的负载量，通过结果来适当调整smtp(output)/smtpd(input)的最大进程数来达到一个综合的最优化结果。

同时注意系统的调节，Linux的注意打开文件数/用户进程上限及内存、资源分配，FreeBSD则注意对内核进行微调，如kern.maxprocperfiles等（详细看freebsd的tunning）

◎注意对Postfix里的一些小参数仔细设置，如一些timeout值，进程上限,lock_delay, queue的lifetime,refresh time,ipc等的timeout 和idle timeout值，这些可仔细看man

◎如果virtual_domains不多或者更新较少的话，强烈建议使用hash来保存，而不是ldap。因为至少每一封信，postfix都至少要查询3-5次ldap，如果对virtual_maps的查询改成了hash后，就减少了20%-25%的查询量，意义非比寻常。

◎尽量在SMTP会话过程中就reject掉垃圾信，可以减少很多无用工作。因此各种check及限制的手段就显必不可少了。

◎尽量向本地的ldap查询，并且严格注意查询的timeout设置，并安排多个ldap的server做冗余（提高可靠性）并使用快速的网络连接。最好能设计成并发的query就比较好了。

◎注意使用高速的DB（例如最新的berkeley db）而不是用陈旧的缓慢的db。并且配置适当的slave ldap服务器提高分布能力。

◎对virtual_mailbox_maps及access_maps等分别指定不同的ldap服务器，可人为的进行分布查询，将负载分担。由于基本上每个mail都要同时查询这些maps，所以负载是相当均衡的。

◎对pipe改造。利用ldap查询时得到的用户目录（前提是本地的虚拟用户）直接传递给MDA (maildrop)进行直接投递，免除了maildrop再次查询ldap所带来的重复问题。

part2
补充一点点：
◎使用Berkerly DB代替GDBM，作为ldap的backend，db3/db4等的速度大大的快于gdbm，甚至gdbm使用了优化的index也没办法达到dbx的一般速度。

◎关闭log功能，即"loglevel 0"，这样可以大幅度提高ldap查询速度，难以置信的快！

详细见：
Close loglevel to improve performance

◎mydestination部分，如果要自动辨别新加的domain，那么使用$virtual_maps进去，但这可能会导致postfix多次查询ldap（具体待验证）如果是，那么最好取消掉，换之以确定的数据（如显式的填写好domain，或者以hash的形式作为virtual_maps比较好，好处同上）

part3
有人做了个mydestination用ldap的简单测试结果以及一些讨论，可以看看：

Mydest via ldap

part4 (updated!)
一些有关于Email及LDAP的RFC/Draft，值得一看，也是我的参考文章。

LDAP Schema for Internet Mail

MIME Directory Profile for LDAP Schema

有关Postfix VDA讨论，其中不少关于LDAP schema的内容

LDAP Schema for Intranet Mail Routing

LDAP-based Routing of SMTP Messages

LDAP: User Schema

LDAP-based Routing of SMTP Messages

Using LDAP for SMTP Mailing Lists & Aliases

LDAPv3: A Collection of User Schema (最新的user schema设计)

Schema Functions for the LDAP C API (不错的c API参考)

Posted by hzqbbc at 07:39 PM | Comments (1)

有关Postfix 的虚拟用户支持的大讨论

原始地址：
virtualizing local users

--------------
Message 1 in thread
寄件者：Phil Howard (phil-postfix-users@ipal.net)
主旨：virtualizing local users

View this article only
新闻群组：mailing.postfix.users
日期：2002-07-01 02:43:07 PST

I am trying to virtualize the users on a new mail server.
What I mean by that is that the user names will not be listed
in the /etc/passwd file. There is another file which looks
like the first 2 fields of /etc/passwd or /etc/shadow which
contains the list of valid users. The main question is how
might I get Postfix to check this file for known users?

I do have the option to make this be virtual or not virtual
since this mail server is serving only one space of usernames
(that is, xyzzy@foo and xyzzy@bar are the same mailbox and
same user). But I have been looking through the various ways
to configure either approach and I simply don't find a way to
do this. Mail is being delivered OK, and picked up OK using
vm-pop3d as the pop3 server (which reads the alternate file
to get user passwords). So once I can get Postfix to know
the users are existing (if they are in that file), then it
should all be working.

I definitely do not want to add these users to /etc/passwd or
/etc/shadow. There are conflicts between the user space for
mail and the user space for shell logins (which have no local
mail), so this isn't an option.

I also want to avoid (means if there is no other way, then this
can be done, but I definitely want to exhaust all other possible
approaches first) having a separate file to maintain just for
Postfix to do user lookups. Basically this means if there is
a way to use the shadow file format, that's preferred, but any
map format Postfix can handle will do as a last resort.

No single configuration description pops out at me and says this
will do what I want. And I can't see any combinations that
might create the same effect. Anyone have any ideas on this?

Note that there is no virtual translation. User "xyzzy" has a
mailbox named "xyzzy" in (via symlink) /var/spool/mail in the
traditional mailbox format.

To unsubscribe, send mail to majordomo@postfix.org with content
(not subject): unsubscribe postfix-users
Message 2 in thread
寄件者：Jean-Pierre Schwickerath (lists@schwicky.net)
主旨：Re: virtualizing local users

View this article only
新闻群组：mailing.postfix.users
日期：2002-07-01 03:03:54 PST

Phil Howard wrote:

> What I mean by that is that the user names will not be listed
> in the /etc/passwd file. There is another file which looks
> like the first 2 fields of /etc/passwd or /etc/shadow which
> contains the list o

Zhiqiang::He

Happy + Health = Life^2

November 28, 2002

关于Postfix虚拟域的相关释疑

November 13, 2002

有关于mjt所写的tinycdb的技术内幕讨论

November 11, 2002

对Postfix进行性能调整的高级策略

Wietse关于Postfix优化的看法

使用内存RAM来提高Postfix的性能

为实现大容量邮件系统所设计LDAP ldif 的构思

有关Postfix 的虚拟用户支持的大讨论