Zhiqiang::He

寄件者：Gianugo Rabellino (gianugo@apache.org)
主旨：Scalability issues
View: Complete Thread (14 articles)
Original Format
新闻群组：mailing.postfix.users
日期：2002-10-30 01:54:04 PST

(note: I sent this message already two days ago, but it didn't show up
on the list. I'm sending it again, hoping this time will get through.
Just in case, sorry for the duplicate)

Hi,

I'm running a huge email gateway (~1.000.000+ e-mail per day). The
system has postfix running as a frontend SMTP gateway, dealing with
approx. half of the traffic (there is another - commercial - mail router
splitting the load with postfix).

~80% of the email traffic is sent to a farm of inbound hosts, the rest
is sent to the outside world by users authenticated via SMTP-AUTH. The
system relies heavily on LDAP lookups both for canonicalization and
transport lookup.

The postfix machine is a Linux 2.4.18-3, queue files are on a separate
disk, as an ext3 fs with data=journal and noatime. The kernel file-max
and threads have been risen up. The machine is running its own dnscache
process to speed up lookups. I've spent several days tuning up the
system, reading documentation and lurking the list and the archives: I'm
pretty confident, although I might have done something stupid on the
way, that I optimized the system quite heavily, following Ralf
Hildebrandt's most helpful hints as well as other documentation I found
on the net (Victor Duchovni really enlightened me with
http://archives.neohapsis.com/archives/postfix/2001-07/0871.html), but
still the system sometimes just screws up. When a peak load of inbound
traffic hits the system (peak = ~150 concurrent and sustained smtp
connections), Postfix queues start to grow from the average of 800-1300
up to 5-10.000.

When Postfix hits this number of mails in queue, it starts spiraling
down: email still flows in, but the outgoing rate slows dramatically.
The system climbs up to 20, 30, 40.000 emails in queue and there is no
way out, with loads of e-mails in the active queue. The workaround I
found is to stop postfix, mount the queue directory on another location,
start postfix with a fresh queue and fire a second postfix instance on
the old queue directory to clean up the remains.

The good news is that the two postfix instances keep up nicely: in a
matter of a few hours the queues have been spooled out completely and I
can rollback to the original setup.

My first suspect was LDAP, but since the two postfix instance happily do
their job with a huge load on the LDAP server. The problem isn't the MTA
behind postfix too, for the same reason among others. I strongly suspect
in this scenario an heavy I/O issue, but I'm wondering if I did
something dead wrong or if there is a real scalability issue with
postfix on large amounts of traffic, so that there might be next to
nothing to do to solve this issue in a clean way (true, I can put more
servers, buy faster disks and so on, but iron-based only scalability
worries me a lot).

Following is the relevant part of main.cf. In master.cf I'm using nqmgr
and I have 100 smtpd and 700 smtp processes.

default_process_limit = 2000
smtp_destination_concurrency_limit = 100
trigger_timeout = 1
in_flow_delay = 10s
qmgr_message_active_limit = 10000
qmgr_message_recipient_limit = 10000
hash_queue_names = defer,deferred,incoming,active

queue_run_delay = 120s
minimal_backoff_time = 60s
maximal_backoff_time = 180s
smtp_connect_timeout = 15
qmgr_site_hog_factor = 100

Is there anything stupid that I did? Is this a (un)known postfix limit,
of which I doubt? Would using the two instance trick as a permanent
setup help? Where can I start working on this issue?

Thanks in advance for your assistance, fire at will :-)

--
Gianugo Rabellino

寄件者：Ralf Hildebrandt (Ralf.Hildebrandt@charite.de)
主旨：Re: Scalability issues
View: Complete Thread (14 articles)
Original Format
新闻群组：mailing.postfix.users
日期：2002-10-30 02:00:05 PST

On Wed, Oct 30, 2002 at 10:50:08AM +0100, Gianugo Rabellino wrote:

> The postfix machine is a Linux 2.4.18-3, queue files are on a separate
> disk, as an ext3 fs with data=journal and noatime.

You read http://www.stahl.bau.tu-bs.de/~hildeb/postfix/ext3.shtml ?
(especially the "chattr -R -S /var/spool/postfix" bit?)

> The kernel file-max and threads have been risen up. The machine is
> running its own dnscache process to speed up lookups.

Excellent. This all sounds good.

> I've spent several days tuning up the system, reading documentation
> and lurking the list and the archives: I'm pretty confident, although
> I might have done something stupid on the way, that I optimized the
> system quite heavily, following Ralf Hildebrandt's most helpful hints
> as well as other documentation I found on the net (Victor Duchovni
> really enlightened me with
> http://archives.neohapsis.com/archives/postfix/2001-07/0871.html),
> but still the system sometimes just screws up.

oops.

> When a peak load of
> inbound traffic hits the system (peak = ~150 concurrent and sustained
> smtp connections), Postfix queues start to grow from the average of
> 800-1300 up to 5-10.000.

OK. That means the incoming traffic starves the outgoing traffic.

in_flow_delay = 5s

> When Postfix hits this number of mails in queue, it starts spiraling
> down: email still flows in, but the outgoing rate slows dramatically.

Yup. Two ideas:

Reduce the max. number of smtpd processes so the qmgr can keep up and
the smtp's can send out mail.

> My first suspect was LDAP, but since the two postfix instance happily do
> their job with a huge load on the LDAP server. The problem isn't the MTA
> behind postfix too, for the same reason among others. I strongly suspect
> in this scenario an heavy I/O issue, but I'm wondering if I did

Probably.

> Following is the relevant part of main.cf. In master.cf I'm using nqmgr
> and I have 100 smtpd and 700 smtp processes.

Sounds sane.

> default_process_limit = 2000
> smtp_destination_concurrency_limit = 100
> trigger_timeout = 1
> in_flow_delay = 10s

According to the docs, this can be only in the range 0s-5s!

> queue_run_delay = 120s

You run the queue quite often!

--
Ralf Hildebrandt Ralf.Hildebrandt@charite.de
Postfix Tips: http://www.arschkrebs.de/postfix/ Tel. +49 (0)30-450 570-155
perl -e 'print $i=pack(c5,(41*2),sqrt(7056),(unpack(c,H)-2),oct(115),10);'

寄件者：Gianugo Rabellino (gianugo@apache.org)
主旨：Re: Scalability issues
View: Complete Thread (14 articles)
Original Format
新闻群组：mailing.postfix.users
日期：2002-10-30 02:28:04 PST

Ralf Hildebrandt wrote:

> >The postfix machine is a Linux 2.4.18-3, queue files are on a separate
> >disk, as an ext3 fs with data=journal and noatime.
>
>
>
> You read http://www.stahl.bau.tu-bs.de/~hildeb/postfix/ext3.shtml ?
> (especially the "chattr -R -S /var/spool/postfix" bit?)

Yep:

1. mount │ grep postfix
   /dev/sdb1 on /var/spool/postfix type ext3 (rw,noatime,data=journal)

1. lsattr /var/spool/postfix
   -------------- /var/spool/postfix/lost+found
   -------------- /var/spool/postfix/active
   -------------- /var/spool/postfix/bounce
   -------------- /var/spool/postfix/corrupt
   -------------- /var/spool/postfix/defer
   -------------- /var/spool/postfix/deferred
   -------------- /var/spool/postfix/flush
   -------------- /var/spool/postfix/hold
   -------------- /var/spool/postfix/incoming
   -------------- /var/spool/postfix/private
   -------------- /var/spool/postfix/maildrop
   -------------- /var/spool/postfix/public
   -------------- /var/spool/postfix/pid

> Yup. Two ideas:
>
> Reduce the max. number of smtpd processes so the qmgr can keep up and
> the smtp's can send out mail.

That's what I did ATM. My issue is that lowering the smtpd processes I'm
slowing down the incoming email traffic, so I'm trying to understand the
figures I'm able to keep up with, in order to understand if I have to
plan for more powerful iron, more servers, or just another architecture.
I basically need to understand what is the peak load of inbound traffic
that I can sustain.

> >default_process_limit = 2000
> >smtp_destination_concurrency_limit = 100
> >trigger_timeout = 1
> >in_flow_delay = 10s
>
>
> According to the docs, this can be only in the range 0s-5s!

I couldn't find any doc mentioning that limit, just some mailing list
archives. I resorted to the source code and read in in
src/global/mail_params.c

static CONFIG_TIME_TABLE time_defaults[] = {
[...]
VAR_IN_FLOW_DELAY, DEF_IN_FLOW_DELAY, &var_in_flow_delay, 0, 10,
0,
};

So I assumed that I could go as high as 10 sec.

> >queue_run_delay = 120s
>
>
> You run the queue quite often!

My point was to not let many incoming mails accumulate in the queue.
Does it make sense or is it actually doing more harm than good to the
system?

Thanks for your assistance,

--
Gianugo Rabellino

寄件者："Edward Wildgoose" (Edward.Wildgoose@FRMHedge.com)
主旨：RE: Scalability issues
View: Complete Thread (14 articles)
Original Format
新闻群组：mailing.postfix.users
日期：2002-10-30 03:12:04 PST

Suggest investigating choice of raid controllers before adding disks. A decent SCSI controller with write back cache should make a big difference. On IDE, the 3ware controllers are allegedly pretty good - don't have one, but some people have suggested that transaction rates can increase quite a bit (although they don't have any write back cache, so will never compete with the SCSI stuff).

Also can you refuse any of the incoming email...

A few body checks to eliminate top 3 virus's perhaps, plus use some conservative DNSBL or blocking based on FROM address. Ensure that FROM and TO are real domains and fully qualified, etc...

If you want to go the simple IDE route, then consider that you have several different disk write patterns and simply moving directories over different disks gives you a lot of the benefit of striping, eg put queue dir on a stripe, log on another disk, etc. (Make sure logging is asynchronous!)

Just a few thoughts. Perhaps you have been here already.

Ed W

-----Original Message-----
From: Gianugo Rabellino [mailto:gianugo@apache.org]
Sent: 30 October 2002 10:58
To: postfix-users@postfix.org
Subject: Re: Scalability issues

Ralf Hildebrandt wrote:

> >That's what I did ATM. My issue is that lowering the smtpd processes I'm
> >slowing down the incoming email traffic, so I'm trying to understand the
> >figures I'm able to keep up with, in order to understand if I have to
> >plan for more powerful iron, more servers, or just another architecture.
>
> Try putting the queue_directory on a fast striping RAID

You know what? I'm starting to think whether, scalability wise, this can
be the best solution. I see two different approaches in general to
issues like these:

1. the "big iron" approach. I can buy expensive n-way hardware, load it
with gigs of RAM, put RAID controllers, 15k disk drives, and spend big
money for one machine, hoping that it might be able to keep up with my
traffic peaks;

2. the "Google way": inexpensive hardware added "on demand". Two
machines with a fast SCSI drive, or four machines with a fast IDE drive.
More fault tolerant, more scalability, possibly at the same cost.

I tend to like 2) more. I'm just trying to understand, on my average
machine, how many inbound connections I can sustain before my outgoing
rate is unable to keep up. ATM I tend to think that the figure goes
beyond 50-80, but I want to be sure that this is a reasonable limit and
not a misconfiguration issue (besides, I think that this is a quite
small figure). I was astonished when I saw the dramatic improvement in
performance by "simply" chattr'ing the spool directory, so I'm wondering
if there is another sweet spot still untouched.

> Both incoming and outgoing mail compete for the disk, as does the
> qmgr. That means by running the qmgr too often, you're slowing the
> whole thing down.

Makes absolutely sense. I removed the setting, I'll see what happens in
the next hours.

One more question, though. Since I'm sending 80% of all my email to the
same inbound server, which is currently capable of handling gracefully
the incoming traffic, is there any other setting I can tweak to make
sure that I'm "hammering" it as fast and as much as I can?

Again, thanks for your answers and support.

--
Gianugo Rabellino

-
To unsubscribe, send mail to majordomo@postfix.org with content
(not subject): unsubscribe postfix-users
-

寄件者：lst_hoe (lst_hoe@kwsoft.de)
主旨：Re: Scalability issues
View: Complete Thread (14 articles)
Original Format
新闻群组：mailing.postfix.users
日期：2002-10-30 03:14:01 PST

At 10:50 30.10.2002 +0100, Gianugo Rabellino wrote:
>(note: I sent this message already two days ago, but it didn't show up
>on the list. I'm sending it again, hoping this time will get through.
>Just in case, sorry for the duplicate)
>
>Hi,
>
>I'm running a huge email gateway (~1.000.000+ e-mail per day). The
>system has postfix running as a frontend SMTP gateway, dealing with
>approx. half of the traffic (there is another - commercial - mail router
>splitting the load with postfix).
>
>~80% of the email traffic is sent to a farm of inbound hosts, the rest
>is sent to the outside world by users authenticated via SMTP-AUTH. The
>system relies heavily on LDAP lookups both for canonicalization and
>transport lookup.
>
>The postfix machine is a Linux 2.4.18-3, queue files are on a separate
>disk, as an ext3 fs with data=journal and noatime. The kernel file-max
>and threads have been risen up. The machine is running its own dnscache
>process to speed up lookups. I've spent several days tuning up the
>system, reading documentation and lurking the list and the archives: I'm
>pretty confident, although I might have done something stupid on the
>way, that I optimized the system quite heavily, following Ralf
>Hildebrandt's most helpful hints as well as other documentation I found
>on the net (Victor Duchovni really enlightened me with
>http://archives.neohapsis.com/archives/postfix/2001-07/0871.html), but
>still the system sometimes just screws up. When a peak load of inbound
>traffic hits the system (peak = ~150 concurrent and sustained smtp
>connections), Postfix queues start to grow from the average of 800-1300
>up to 5-10.000.
>
>When Postfix hits this number of mails in queue, it starts spiraling
>down: email still flows in, but the outgoing rate slows dramatically.
>The system climbs up to 20, 30, 40.000 emails in queue and there is no
>way out, with loads of e-mails in the active queue. The workaround I
>found is to stop postfix, mount the queue directory on another location,
>start postfix with a fresh queue and fire a second postfix instance on
>the old queue directory to clean up the remains.
>
>The good news is that the two postfix instances keep up nicely: in a
>matter of a few hours the queues have been spooled out completely and I
>can rollback to the original setup.
>
>My first suspect was LDAP, but since the two postfix instance happily do
>their job with a huge load on the LDAP server. The problem isn't the MTA
>behind postfix too, for the same reason among others. I strongly suspect
>in this scenario an heavy I/O issue, but I'm wondering if I did
>something dead wrong or if there is a real scalability issue with
>postfix on large amounts of traffic, so that there might be next to
>nothing to do to solve this issue in a clean way (true, I can put more
>servers, buy faster disks and so on, but iron-based only scalability
>worries me a lot).
>
>Following is the relevant part of main.cf. In master.cf I'm using nqmgr
>and I have 100 smtpd and 700 smtp processes.
>
>default_process_limit = 2000
>smtp_destination_concurrency_limit = 100
>trigger_timeout = 1
>in_flow_delay = 10s
>qmgr_message_active_limit = 10000
>qmgr_message_recipient_limit = 10000
>hash_queue_names = defer,deferred,incoming,active
>
>queue_run_delay = 120s
>minimal_backoff_time = 60s
>maximal_backoff_time = 180s
>smtp_connect_timeout = 15
>qmgr_site_hog_factor = 100
>
>Is there anything stupid that I did? Is this a (un)known postfix limit,
>of which I doubt? Would using the two instance trick as a permanent
>setup help? Where can I start working on this issue?
>
>Thanks in advance for your assistance, fire at will :-)

Just some ideas (you don′t mention your hardware):

1.) qmgr_message_active_limit = 10000 is a lot of stuff to process/sort in
the active queue. Maybe less would be better for overall throughput.

2.) smtp_destination_concurrency_limit = 100 means you do up to 100
connections to one destination. Depending on the target hosts you should
lower this. Maybe have a look at smtp_destination_recipient_limit for you
internal hosts.

3.) For LDAP have a look at "ldap_bind" and "ldap_domain" parameter.

4.) Transport entries for your internal servers should be with IP-Adress
and "[]" around to save DNS lookups. Or even better dedicated services in
master.cf with dns lookups disabled. There have been discussion about
LDAP-maps for transport and a memory leak some time ago.

Helpful would be if you could provide the numbers for memory/processor load
and maybe some entries from the logfiles (warnings/errors etc.) from the
point your machine starts spiraling down.

Regards

--

Andreas H?dle

Kühn & Weyh Software GmbH

WWW.KWSOFT.DE

寄件者：Galen Johnson (gjohnson@trantor.org)
主旨：Re: Scalability issues

View this article only
新闻群组：mailing.postfix.users
日期：2002-10-30 05:48:06 PST

One thing I have yet to see mentioned, and since you are looking at the
possibilty of multiple machines anyway, is what about having separate
servers for incoming and outgoing. That way you split the load between
separate machines and forego the contention of incoming and outgoing
fighting for i/o (mostly).

=G=

-

寄件者：Gianugo Rabellino (gianugo@apache.org)
主旨：Re: Scalability issues

View this article only
新闻群组：mailing.postfix.users
日期：2002-10-30 07:02:05 PST

Ralf Hildebrandt wrote:

> >if there is another sweet spot still untouched.
>
>
> Check /etc/resolv.conf and /etc/nsswitch.conf

What for? In resolv.conf I only have my local (127.0.0.1) nameserver
configured plus another two for failover. nsswitch.conf is "dns files"
for hosts.

What exactly should I look for?

--
Gianugo Rabellino

寄件者：Ralf Hildebrandt (Ralf.Hildebrandt@charite.de)
主旨：Re: Scalability issues

View this article only
新闻群组：mailing.postfix.users
日期：2002-10-30 07:16:04 PST

On Wed, Oct 30, 2002 at 04:00:04PM +0100, Gianugo Rabellino wrote:

> >Check /etc/resolv.conf and /etc/nsswitch.conf
>
> What for? In resolv.conf I only have my local (127.0.0.1) nameserver
> configured plus another two for failover. nsswitch.conf is "dns files"
> for hosts.
>
> What exactly should I look for?

For exactly this.

--
Ralf Hildebrandt Ralf.Hildebrandt@charite.de
Postfix Tips: http://www.arschkrebs.de/postfix/ Tel. +49 (0)30-450 570-155
If you give someone a program, you will frustrate them for a day; if
you teach them how to program, you will frustrate them for a lifetime.

寄件者：Gianugo Rabellino (gianugo@apache.org)
主旨：Re: Scalability issues

View this article only
新闻群组：mailing.postfix.users
日期：2002-10-30 07:44:01 PST

Victor.Duchovni@morganstanley.com wrote:

> Do you have a dedicated transport for relaying inbound mail? This was the
> most important piece of advice in my Jul 2001 posting. From what follows
> below, the answer appears to be no. The dedicated inbound transport needs
> to have a high (O(100)) destination concurrency.

Unfortunately I'm unable to do it. The problem is that the overall
architecture is based on a mailhost: attribute in the LDAP server that
is being used by other parts of the systems. This atribute contains just
the plain host, not in the postfix syntax transport:[host]. I had to put
a two-liner patch in transport.c to fix this, and I resorted to
default_transport to be put in place of the transport: keyword.

This, however, forbids me from using a dedicated transport. Or do you
see a way to implement it?

> How many "smtp" processes are actually observed to be running when the
> queue is full?
>
Approx. 100, but I want to ckeck better when (if) the problem rises up
again.

As per your suggestions, I will try them all. Just a few comments:

>
> >trigger_timeout = 1
>
> Don't.

OK. But why? :-)

> >queue_run_delay = 120s
> >minimal_backoff_time = 60s
> >maximal_backoff_time = 180s
>
>
> This is insane. I recommended raising maximal backoff time from 4000 to
> 14400 and you take it down to 180s! All the deferred messages are retried
> every 3 minutes, a sure recipe for disaster.

Makes sense, must have missed that. Done.

> >smtp_connect_timeout = 15
>
> Good, but 30s is more reasonable for the Internet at large. Set to
> 5s or 1s only when forwarding to your internal mailhubs.

I set it to 15, as a compromise between outgoing mail and dedicated
inbound, since I'm not able to set up the dedicated transport.

>
> Make sure your LDAP server is fast (<5ms query latency). Use > ldapsource_domain = mydom.ain to limit the domains for which queries are
> sent to LDAP.

This makes a huge lot of sense. Will do it as well.

Thanks a lot for your hints,

--
Gianugo Rabellino

寄件者：Victor.Duchovni@morganstanley.com (Victor.Duchovni@morganstanley.com)
主旨：Re: Scalability issues

View this article only
新闻群组：mailing.postfix.users
日期：2002-10-30 08:10:05 PST

On Wed, 30 Oct 2002, Wietse Venema wrote:

> Gianugo Rabellino:
> > still the system sometimes just screws up. When a peak load of inbound
> > traffic hits the system (peak = ~150 concurrent and sustained smtp
> > connections), Postfix queues start to grow from the average of 800-1300
> > up to 5-10.000.
>
> With that many clients pounding the disk with incoming mail,
> nothing goes out unless you switch to a fater disk.
>
> To keep mail flowing out, reduce inflow. For example, reduce
> the number of SMTPD processes to something that your disk can
> actually handle.
>

I have ~1000000 messages a day, but this is accross 4 machines, each with
a 3 way stripe of fibre attached RAID-5 disks with a battery backed cache.
So I/O almost never hits raw disk and there are effectively 40 disks of
I/O bandwidth to soak up the traffic. Not surprisingly given the 3 layers
of conent filters on my system, it is CPU bound, I/O is handled quite
nicely.

So it is possible the system in question is running out of I/O muscle
(which iostat should easily reveal). That said, my heavy iron was also not
handling the load until it was configured correctly, the main ingredient
was the dedicated inbound transport.

--
Viktor.

寄件者：Russell Mosemann (mose@ns.cune.edu)
主旨：Re: Scalability issues

View this article only
新闻群组：mailing.postfix.users
日期：2002-10-30 08:12:05 PST

On Wed, 30 Oct 2002, Gianugo Rabellino wrote:

> Unfortunately I'm unable to do it. The problem is that the overall
> architecture is based on a mailhost: attribute in the LDAP server that
> is being used by other parts of the systems. This atribute contains just
> the plain host, not in the postfix syntax transport:[host]. I had to put
> a two-liner patch in transport.c to fix this, and I resorted to
> default_transport to be put in place of the transport: keyword.

The patch is not necessary. Besides, you don't want to be customizing
postfix, because then you are locked into maintaining it.

ldapsource_result_attribute = mailHost
ldapsource_result_filter = smtp:[%s]

----
Russell Mosemann, Ph.D. * Computing Services * Concordia University, Nebraska
"A thousand people will stop smoking today. Their funerals will be held
sometime in the next three or four days." C. E. Koop, Former Surg. Gen.

寄件者：Gianugo Rabellino (gianugo@apache.org)
主旨：Re: Scalability issues

View this article only
新闻群组：mailing.postfix.users
日期：2002-10-30 08:28:04 PST

Russell Mosemann wrote:

> The patch is not necessary. Besides, you don't want to be customizing
> postfix, because then you are locked into maintaining it.
>
> ldapsource_result_attribute = mailHost
> ldapsource_result_filter = smtp:[%s]

Ouch, where is my dumbass hat? I blatantly overlooked that setting.
Thanks for showing me the way, you solved one of my greatest PITA with
this solution. Now I think I can set up that dedicated transport after
all. :-)

Thanks again, you sorted me out.

寄件者：Wietse Venema (wietse@porcupine.org)
主旨：Re: Scalability issues

View this article only
新闻群组：mailing.postfix.users
日期：2002-10-30 09:20:03 PST

Gianugo Rabellino:
> still the system sometimes just screws up. When a peak load of inbound
> traffic hits the system (peak = ~150 concurrent and sustained smtp
> connections), Postfix queues start to grow from the average of 800-1300
> up to 5-10.000.

With that many clients pounding the disk with incoming mail,
nothing goes out unless you switch to a fater disk.

To keep mail flowing out, reduce inflow. For example, reduce
the number of SMTPD processes to something that your disk can
actually handle.

Wietse

寄件者：Victor.Duchovni@morganstanley.com (Victor.Duchovni@morganstanley.com)
主旨：Re: Scalability issues

View this article only
新闻群组：mailing.postfix.users
日期：2002-10-30 09:32:07 PST

On Wed, 30 Oct 2002, Russell Mosemann wrote:

> The patch is not necessary. Besides, you don't want to be customizing
> postfix, because then you are locked into maintaining it.
>
> ldapsource_result_attribute = mailHost
> ldapsource_result_filter = smtp:[%s]
>

This is what I was about to suggest, but is this available in stock
1.1.11? I know that result_filter is in the snapshot release, but I have a
feeling that this is not available in the official release.

This is critical for high volume inbound relay throughput. Gianugo should
either switch to non-LDAP transport maps or upgrade to the snapshot or both.

Given that 1.1.11 does not support per user transports, using LDAP for
transports seems quite pointless. Perhaps he is using the experimental
release after all.

--
Viktor.