sa-update without internet breaks spamd/amavisd

[Filippo Carletti]

Hi all,
I'm unsure on how to fix a problem I have with sa-update (SA 3.4.1) on
CentOS 6, where /var/lib/spamassassin/3.004001/ doesn't contain rules.

Steps to reproduce:
1. install spamassassin
2. remove connection to the internet (ip ro del default)
3. run sa-update
4. look into /var/lib/spamassassin/3.004001: no rules (only empty dirs)
5. restart spamd or amavisd: failure
spamd[10720]: config: no rules were found! Do you need to run 'sa-update'?
amavis[3575]: (!!)TROUBLE in pre_loop_hook: config: no rules were
found! Do you need to run 'sa-update'?

Moreover, the default centos sa-update cron job, checks for a running
copy of spamd or amavisd, that are probably not to be found, so the
problem will never fix by itself.

I think I've tracked the problem to the following commit:

https://github.com/apache/spamassassin/commit/2fbb0ec17904ffd9369f53c891e75ad05211bd53

but I'm unsure on hot to fix it.
Should I move the mkpath call to where it was before the commit?

Thank you.

[Joe Quinn]

Post by Filippo Carletti
Hi all,
I'm unsure on how to fix a problem I have with sa-update (SA 3.4.1) on
CentOS 6, where /var/lib/spamassassin/3.004001/ doesn't contain rules.
1. install spamassassin
2. remove connection to the internet (ip ro del default)
3. run sa-update
4. look into /var/lib/spamassassin/3.004001: no rules (only empty dirs)
5. restart spamd or amavisd: failure
spamd[10720]: config: no rules were found! Do you need to run 'sa-update'?
amavis[3575]: (!!)TROUBLE in pre_loop_hook: config: no rules were
found! Do you need to run 'sa-update'?
Moreover, the default centos sa-update cron job, checks for a running
copy of spamd or amavisd, that are probably not to be found, so the
problem will never fix by itself.
https://github.com/apache/spamassassin/commit/2fbb0ec17904ffd9369f53c891e75ad05211bd53
but I'm unsure on hot to fix it.
Should I move the mkpath call to where it was before the commit?
Thank you.

Why are you running sa-update without an internet connection? It
requires one to fetch the latest rules.

[Quanah Gibson-Mount]

Why are you running sa-update without an internet connection? It requires
one to fetch the latest rules.

Imagine a world in which cron jobs run and update the rules nightly. Due
to X reason, the internet is not reachable. Perfectly fine working
installations are suddenly destroyed. Seems like it needs fixing...

--Quanah


--

Quanah Gibson-Mount
Platform Architect
Zimbra, Inc.
--------------------
Zimbra :: the leader in open source messaging and collaboration

[Joe Quinn]

--On Tuesday, December 01, 2015 1:46 PM -0500 Joe Quinn

Why are you running sa-update without an internet connection? It requires
one to fetch the latest rules.

Imagine a world in which cron jobs run and update the rules nightly.
Due to X reason, the internet is not reachable. Perfectly fine working
installations are suddenly destroyed. Seems like it needs fixing...
--Quanah
--
Quanah Gibson-Mount
Platform Architect
Zimbra, Inc.
--------------------
Zimbra :: the leader in open source messaging and collaboration

In which case, perhaps the test case should be the below? I'm not sure
what's being tested by trying to run SA without running sa-update even
once successfully.

Do a fresh installation
run sa-update with an internet connection
run sa-update without an internet connection

[Kevin A. McGrail]

Post by Joe Quinn

--On Tuesday, December 01, 2015 1:46 PM -0500 Joe Quinn

Why are you running sa-update without an internet connection? It requires
one to fetch the latest rules.

Imagine a world in which cron jobs run and update the rules nightly.
Due to X reason, the internet is not reachable. Perfectly fine
working installations are suddenly destroyed. Seems like it needs
fixing...

In which case, perhaps the test case should be the below? I'm not sure
what's being tested by trying to run SA without running sa-update even
once successfully.

SA must have rules installed to function as part of the installation.
Are you doing a manual rule file installation?

See SpamAssassin sa-update rules tarball
<http://ftp.wayne.edu/apache//spamassassin/source/Mail-SpamAssassin-rules-3.4.1.r1675274.tgz>,
for use if you cannot run sa-update" to download these automatically
after installing. Though you might want to grab the latest tarball from
a mirror for the manual installation

Regards,KAM

[John Hardin]

Post by Filippo Carletti
1. install spamassassin
2. remove connection to the internet (ip ro del default)
3. run sa-update

In case it's not clear from the responses so far: step 1 does not install
any rules, as any rules that could be included with the install set would
quickly go stale.

You need to run sa-update once to get the current rules before
disconnecting from the Internet.

[Filippo Carletti]

You need to run sa-update once to get the current rules before disconnecting
from the Internet.

Exactly.
Real world scenario:
1. install new system with spamassassin in the afternoon in the office
2. verify everything it's working, SA runs with default rules from /usr/share
3. change ip address to match customer network, leave system on
4. during the night, sa-update runs, leave rules dir empty
5. next morning, shutdown system and drive to client office
6. boot system, verify it works in customer lan
7. spamd or amavisd will not run

I could workaround the problem, but I'm interested in opinions about a
patch to sa-update.

@Kevin, I'm using rpm packages.

[John Hardin]

Post by Filippo Carletti

You need to run sa-update once to get the current rules before disconnecting
from the Internet.

Exactly.
1. install new system with spamassassin in the afternoon in the office
2. verify everything it's working, SA runs with default rules from /usr/share

Where did those default rules come from? SA *does not ship* with rules in
the install set.

(Note: a packager may add a static rules snapshot to the standard SA
install set; this might have been done by whoever packaged up the RPM.)

Post by Filippo Carletti
3. change ip address to match customer network, leave system on
4. during the night, sa-update runs, leave rules dir empty
5. next morning, shutdown system and drive to client office
6. boot system, verify it works in customer lan
7. spamd or amavisd will not run
I could workaround the problem, but I'm interested in opinions about a
patch to sa-update.

Are you saying that running sa-update without network access *deletes*
existing rules? It's still not clear if that's what you're reporting. If
that's the case, yes, that's definitely a bug.

Post by Filippo Carletti
@Kevin, I'm using rpm packages.

--
John Hardin KA7OHZ http://www.impsec.org/~jhardin/
***@impsec.org FALaholic #11174 pgpk -a ***@impsec.org
key: 0xB8732E79 -- 2D8C 34F4 6411 F507 136C AF76 D822 E6E6 B873 2E79
-----------------------------------------------------------------------
Rights can only ever be individual, which means that you cannot
gain a right by joining a mob, no matter how shiny the issued
badges are, or how many of your neighbors are part of it. -- Marko
-----------------------------------------------------------------------
14 days until Bill of Rights day

[Filippo Carletti]

Post by John Hardin

Post by Filippo Carletti
2. verify everything it's working, SA runs with default rules from /usr/share

Where did those default rules come from? SA *does not ship* with rules in
the install set.

$ rpm -qf /usr/share/spamassassin/50_scores.cf
spamassassin-3.4.1-8.el6.x86_64

Post by John Hardin
(Note: a packager may add a static rules snapshot to the standard SA install
set; this might have been done by whoever packaged up the RPM.)

Right.
https://git.centos.org/blob/rpms!spamassassin/6c4364e0c267abccbafcaaa3ed79b2fe4d8e54b2/SPECS!spamassassin.spec;jsessionid=mc5igseasal0axkra6qhd4it
Line 229:
# Default rules from separate tarball
cd $RPM_BUILD_ROOT%{_datadir}/spamassassin/
tar xfvz %{SOURCE1}

Post by John Hardin
Are you saying that running sa-update without network access *deletes*
existing rules?

No. If the first time sa-update runs there's no network access, rules
are never downloaded.
"Never", because the centos cron job runs sa-update only if spamd or
amavisd are running (and they stop when there are no rules).

Please, bear with me not being able to report the problem clearly.

[Kevin A. McGrail]

Post by Filippo Carletti
No. If the first time sa-update runs there's no network access, rules
are never downloaded. "Never", because the centos cron job runs
sa-update only if spamd or amavisd are running (and they stop when
there are no rules). Please, bear with me not being able to report the
problem clearly.

For the SpamAssassin Project, that sounds like expected behavior. SA
does not include rules. You either use sa-update or manually install
the rules.

I do not know how CentOS has handled this issue.

Regards,
KAM

[John Hardin]

Post by Filippo Carletti

Post by John Hardin

Post by Filippo Carletti
2. verify everything it's working, SA runs with default rules from /usr/share

Where did those default rules come from? SA *does not ship* with rules in
the install set.

$ rpm -qf /usr/share/spamassassin/50_scores.cf
spamassassin-3.4.1-8.el6.x86_64

Post by John Hardin
(Note: a packager may add a static rules snapshot to the standard SA install
set; this might have been done by whoever packaged up the RPM.)

Right.
https://git.centos.org/blob/rpms!spamassassin/6c4364e0c267abccbafcaaa3ed79b2fe4d8e54b2/SPECS!spamassassin.spec;jsessionid=mc5igseasal0axkra6qhd4it
# Default rules from separate tarball
cd $RPM_BUILD_ROOT%{_datadir}/spamassassin/
tar xfvz %{SOURCE1}

Post by John Hardin
Are you saying that running sa-update without network access *deletes*
existing rules?

No. If the first time sa-update runs there's no network access, rules
are never downloaded.
"Never", because the centos cron job runs sa-update only if spamd or
amavisd are running (and they stop when there are no rules).

Ah, ok, that sounds like the problem to me: the RH/Centos sa-update cron
job should not depend on whether or not spamd/amavisd are currently
running.

I don't think this is a problem in base SA, it sounds more like a problem
in the packaging addon code provided by RH/Centos.

Post by Filippo Carletti
Please, bear with me not being able to report the problem clearly.

Not a problem.
--
John Hardin KA7OHZ http://www.impsec.org/~jhardin/
***@impsec.org FALaholic #11174 pgpk -a ***@impsec.org
key: 0xB8732E79 -- 2D8C 34F4 6411 F507 136C AF76 D822 E6E6 B873 2E79
-----------------------------------------------------------------------
Rights can only ever be individual, which means that you cannot
gain a right by joining a mob, no matter how shiny the issued
badges are, or how many of your neighbors are part of it. -- Marko
-----------------------------------------------------------------------
14 days until Bill of Rights day

[Quanah Gibson-Mount]

--On Tuesday, December 01, 2015 2:57 PM -0800 John Hardin

Post by John Hardin

Post by Filippo Carletti

Post by John Hardin

Post by Filippo Carletti
2. verify everything it's working, SA runs with default rules from /usr/share

Where did those default rules come from? SA *does not ship* with rules
in the install set.

$ rpm -qf /usr/share/spamassassin/50_scores.cf
spamassassin-3.4.1-8.el6.x86_64

Post by John Hardin
(Note: a packager may add a static rules snapshot to the standard SA
install set; this might have been done by whoever packaged up the RPM.)

Right.
https://git.centos.org/blob/rpms!spamassassin/6c4364e0c267abccbafcaaa3ed
79b2fe4d8e54b2/SPECS!spamassassin.spec;jsessionid=mc5igseasal0axkra6qhd4
# Default rules from separate tarball
cd $RPM_BUILD_ROOT%{_datadir}/spamassassin/
tar xfvz %{SOURCE1}

Post by John Hardin
Are you saying that running sa-update without network access *deletes*
existing rules?

No. If the first time sa-update runs there's no network access, rules
are never downloaded.
"Never", because the centos cron job runs sa-update only if spamd or
amavisd are running (and they stop when there are no rules).

Ah, ok, that sounds like the problem to me: the RH/Centos sa-update cron
job should not depend on whether or not spamd/amavisd are currently
running.
I don't think this is a problem in base SA, it sounds more like a problem
in the packaging addon code provided by RH/Centos.

Agreed. :P That's a failed installation/logic issue.

--Quanah

--

Quanah Gibson-Mount
Platform Architect
Zimbra, Inc.
--------------------
Zimbra :: the leader in open source messaging and collaboration

[John Hardin]

Post by Quanah Gibson-Mount
--On Tuesday, December 01, 2015 2:57 PM -0800 John Hardin

Post by John Hardin
Ah, ok, that sounds like the problem to me: the RH/Centos sa-update cron
job should not depend on whether or not spamd/amavisd are currently
running.
I don't think this is a problem in base SA, it sounds more like a problem
in the packaging addon code provided by RH/Centos.

Agreed. :P That's a failed installation/logic issue.

I'd suggest that the next step is either to bring this up on the SA Users
list, where the person who packages SA for RH/Centos may lurk, or open a
formal bug in the RH/Centos bugtraq.
--
John Hardin KA7OHZ http://www.impsec.org/~jhardin/
***@impsec.org FALaholic #11174 pgpk -a ***@impsec.org
key: 0xB8732E79 -- 2D8C 34F4 6411 F507 136C AF76 D822 E6E6 B873 2E79
-----------------------------------------------------------------------
Anyone who uses the word "profit" as a dirty word should be
watched very, very carefully. If they hate the idea of gain
through free trade it can only mean that they¢re looking to
get it through robbery. -- ***@Ultimak
-----------------------------------------------------------------------
14 days until Bill of Rights day

[RW]

On Tue, 1 Dec 2015 14:57:14 -0800 (PST)

Post by John Hardin
I don't think this is a problem in base SA, it sounds more like a
problem in the packaging addon code provided by RH/Centos.

It does sound like that might have been exposed by a SA regression:


https://bz.apache.org/SpamAssassin/show_bug.cgi?id=4941

[John Hardin]

Post by RW
On Tue, 1 Dec 2015 14:57:14 -0800 (PST)

Post by John Hardin
I don't think this is a problem in base SA, it sounds more like a
problem in the packaging addon code provided by RH/Centos.

https://bz.apache.org/SpamAssassin/show_bug.cgi?id=4941

Do you mean that in combination with bug 6655 as Filippo speculated?

I think I was too quick to suggest that this wasn't an issue in base SA,
apologies.

Filippo's timeline:

2. verify everything it's working, SA runs with default rules from /usr/share
3. change ip address to match customer network, leave system on
4. during the night, sa-update runs, leave rules dir empty

...indicates that sa-update *is* deleting good rules if the Internet is
inaccessible, so there seems to be two issues here: a bug in sa-update
that deletes good rules, and a bug in the RH/Centos cron job that keeps
the system from recovering from that when the network comes back.

Can else anyone confirm this in base (non-RH/Centos) SA? I'm not really in
a position to do so right now.
--
John Hardin KA7OHZ http://www.impsec.org/~jhardin/
***@impsec.org FALaholic #11174 pgpk -a ***@impsec.org
key: 0xB8732E79 -- 2D8C 34F4 6411 F507 136C AF76 D822 E6E6 B873 2E79
-----------------------------------------------------------------------
Rights can only ever be individual, which means that you cannot
gain a right by joining a mob, no matter how shiny the issued
badges are, or how many of your neighbors are part of it. -- Marko
-----------------------------------------------------------------------
14 days until Bill of Rights day

[RW]

On Tue, 1 Dec 2015 17:38:26 -0800 (PST)

Post by John Hardin

Post by RW
On Tue, 1 Dec 2015 14:57:14 -0800 (PST)

Post by John Hardin
I don't think this is a problem in base SA, it sounds more like a
problem in the packaging addon code provided by RH/Centos.

https://bz.apache.org/SpamAssassin/show_bug.cgi?id=4941

...indicates that sa-update *is* deleting good rules if the Internet
is inaccessible, so there seems to be two issues here: a bug in
sa-update that deletes good rules,

The rules aren't being deleted. What the OP reported is that the
sa-update failure leaves behind an empty 3.004001 directory which
overrides rules installed from package.

It's not clear whether it's a regression or a corner case that was
missed when bug 4941 was fixed, but if the OP is right that it came in
with the linked commit, then it sounds like a regression.

[Kevin A. McGrail]

Post by RW
On Tue, 1 Dec 2015 17:38:26 -0800 (PST)

Post by John Hardin

Post by RW
On Tue, 1 Dec 2015 14:57:14 -0800 (PST)

Post by John Hardin
I don't think this is a problem in base SA, it sounds more like a
problem in the packaging addon code provided by RH/Centos.

https://bz.apache.org/SpamAssassin/show_bug.cgi?id=4941

...indicates that sa-update *is* deleting good rules if the Internet
is inaccessible, so there seems to be two issues here: a bug in
sa-update that deletes good rules,

The rules aren't being deleted. What the OP reported is that the
sa-update failure leaves behind an empty 3.004001 directory which
overrides rules installed from package.
It's not clear whether it's a regression or a corner case that was
missed when bug 4941 was fixed, but if the OP is right that it came in
with the linked commit, then it sounds like a regression.

Overall, I'm very unclear of the issue. sa-update not working when not
on the internet is expected behavior. sa-update having a dependency to
run based on spamd/amavisd sounds like an implementation concern. sa
not working without rules which must be manually installed without
sa-update is also expected outcome.

Regards,
KAM

[Filippo Carletti]

sa-update not working when not on the internet is expected behavior.

Agreed, but sa-update leaves an empty rules directory that can break
spamd if it gets restarted after the empty dir has been created.
While it may seem an unlikely scenario, in real world it happened to
me at least 3 times in 2 months.

In 3.3.1 sa-update didn't leave the empty dir when it failed to download rules.

[Kevin A. McGrail]

Post by Filippo Carletti

sa-update not working when not on the internet is expected behavior.

Agreed, but sa-update leaves an empty rules directory that can break
spamd if it gets restarted after the empty dir has been created.
While it may seem an unlikely scenario, in real world it happened to
me at least 3 times in 2 months.
In 3.3.1 sa-update didn't leave the empty dir when it failed to download rules.

And I don't know if there was a reason that changed. 3.3.1 is not
exactly a very recent version with 3.3.2 replacing it in 2011.

Best I can suggest is open a bug in bugzilla, add an rmdir to your
update script, or disable the cron job since you have a machine not on
the internet.

Regards,
KAM

[John Hardin]

Post by RW
The rules aren't being deleted. What the OP reported is that the
sa-update failure leaves behind an empty 3.004001 directory which
overrides rules installed from package.

Ah, ok, thanks for clarifying.

Post by RW
It's not clear whether it's a regression or a corner case that was
missed when bug 4941 was fixed, but if the OP is right that it came in
with the linked commit, then it sounds like a regression.

It sounds like this part of it is worthy of a SA bug, then.
--
John Hardin KA7OHZ http://www.impsec.org/~jhardin/
***@impsec.org FALaholic #11174 pgpk -a ***@impsec.org
key: 0xB8732E79 -- 2D8C 34F4 6411 F507 136C AF76 D822 E6E6 B873 2E79
-----------------------------------------------------------------------
Justice is justice, whereas "social justice" is code for one set
of rules for the rich, another for the poor; one set for whites,
another set for minorities; one set for straight men, another for
women and gays. In short, it's the opposite of actual justice.
-- Burt Prelutsky
-----------------------------------------------------------------------
13 days until Bill of Rights day

[Kevin A. McGrail]

Post by Filippo Carletti

You need to run sa-update once to get the current rules before disconnecting
from the Internet.

Exactly.
1. install new system with spamassassin in the afternoon in the office
2. verify everything it's working, SA runs with default rules from /usr/share
3. change ip address to match customer network, leave system on
4. during the night, sa-update runs, leave rules dir empty
5. next morning, shutdown system and drive to client office
6. boot system, verify it works in customer lan
7. spamd or amavisd will not run
I could workaround the problem, but I'm interested in opinions about a
patch to sa-update.
@Kevin, I'm using rpm packages.

OK so running sa-update without internet is clearing your rules dir
before getting a new package?

Regards,
KAM