crm114-developers Mailing List for CRM114 Discriminator - Controllable Rege

<Mar...@Mc...> writes:
> [1:multipart/alternative Hide]
>
>
> [1/1:text/plain Hide]
>
> Hello, I looked around the code and couldn't find any mention of
> whether base64 encoded content is handled.. so I guess that README at
> http://crm114.sourceforge.net/docs/README should not be trusted :-) In
> the event that libcrm114 is looking at a message containing either
> text or attachments encoded in base64, does that mean I need to
> preprocess the content before using the lib?
That depends... you can do decent spam filtering without bothering
to unexpand base64, but it will *usually* work better if you do.
There are a number of base64 (or MIME-) expanders out there; pick
whichever one you like the best and fits best into your workflow.
 - Bill

2013年4月20日 <Mar...@mc...>:
> Hello,
>
> I looked around the code and couldn’t find any mention of whether base64
> encoded content is handled.. so I guess that README at
> http://crm114.sourceforge.net/docs/README should not be trusted :-)
it refers to mailfilter.crm which uses an external *coder like
mimencode, see mailfilter.cf
#:do_base64: /no/
:do_base64: /yes/
#
#:mime_decoder: /mewdecode/
#:mime_decoder: /mimencode -d/
#:mime_decoder: /mimencode -u/
#:mime_decoder: /base64 -d/
:mime_decoder: /openssl base64 -d/
#:mime_decoder: /normalizemime/
there was the idea to bring in the b64 *coder for obvious reasons, but
I guess it's not been done.
> In the event that libcrm114 is looking at a message containing either text
> or attachments encoded in base64, does that mean I need to preprocess the
> content before using the lib?
AFAICT libcrm114 is about classifiers only, so yes you need to preprocess.
HTH
-- paolo

Hello,
I looked around the code and couldn't find any mention of whether base64 encoded content is handled.. so I guess that README at http://crm114.sourceforge.net/docs/README should not be trusted :-)
In the event that libcrm114 is looking at a message containing either text or attachments encoded in base64, does that mean I need to preprocess the content before using the lib?
Thanks,
Martin
Martin Thomas
Research Tools Software Architect
McAfee Labs Messaging Security
McAfee, Inc.
972.963.7680 Direct
mar...@mc...<mailto:mar...@mc...>

On Wed, Jun 03, 2009 at 01:59:31AM +0200, paolo wrote:
> In HS, LEARNing into a non existing cssfile results in segv.
also in CLASSIFYing, if no cssfile could be opened then hashname is NULL
and sprintf() @<~1499 bangs out with segv.
Proposed patch attached.
-- 
paolo

In HS, LEARNing into a non existing cssfile results in segv.
Missing FILE* == NULL check, see attached patch.
BTW:
"vlaurika just announced version 0.7.6 of TRE on freshmeat.net.
 The changes are as follows:
 The license has been changed from LGPL to a 2-clause BSD-style license.
 ...
 http://freshmeat.net/projects/tre#release_299841
"
-- 
paolo

 From: "Eric S. Johansson" <es...@ha...>
 they have a 'free for oss' service. it has helped the Linux kernel
 and other complex projects improve code quality. think crm114 could
 benefit?
 http://coverity.com/
I've been approached by them before.
* it would probably help...
* but the time investiment to learn how to use the tools is 
 way out of budget right now.
Maybe after we've transitioned to libcrm114 (which involves massive
refactoring and de-crufting some 10-year-old "give it a try" code).
	 - Bill Yerazunis

they have a 'free for oss' service. it has helped the Linux kernel and other
complex projects improve code quality. think crm114 could benefit?
http://coverity.com/

In latest sf.net news (for those not subscr to the newslwetter):
Driven by unmistakably clear user demand, we have added support for Git, a
distributed source control system. We feel that Git, accompanied by our
existing CVS and Subversion support, puts us on the path toward a
well-rounded source code management toolset - even if we don't support
every major system (yet.)
-- 
 paolo
 
 GPG/PGP id:0x3A47DE45 - B5F9 AAA0 44BD 2B63 81E0 971F C6C0 0B87 3A47 DE45

On Mon, Feb 23, 2009 at 8:01 PM, Bill Yerazunis <ws...@me...> wrote:
> I don't suppoose you could also reveal the pre-fixed code
> and line numbers, could you, with +/- line prefixes?
>
> - Bill
Sure, here's the hacked diff file with GerH (anyway, it's just the
same as grabbing a line from those snippets and / or Ctrl-F finding
them in the source: it's safe as those bits only occur at one spot in
the source file (no [semi-]duplicates in other sections).
@@ -637,17 +1022,18 @@
 //
 // Again, we sacrifice neuron 0 of the hidden layer to be the
 // bias weights.
-
- nn->output_layer[0] = *arefWout (nn, 0, 0);
- nn->output_layer[0] = *arefWout (nn, 1, 0);
+ nn->output_layer[0] = nn->Wout[0]; // *arefWout(nn, 0, 0);
+ nn->output_layer[1] = nn->Wout[(1 * nn->hidden_layer_size) + 0];
// *arefWout(nn, 1, 0); [i_a] bug
It's the second [0] anyway.
plus:
@@ -1523,20 +2270,26 @@
 	
 	// Calculate the SUM "in place" (the error mag term)
 	for (neuron = 0; neuron < nn->first_layer_size; neuron++)
+ {
 	 for (channel = 0; channel < nn->hidden_layer_size; channel++)
+ {
 	 nn->delta_first_layer[neuron] +=
-	 nn->delta_hidden_layer[neuron]
-	 * (*arefWhid(nn, channel, neuron));
+ nn->delta_hidden_layer[channel] // [i_a] bug
+ * nn->Whid[channel *
nn->first_layer_size + neuron];
+	// (*arefWhid(nn, channel, neuron));
+ }
+ }
-- 
Met vriendelijke groeten / Best regards,
Ger Hobbelt
--------------------------------------------------
web: http://www.hobbelt.com/
 http://www.hebbut.net/
mail: ge...@ho...
mobile: +31-6-11 120 978
--------------------------------------------------

 From: Ger Hobbelt <ge...@ho...>
 Before I forget again: here's two bugs in the implementation:
I don't suppoose you could also reveal the pre-fixed code 
and line numbers, could you, with +/- line prefixes?
 - Bill
 snippets are the FIXED code for GerH crm_neural_net.c (to be released):
 // Generate the values in the final output layer (both neurons)
 //
 // Again, we sacrifice neuron 0 of the hidden layer to be the
 // bias weights.
 nn->output_layer[0] = nn->Wout[0]; // *arefWout(nn, 0, 0);
 nn->output_layer[1] = nn->Wout[(1 * nn->hidden_layer_size) + 0];
 // *arefWout(nn, 1, 0); [i_a] bug
 and
		 // Calculate the SUM "in place" (the error mag term)
		 for (neuron = 0; neuron < nn->first_layer_size; neuron++)
		 {
		 for (channel = 0; channel < nn->hidden_layer_size;
 channel++)
		 {
			 nn->delta_first_layer[neuron] +=
			 nn->delta_hidden_layer[channel] // [i_a] bug
			 * nn->Whid[neuron * nn->first_layer_size
 + channel];
	 // (*arefWhid(nn, channel, neuron));
		 }
		 }
 Number two will cause spurious coredumps on well behaved platforms,
 thanks to addressing way out of (allocated) array bounds. (Memory
 protection errors)
 Number one is an init issue, which I am not entirely sure about yet.
 anyway, > 90%
 -- 
 Met vriendelijke groeten / Best regards,
 Ger Hobbelt
 --------------------------------------------------
 web: http://www.hobbelt.com/
	 http://www.hebbut.net/
 mail: ge...@ho...
 mobile: +31-6-11 120 978
 --------------------------------------------------
 ------------------------------------------------------------------------------
 Open Source Business Conference (OSBC), March 24-25, 2009, San Francisco, CA
 -OSBC tackles the biggest issue in open source: Open Sourcing the Enterprise
 -Strategies to boost innovation and cut costs with open source participation
 -Receive a 600ドル discount off the registration fee with the source code: SFAD
 http://p.sf.net/sfu/XcvMzF8H
 _______________________________________________
 CRM114-developers mailing list
 CRM...@li...
 https://lists.sourceforge.net/lists/listinfo/crm114-developers

This one was still waiting in the sent queue since oct/2008 :-(
On Mon, Oct 6, 2008 at 7:42 PM, Bill Yerazunis <ws...@me...> wrote:
> ../src/crm114 '-{window;output/:*:arg2:\n/}' --arg2=hooba
[...]
> Nope. Things that aren't set are ... well, not set! An unset
> variable :foo: evaluates to exactly that --> ":foo:".
>
> The "--" command line operator is just like the one in X windows- args
> before it are used by the controlling program _exclusively_ and args
> after it are used by the controlled program _exclusively_.
>
> Given that, the above behavior is correct.
Okay. Verified.
Here's another one that, contrary to this, is a bug:
./crm114 '-{window;output /:*:_posc:\n/}'
should report? 2? (given the comments in there:
		// :_pos0: is always the name of the CRM114 engine.
		// :_pos1: is always the name of the program being run.
		// :_pos2: and so on are the command line args.
Now try the same with one or 2 args:
./crm114 '-{window;output /:*:_posc:\n/}' A
-->
1
./crm114 '-{window;output /:*:_posc:\n/}' A B
-->
2
The only script which used this (up to now; I'm using them as well)
was pad.crm which went along with the comments by peeking at :_pos2:.
Which makes sense, because if you wish to know the script filename,
one should always use :_pos1: and not :_argv1: as the script doesn't
have to be argv[1], e.g.
./crm114 -u . script.crm
where the script is at argv[3] but still at the (documented) :_pos1:
GerH build is fixed for this, back in '08.
-- 
Met vriendelijke groeten / Best regards,
Ger Hobbelt
--------------------------------------------------
web: http://www.hobbelt.com/
 http://www.hebbut.net/
mail: ge...@ho...
mobile: +31-6-11 120 978
--------------------------------------------------

refute marker uses wrong k index: vanilla snippet which is wrong:
 // Did we find a matching sum
 if(found_duplicate)
 {
 k[1] = 0;
 if (apb->sflags & CRM_REFUTE)
	{
	 if (neural_trace)
	 fprintf (stderr, "Marking this doc as out-of-class.\n");
	 k[1] = 1;
here, k[0] should've been used as that index (k[0]) is used to signal
in/out of class, while k[1] is used (see further down the code) for
counting the number of rounds - which in vanilla is, alas, unused ATM.
Nevertheless, it *does* incapacitate <refute> for NN. Especially note
this bit:
	 // Offset 1 - Write the number of passes without a
	 // retrain on this one
and the code here:
	 k++;
	 // save a pointer to the "OK this many times" counter
	 l = k;
Found while implementing a speedup which scans through the documents
in the CSS far faster than the linear scans in the current vanilla
code, which assumes the bag-of-hashes cannot contain zero or 1 valued
hashes[*] (the new code doesn't rely on that assumptions, so 1-valued
hashes are perfectly fine, as they should).
[*] This vanilla behaviour comes with its own bug, but I'll post that
one when I'm in the mood again, as that turned out to be the cause of
almost all those fine coredumps I had in megatest since the advent of
NN. And experimental or not: stuff shouldn't coredump. But that's
probably a mere 'preference' instead of a basic engineering quality
indicator.
-- 
Met vriendelijke groeten / Best regards,
Ger Hobbelt
--------------------------------------------------
web: http://www.hobbelt.com/
 http://www.hebbut.net/
mail: ge...@ho...
mobile: +31-6-11 120 978
--------------------------------------------------

Before I forget again: here's two bugs in the implementation:
snippets are the FIXED code for GerH crm_neural_net.c (to be released):
 // Generate the values in the final output layer (both neurons)
 //
 // Again, we sacrifice neuron 0 of the hidden layer to be the
 // bias weights.
 nn->output_layer[0] = nn->Wout[0]; // *arefWout(nn, 0, 0);
 nn->output_layer[1] = nn->Wout[(1 * nn->hidden_layer_size) + 0];
// *arefWout(nn, 1, 0); [i_a] bug
and
 // Calculate the SUM "in place" (the error mag term)
 for (neuron = 0; neuron < nn->first_layer_size; neuron++)
 {
 for (channel = 0; channel < nn->hidden_layer_size;
channel++)
 {
 nn->delta_first_layer[neuron] +=
 nn->delta_hidden_layer[channel] // [i_a] bug
 * nn->Whid[neuron * nn->first_layer_size
+ channel];
	// (*arefWhid(nn, channel, neuron));
 }
 }
Number two will cause spurious coredumps on well behaved platforms,
thanks to addressing way out of (allocated) array bounds. (Memory
protection errors)
Number one is an init issue, which I am not entirely sure about yet.
anyway, > 90%
-- 
Met vriendelijke groeten / Best regards,
Ger Hobbelt
--------------------------------------------------
web: http://www.hobbelt.com/
 http://www.hebbut.net/
mail: ge...@ho...
mobile: +31-6-11 120 978
--------------------------------------------------

On Thu, Feb 12, 2009 at 11:10 PM, Kurt Hackenberg <hac...@me...> wrote:
> The immediate reason is that I wondered why inttypes.h (C) was included
> only in a Unix compile. More generally, I'm doing some work on it, and
> would like not to break anything I don't have to.
inttypes.h is not available with MSVC2005, alas. Hence the fallback to
the MSVC alternative in config.h
-- 
Met vriendelijke groeten / Best regards,
Ger Hobbelt
--------------------------------------------------
web: http://www.hobbelt.com/
 http://www.hebbut.net/
mail: ge...@ho...
mobile: +31-6-11 120 978
--------------------------------------------------

Ger Hobbelt wrote:
> Compilers used on Windows: MSVC2003 (now not supported any longer) and
> MSVC2005. Move to MSVC2008 is considered for Q2 / Q3 this year.
>
> And ISO C compliant... which one? C89? C99?
>
> C89? yep.
> C99? yep. with caveats. (most importantly, no int8_t, int16_t,
> int32_t, int64_t but the Microsoftie equivalents __int8, __int16, etc.
> Nothing a little config.h can't handle :-) )
>
> 
Thanks.
> Why the ISO question, by the way? (Or were you expecting some
> lingering pre-ANSI C code in there? ;-P )
>
> 
The immediate reason is that I wondered why inttypes.h (C) was included 
only in a Unix compile. More generally, I'm doing some work on it, and 
would like not to break anything I don't have to.

Compilers used on Windows: MSVC2003 (now not supported any longer) and
MSVC2005. Move to MSVC2008 is considered for Q2 / Q3 this year.
And ISO C compliant... which one? C89? C99?
C89? yep.
C99? yep. with caveats. (most importantly, no int8_t, int16_t,
int32_t, int64_t but the Microsoftie equivalents __int8, __int16, etc.
Nothing a little config.h can't handle :-) )
For any other platforms, it's GCC (v3, v4 both tested; v2 is a
probably maybe). And I'd LOVE to see this bugger built on a VMS
system, preferably with a DEC C compiler. Ah well, we all have our
dreams...
On the other hand, the code is C89 / C99 compliant.
Of course everything above is for crm114-GerH (available at
hebbut.net); if you wish to travel the vanilla route, the motto of the
day is patch your Makefile and have at it.
Regarding Windows ports: there are two around google-wise, at least
that I know of: the original one by Jesusfreke, on which I built my
own, and then the one at hebbut. AFAIK Jesusfreke has stopped
supporting his (see his website), so that leaves one. ;-)
Why the ISO question, by the way? (Or were you expecting some
lingering pre-ANSI C code in there? ;-P )
Cheers,
Ger
On Thu, Feb 12, 2009 at 9:37 PM, Kurt Hackenberg <hac...@me...> wrote:
> I understand there have been a couple ports of CRM114 to Microsoft
> Windows, not done or supported by Bill (the author), but partly included
> in the distributed code. Anybody know what compilers/systems were
> used? Specifically, were they ISO C compilers?
-- 
Met vriendelijke groeten / Best regards,
Ger Hobbelt
--------------------------------------------------
web: http://www.hobbelt.com/
 http://www.hebbut.net/
mail: ge...@ho...
mobile: +31-6-11 120 978
--------------------------------------------------

I understand there have been a couple ports of CRM114 to Microsoft 
Windows, not done or supported by Bill (the author), but partly included 
in the distributed code. Anybody know what compilers/systems were 
used? Specifically, were they ISO C compilers?

 From: Paolo <oo...@us...>
		 // "to a nonexistent variable; I'll created an "
		 //"isolated one. Hope that's OK. Varname was",
		 // outbuf);
 ...
 but the problem is deeper:
 $ ./crm114 '-{window; isolate (xxx) //}'
 uh? isn't that an illegal varname ? ...
Yeah, it is.
 $ ./crm114 '-{window; isolate (xxx) //; output /:*:xxx:\n/}'
 Ztart corrupted Zend corrupted i 3470 len 3 name = -xxx- :xxx:
 ah, it is - only that we don't check (anymore) at due time. Did the specs 
 change meanwhile?
Nope, it's still an illegal variable.
The code to test is somewhere broken though.
I need to look into that.
 - Bill Yerazunis

 From: Paolo <oo...@us...>
 On Sat, Oct 04, 2008 at 09:25:57PM +0200, Ger Hobbelt wrote:
 > # test the alius statement
 > {
 > 	# test comment to skip \# window
 > 	{ # and this one \#
 > 		# and another one \#{	
 seems to work so far 
 > 		
 > 		# so? \#	output # what? \#/checking for a foo.../#if we mingle
 > comments with code? \# # like this?
 out of spec/unsupported ATM; good to write steganocode ;)
Well, the specification says it's OK.
However, somehow that code got broken. I *suspect* it was when
the "uncomment" */ got put in.
That's the second thing that needs fixing.
 - Bill Yerazunis

On Sat, Oct 04, 2008 at 09:21:42PM +0200, Ger Hobbelt wrote:
> 		}
> 		alius # what to do if we don't use/have a cache:
> 		{
...
> FIX?
1. don't inline comments (too much - or do simple tests 1st ;} )
2. protected inline, different indenting:
 }
 alius { # what to do if we don't use/have a cache:
 match /bar/
yes, simpler than hacking the parser ;)
-- 
paolo

On Sun, Oct 05, 2008 at 09:30:53PM +0200, Ger Hobbelt wrote:
> :_lazy:
> 
> remains a space for ever.
unless you set otherwise to your taste.
 
> :_fault:
> 
> despite the comments in the code, it'll be empty. Always. Hence the
> test in 'uncaughttraptest.crm' will always produce the same result.
in README:
 ...
 not turn into a fatal. Also, the :_fault: variable is gone, each
 TRAP now specifies it's own fault code.
 ...
-- 
paolo

 From: Paolo <oo...@us...>
 **** Neural Network Classifier 
 type Q 
 -CLASSIFY fails; success probability: 0.398096 pR: -52.5471
 -Best match to file #1 (q_test.css) prob: 0.6019 pR: 52.5471 
 +CLASSIFY fails; success probability: 0.239376 pR: 13.6446
 +Best match to file #1 (q_test.css) prob: 0.7606 pR: -13.6446 
 Total features in input file: 268
 -#0 (i_test.css): icnr: 0.52 ocnr: 0.51 prob: 3.98e-01, pR: 0.13
 -#1 (q_test.css): icnr: 0.77 ocnr: 0.25 prob: 6.02e-01, pR: 52.55
 +#0 (i_test.css): icnr: 0.14 ocnr: 0.86 prob: 2.39e-01, pR: -72.75
 +#1 (q_test.css): icnr: 0.46 ocnr: 0.59 prob: 7.61e-01, pR: -13.64
 type Q 
 -CLASSIFY fails; success probability: 0.216038 pR: -55.2649
 -Best match to file #1 (q_test.css) prob: 0.7840 pR: 55.2649 
 +CLASSIFY fails; success probability: 0.165702 pR: -93.5506
 +Best match to file #1 (q_test.css) prob: 0.8343 pR: 93.5506 
 Total features in input file: 372
 -#0 (i_test.css): icnr: 0.22 ocnr: 0.80 prob: 2.16e-01, pR: -59.18
 -#1 (q_test.css): icnr: 0.79 ocnr: 0.26 prob: 7.84e-01, pR: 55.26
 +#0 (i_test.css): icnr: 0.18 ocnr: 0.80 prob: 1.66e-01, pR: -61.08
 +#1 (q_test.css): icnr: 0.97 ocnr: 0.05 prob: 8.34e-01, pR: 93.55
 Alternating Example Neural Network Classifier TRAINING 
 and that's on Debian Woody (gcc 2.95.4, libc 2.2.5, x86).
 So either _knowngood is out of sync, or gcc/libc made a big diff here.
It's entirely possible.
Neural network learning is inherently a chaotic process. I've 
seen _trivial_ changes in initial conditions lead to very different
networks and similarly large changes of scale in the results.
Yeah, I know it makes testing very difficult... but I don't have
a good answer for you.
 - Bill Yerazunis

On Sat, Oct 04, 2008 at 09:25:57PM +0200, Ger Hobbelt wrote:
> # test the alius statement
> {
> 	# test comment to skip \# window
> 	{ # and this one \#
> 		# and another one \#{	
seems to work so far 
> 		
> 		# so? \#	output # what? \#/checking for a foo.../#if we mingle
> comments with code? \# # like this?
out of spec/unsupported ATM; good to write steganocode ;)
> 		alius # a line comment
parser bug - see other msg
> 		{
> 			# multiple commands on a single line; no semicolons to separate
> them though ;-) parser should be able to cope...
> 			output /no foo... checking for bar.../			match /bar/			output
> /Found a bar. :*:_nl:/
out of spec (ATM).
-- 
paolo

On Sun, Oct 05, 2008 at 01:00:00PM +0200, Ger Hobbelt wrote:
> Anyway, here's the SECOND bug as Paolo also found out:
...
> isolate (:c:) /ABC/
> 
> call /:func:/ (:d:)
> 
> output /C = :*:c:\n/
> output /D = :*:d:\n/
seems it's by design:
crm_exec_engine.c:786:
...
 if (vht[ret_idx] == NULL)
 {
 // nonfatalerror 
 // ("Your call statement wants to return a value "
 // "to a nonexistent variable; I'll created an "
 //"isolated one. Hope that's OK. Varname was",
 // outbuf);
...
but the problem is deeper:
$ ./crm114 '-{window; isolate (xxx) //}'
uh? isn't that an illegal varname ? ...
$ ./crm114 '-{window; isolate (xxx) //; output /:*:xxx:\n/}'
Ztart corrupted Zend corrupted i 3470 len 3 name = -xxx- :xxx:
ah, it is - only that we don't check (anymore) at due time. Did the specs 
change meanwhile?
-- 
paolo

On Mon, Oct 06, 2008 at 04:09:54PM -0400, Bill Yerazunis wrote:
> 
> 3. $ ./crm114 '-{window;output /:*:arg2:\n:*:arg3:\n/}' \
> 	 --arg2=hooba -- --arg3=bahoo
> :arg2:
> bahoo
> 
> but that's wrong, we should get same as 2 above.
> 
> I don't think so. --arg2 is visible to the CRM114 startup engine,
> but NOT to the program being run. 
so? in (2) both are like arg2 above and both get set, likewise in (4)
both are like arg3 here and both get set. All cases should look the same 
for inline script above.
$./crm114 -help
...
--x=y creates var :x: with value 'y'
so inline script in case (3) above should see a pre-set :arg2: like
eg :_nl: but it doesn't.
I don't see why case (3) above should be deemed correct.
-- 
paolo