Talk:Machine Learning - Revision history

Mischief: Reverted edits by Javascript (talk) to last revision by Danf

2014-03-11T06:50:49Z

Reverted edits by Javascript (talk) to last revision by Danf

← Older revision		Revision as of 23:50, 10 March 2014
Line 26:		Line 26:
	ie write to vector (each index corresponds to a message)		ie write to vector (each index corresponds to a message)

	~~[[File:Javascript.jpg\|left\|link:{{-}}]]~~
	spamicity vector:		spamicity vector:
	msg0 msg1 msg2 msg3		msg0 msg1 msg2 msg3

Javascript at 05:58, 11 March 2014

2014-03-11T05:58:28Z

← Older revision		Revision as of 22:58, 10 March 2014
Line 26:		Line 26:
	ie write to vector (each index corresponds to a message)		ie write to vector (each index corresponds to a message)

			[[File:Javascript.jpg\|left\|link:{{-}}]]
	spamicity vector:		spamicity vector:
	msg0 msg1 msg2 msg3		msg0 msg1 msg2 msg3

Danf: feh

2014-03-07T10:54:16Z

feh

← Older revision		Revision as of 03:54, 7 March 2014
Line 4:		Line 4:

	== March 6, 2014 ==		== March 6, 2014 ==
			<pre>
	Group compared notes on list-scraping code and then delved into details of algorithm via tracing simple example.		Group compared notes on list-scraping code and then delved into details of algorithm via tracing simple example.

Line 117:		Line 117:
	we have been assuming independence		we have been assuming independence
	between words (at the full-message level)		between words (at the full-message level)
			</pre>

	== python to download and decompress nb-discuss archive ==		== python to download and decompress nb-discuss archive ==
	<pre>		<pre>
	import re		import re
	~~from itertools import chain, islice~~
	from StringIO import StringIO		from StringIO import StringIO
	from gzip import GzipFile		from gzip import GzipFile

Danf: meeting notes

2014-03-07T10:53:01Z

meeting notes

← Older revision		Revision as of 03:53, 7 March 2014
Line 2:		Line 2:

	Folks met and hacked on the noisebridge discuss mailing list. We created a 102MB text dump, and a python script to parse it, [[File:Py-piper-parser.txt]]. We wrote pseudo code to implement a Naive Bayesian filter to protect the world from trolls. Will implement soon.		Folks met and hacked on the noisebridge discuss mailing list. We created a 102MB text dump, and a python script to parse it, [[File:Py-piper-parser.txt]]. We wrote pseudo code to implement a Naive Bayesian filter to protect the world from trolls. Will implement soon.

			== March 6, 2014 ==

			Group compared notes on list-scraping code and then delved into details of algorithm via tracing simple example.

			user presented w message
			user labels spam/not spam or scale
			user presented with rating: 50% naive
			after user labels, algorithm readjusts

			very beginning
			first step: assign prior (50% of all messages are spam)
			also have a likelihood that each word is spam or not spam
			- combines to total 50% (also consider log-likelihood)
			next step: algorithm decides per message
			this is end of phase zero (PERFORMANCE PHASE)
			algorithm performing without human intervention

			next: TRAINING PHASE
			first step of training phase
			examine message, assign a rating (spamicity ie 7/7 or drama-tag or spam/not spam)
			then save rating associated with message
			ie write to vector (each index corresponds to a message)

			spamicity vector:
			msg0 msg1 msg2 msg3
			1 0 0 3

			then update dictionary
			what dictionary?
			temporary dictionary in the first step
			every dictionary item has a frequency count

			now ...
			after (say) 1000 messages, algorithm guesses mostly correctly
			'spam or not-spam' (is this TESTING PHASE ?)
			training continues via occasional instances of (human) correction
			update dictionary with each word in (human?) rated message

			one possibly viable dictionary structure:
			{'word':[counted_in_spam, counted_in_not_spam]}

			so, algorithm might operate as per this trace:

			msg[0]: 'foo bar' SPAM
			msg[1]: 'foo foo bar foo bar' HAM
			msg[2]: 'bar bar bar foo' ... WHAT IS THIS ????
			Can consult algorithm because now we have SPAM and HAM
			so can get bayes-informed result

			dictionary
			after msg[0]
			{'foo': [1, 0], 'bar': [1, 0]}

			after msg[1]
			{'foo': [1, 3], 'bar': [1, 2]}

			NOW WE ARE AT msg[2] WHAT IS THIS ???

			THIS LOOKS EASIER TO SOLVE THIS TIME !!!!!!!

			WE HAVE A VECTOR OF SPAM/HAM
			IT LOOKS LIKE THIS: ['s', 'h']
			OR IF YOU LIKE BINARY [True, False]
			OR [0, 1]
			OK I GET IT !!!!

			WE HAVE A VECTOR AND A DICTIONARY and a message NOW WHAT ???
			{'foo': [1, 3], 'bar': [1, 2]}
			[0, 1]
			msg[2]: 'bar bar bar foo' ... WHAT IS THIS ????

			A: probability of 'foo' \| spam = 1
			B: probability of spam = 0.5
			C: probability of 'foo' \| ham = 3
			... wtf ???
			this gets normalized later ? maybe
			sam hopes this cancels out without being painful
			D: probability of ham = 0.5

			= 0.25 likelihood given 'foo'

			(1 * 0.5) / ((1 * 0.5) + (3 * 0.5))
			A * B / ((A * B) + (C * D))
			= 0.25 likelihood given 'foo'

			1(.5)
			1 = prob of foo in message \| spam
			.5 = prob of any word \| spam
			.5 = prob of 'foo' in (first) word \| spam
			.5 = A (normed)

			3 = prob of foo in message(?) \| not spam
			1/5 = prob of word \| ham
			.6 = prob of foo in (first) word \| ham
			.6 = C (normed)

			likelihood given 'bar'
			(1 * 0.5) / ((1 * 0.5) + (2 * 0.5))
			= 0.3333... (1/3)

			(1/3.0)*3 (1/4.0) / (((1/3.0)*3 (1/4.0)) + ((2/3.0)*3 (3/4.0)))
			= 0.04
			The way this is not fully bayesian... p(foo) & p(bar) are interacting...
			Also, are we normalizing correctly?

			If we normalized,
			we take into account the following:
			avg freq of words in spam
			avg freq of words in ham

			but this is not fully bayesian
			because ... so far ...
			we have been assuming independence
			between words (at the full-message level)

	== python to download and decompress nb-discuss archive ==		== python to download and decompress nb-discuss archive ==

Danf: updated codes ftw

2014-03-07T02:49:47Z

updated codes ftw

Show changes

Danf: some more glue code to slurp nb-discuss content

2014-03-03T02:16:21Z

some more glue code to slurp nb-discuss content

@@ Line 5: / Line 5: @@
 == python to download and decompress nb-discuss archive ==
 <pre>
 from StringIO import StringIO
 from gzip import GzipFile
 from time import gmtime
 from urllib import urlopen
 def decompress_from_url(u):
-   # return GzipFile(fileobj = StringIO(urlopen(u).read())).read()
+   with closing(urlopen(u)) as f:
-  f = urlopen(u)
+    with closing(StringIO(f.read())) as fs:
-  fs = StringIO(f.read())
+      with GzipFile(fileobj = fs) as g:
-  g = GzipFile(fileobj = fs)
+        return g.read()
 def discuss_gz_url(m, y):
@@ Line 40: / Line 39: @@
        mm = range(11, 12 + 1)  # start with November 2007
      elif y == now.tm_year:
        mm = range(1, now.tm_mon + 1)  # end with current month
      else:
        mm = range(1, 13)
@@ Line 59: / Line 58: @@
      for s in spew():
        f.write(s)
 </pre>
 == Word parsing python script ==

Danf: utility code to retrieve & decompress archived nb-discuss

2014-03-02T06:47:37Z

utility code to retrieve & decompress archived nb-discuss

← Older revision		Revision as of 23:47, 1 March 2014
Line 1:		Line 1:
	== Feb. 27, 2014 ==		== Feb. 27, 2014 ==

	Folks met and hacked on the noisebridge discuss mailing list. We created a ~~93MB~~ text dump, and a python script to parse it, [[File:Py-piper-parser.txt]]. We wrote pseudo code to implement a Naive Bayesian filter to protect the world from trolls. Will implement soon.		Folks met and hacked on the noisebridge discuss mailing list. We created a 102MB text dump, and a python script to parse it, [[File:Py-piper-parser.txt]]. We wrote pseudo code to implement a Naive Bayesian filter to protect the world from trolls. Will implement soon.

			== python to download and decompress nb-discuss archive ==
			<pre>
			from StringIO import StringIO
			from gzip import GzipFile
			from time import gmtime
			from urllib import urlopen

			def decompress_from_url(u):
			# return GzipFile(fileobj = StringIO(urlopen(u).read())).read()
			f = urlopen(u)
			fs = StringIO(f.read())
			g = GzipFile(fileobj = fs)
			s = g.read()
			for x in (f, fs, g):
			x.close()
			return s

			def discuss_gz_url(m, y):
			if m < 1 or m > 12:
			return None
			if y < 2007:
			return None
			now = gmtime()
			if (y > now.tm_year) or (y == now.tm_year and m > now.tm_mon):
			return None
			mm = ('January', 'February', 'March', 'April', 'May', 'June', 'July', 'August', 'September', 'October', 'November', 'December')
			nb_pre = 'https://www.noisebridge.net/pipermail/noisebridge-discuss/'
			nb_post = '.txt.gz'
			s = '-'.join((str(y), mm[m-1]))
			return ''.join((nb_pre, s, nb_post))

			def all_discuss_gz_urls():
			now = gmtime()
			for y in range(2007, now.tm_year + 1):
			if y == 2007:
			mm = range(11, 12 + 1) # start with November 2007
			elif y == now.tm_year:
			mm = range(1, now.tm_mon + 1) # end with current month
			else:
			mm = range(1, 13)
			for m in mm:
			yield discuss_gz_url(m, y)

			def discuss_a_month(month, year):
			u = discuss_gz_url(month, year)
			s = decompress_from_url(u)
			return s

			def spew():
			for u in all_discuss_gz_urls():
			yield decompress_from_url(u)

			def dump_uncompressed(filename="nb_wtf.txt"):
			with open(filename, "w") as f:
			for s in spew():
			f.write(s)

			</pre>

199.241.200.45: /* Word parsing python script */

2014-02-28T07:06:27Z

Word parsing python script

← Older revision		Revision as of 00:06, 28 February 2014
Line 13:		Line 13:
	m = d['messageline']		m = d['messageline']
	yield m.split()		yield m.split()

			Plans to improve by using nltk[http://nltk.org]

199.241.200.45: /* Word parsing python script */

2014-02-28T07:04:16Z

Word parsing python script

← Older revision		Revision as of 00:04, 28 February 2014
Line 6:		Line 6:
	== Word parsing python script ==		== Word parsing python script ==

	# Function 'get_words' takes list of dictionary of emails.		Function 'get_words' takes list of dictionary of emails.
	# Yields:		Yields lists of words of in the message, for each message:
	~~# Lists~~ of words of in the message, for each message.
	def get_words(lst):		def get_words(lst):
	for d in lst:		for d in lst:
	m = d['messageline']		m = d['messageline']
	yield m.split()		yield m.split()

199.241.200.45: /* Feb. 27, 2014 */

2014-02-28T07:02:22Z

Feb. 27, 2014

← Older revision		Revision as of 00:02, 28 February 2014
Line 2:		Line 2:

	Folks met and hacked on the noisebridge discuss mailing list. We created a 93MB text dump, and a python script to parse it, [[File:Py-piper-parser.txt]]. We wrote pseudo code to implement a Naive Bayesian filter to protect the world from trolls. Will implement soon.		Folks met and hacked on the noisebridge discuss mailing list. We created a 93MB text dump, and a python script to parse it, [[File:Py-piper-parser.txt]]. We wrote pseudo code to implement a Naive Bayesian filter to protect the world from trolls. Will implement soon.


			== Word parsing python script ==

			# Function 'get_words' takes list of dictionary of emails.
			# Yields:
			# Lists of words of in the message, for each message.
			def get_words(lst):
			for d in lst:
			m = d['messageline']
			yield m.split()