Machine Learning/Kaggle Social Network Contest/load data - Revision history

SpammerHellDontDelete: /* igraph */

2010-11-23T20:28:58Z

igraph

← Older revision		Revision as of 13:28, 23 November 2010
Line 13:		Line 13:
	dg <- graph.edgelist(data, directed=TRUE)		dg <- graph.edgelist(data, directed=TRUE)
	</pre>		</pre>

			Note that the resulting graph contains an additional vertex with id zero. If you delete this vertex the id names will not be preserved, and so it is a good idea to just leave it in there. The vertex zero has no edges.

	=Python=		=Python=

SpammerHellDontDelete: moved R to the top of the page

2010-11-23T06:30:05Z

moved R to the top of the page

@@ Line 1: / Line 1: @@
 =Python=
 == How to load the network into networkx ==
 There is a network analysis package for Python called [http://networkx.lanl.gov/ networkx]. This package can be installed using easy_install.
@@ Line 75: / Line 89: @@
 adj_list_lookup = load_adj_list_faster('adj_list.out.csv')
 rev_adj_list_lookup = load_adj_list_faster('reverse_adj_list.out.csv')
 </pre>

Jjhale: /* R */

2010-11-23T02:24:35Z

← Older revision		Revision as of 19:24, 22 November 2010
Line 79:		Line 79:

	= R =		= R =
			== igraph ==
			The full dataset loaded pretty fast using the R package igraph. With the full data set loaded R is using less than 900MB of RAM.

			Grab the package with:
			<pre>
			install.packages("igraph")
			</pre>

			Load the data using:
			<pre>
			data <-as.matrix(read.csv("social_train.csv", header = FALSE));
			dg <- graph.edgelist(data, directed=TRUE)
			</pre>

Jared: /* Note on CSV Libraries */

2010-11-20T11:22:31Z

Note on CSV Libraries

← Older revision		Revision as of 04:22, 20 November 2010
Line 57:		Line 57:

	== Note on CSV Libraries ==		== Note on CSV Libraries ==
	If you happen to be using Ruby (like Jared) for loading data in and out of CSV files, you should definitely try [[ http://fastercsv.rubyforge.org/ \| FasterCSV ]] instead of the stock CSV (~~import~~ 'csv'). For example, when loading the adjacency list it was literally ten times faster using FasterCSV than using the normal CSV.		If you happen to be using Ruby (like Jared) for loading data in and out of CSV files, you should definitely try [[ http://fastercsv.rubyforge.org/ \| FasterCSV ]](require 'faster_csv') instead of the stock CSV (require 'csv'). For example, when loading the adjacency list it was literally ten times faster using FasterCSV than using the normal CSV.

	== Loading Adjacency Lists ==		== Loading Adjacency Lists ==

Jared: /* Note on CSV Libraries */

2010-11-20T11:09:28Z

Note on CSV Libraries

← Older revision		Revision as of 04:09, 20 November 2010
Line 57:		Line 57:

	== Note on CSV Libraries ==		== Note on CSV Libraries ==
	If you happen to be using Ruby (like Jared) for loading data in and out of CSV files, you should definitely try [[http://fastercsv.rubyforge.org/\|FasterCSV]] instead of the stock CSV (import 'csv'). For example, when loading the adjacency list it was literally ten times faster using FasterCSV than using the normal CSV.		If you happen to be using Ruby (like Jared) for loading data in and out of CSV files, you should definitely try [[ http://fastercsv.rubyforge.org/ \| FasterCSV ]] instead of the stock CSV (import 'csv'). For example, when loading the adjacency list it was literally ten times faster using FasterCSV than using the normal CSV.

	== Loading Adjacency Lists ==		== Loading Adjacency Lists ==

Jared: /* How to load the network into networkx */

2010-11-20T11:08:45Z

How to load the network into networkx

@@ Line 1: / Line 1: @@
 == How to load the network into networkx ==
 There is a network analysis package for Python called [http://networkx.lanl.gov/ networkx]. This package can be installed using easy_install.
@@ Line 6: / Line 8: @@
 NOTE: John found that it took up about 5.5GB of memory to load the entire network. We may need to process it in chunks - or maybe decompose it into smaller sub networks.
-Method 1
+'''Method 1'''
 <pre>
 import networkx as nx
@@ Line 12: / Line 14: @@
 </pre>
+'''Method 2'''
 <pre>
 import networkx as nx
@@ Line 52: / Line 52: @@
 | 86s
 |}

SpammerHellDontDelete: /* How to load the network into networkx */

2010-11-20T01:35:13Z

How to load the network into networkx

← Older revision		Revision as of 18:35, 19 November 2010
Line 34:		Line 34:
	</pre>		</pre>

	{\| border="1"		Below is the time to load different numbers of row using the two methods on a 2.8Ghz Quad core machine with 3GB RAM. The second method seems quicker. Note that these are just based on single loads and are intended to be a guide rather than a rigorous analysis of the methods!
			{\| border="1"
	\|-		\|-
	!\|Rows		!\|Rows

Jjhale: /* How to load the network into networkx */

2010-11-19T06:24:32Z

How to load the network into networkx

← Older revision		Revision as of 23:24, 18 November 2010
Line 3:		Line 3:

	The network can be loaded using the [http://networkx.lanl.gov/reference/generated/networkx.read_edgelist.html read_edgelist] function in networkx or by manually adding edges		The network can be loaded using the [http://networkx.lanl.gov/reference/generated/networkx.read_edgelist.html read_edgelist] function in networkx or by manually adding edges

			NOTE: John found that it took up about 5.5GB of memory to load the entire network. We may need to process it in chunks - or maybe decompose it into smaller sub networks.

	Method 1		Method 1

Jjhale at 06:19, 19 November 2010

2010-11-19T06:19:28Z

@@ Line 2: / Line 2: @@
 There is a network analysis package for Python called [http://networkx.lanl.gov/ networkx]. This package can be installed using easy_install.
-The network can be loaded using the [http://networkx.lanl.gov/reference/generated/networkx.read_edgelist.html read_edgelist] function in networkx
+The network can be loaded using the [http://networkx.lanl.gov/reference/generated/networkx.read_edgelist.html read_edgelist] function in networkx or by manually adding edges
-eg:
 <pre>
 import networkx as nx
@@ Line 10: / Line 11: @@
 <pre>
 import networkx as nx
@@ Line 32: / Line 31: @@
 print "Loaded in ", str(time.clock() - t0), "s"
 </pre>

Jjhale: Created page with '== How to load the network into networkx == There is a network analysis package for Python called [http://networkx.lanl.gov/ networkx]. This package can be installed using easy_i…'

2010-11-19T05:40:31Z

Created page with '== How to load the network into networkx == There is a network analysis package for Python called [http://networkx.lanl.gov/ networkx]. This package can be installed using easy_i…'

New page

== How to load the network into networkx ==
There is a network analysis package for Python called [http://networkx.lanl.gov/ networkx]. This package can be installed using easy_install.

The network can be loaded using the [http://networkx.lanl.gov/reference/generated/networkx.read_edgelist.html read_edgelist] function in networkx
eg:
<pre>
import networkx as nx
DG = nx.read_edgelist('social_train.csv', create_using=nx.DiGraph(), nodetype=int, delimiter=',')
</pre>

Loading 1M rows of the edge list took 21s on a MacPro with 3GB mem and 2.8Ghz Quad-Core processor. I can do this in 15s with the following.

An alternate method of loading it is the follow which seems to run quicker for me (Joe).

<pre>
import networkx as nx
import csv
import time

t0 = time.clock()
DG = nx.DiGraph()

netcsv = csv.reader(open('social_train.csv', 'rb'), delimiter=',')

for row in netcsv:
tmp1 = int(row[0])
tmp2 = int(row[1])
DG.add_edge(tmp1, tmp2)

print "Loaded in ", str(time.clock() - t0), "s"
</pre>