-
Notifications
You must be signed in to change notification settings - Fork 0
/
Commands_For_Full_Example.txt
78 lines (61 loc) · 4.49 KB
/
Commands_For_Full_Example.txt
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
For constructing only Entity (or Element) Index
Pre-Processing Steps: Download entities.zip and sameAs.zip from =http://islcatalog.ics.forth.gr/dataset/lodsyndesis and upload them to HDFS.
For URIs
hadoop fs -mkdir URIs
Unzip entities.zip and upload each file to HDFS: hadoop fs -put <file> URIs/
For SameAs
Unzip sameAs.zip
hadoop fs -mv 1000_sameAs.nt URIs/
1. Prefix Index \& SameAs Prefix Index
hadoop jar LODsyndesis.jar gr.forth.ics.isl.indexes.CreatePrefixIndex URIs prefixIndexes 1
2. EntityEquivalenceCatalog
hadoop jar LODsyndesis.jar gr.forth.ics.isl.sameAsCatalog.GetNeighborsSameAs URIs/1000_sameAs.nt nbrs 32
hadoop jar LODsyndesis.jar gr.forth.ics.isl.sameAsCatalog.HashToMin nbrs/sameAsP sameAs prefixIndexes/sameAsPrefix/sameAsPrefix.txt-r-00000 32 1000000 1
hadoop fs -getmerge sameAs/sameAs1/sameAsCatalog/ sameAsCatalog1.txt
hadoop fs -put sameAsCatalog1.txt URIs/
hadoop fs -getmerge sameAs/sameAs2/sameAsCatalog/ sameAsCatalog2.txt
hadoop fs -put sameAsCatalog2.txt URIs/
hadoop fs -getmerge sameAs/sameAs3/sameAsCatalog/ sameAsCatalog3.txt
hadoop fs -put sameAsCatalog3.txt URIs/
hadoop fs -getmerge sameAs/sameAs4/sameAsCatalog/ sameAsCatalog4.txt
hadoop fs -put sameAsCatalog4.txt URIs/
3. Entity (or Element Index) Creation
hadoop jar LODsyndesis.jar gr.forth.ics.isl.indexes.CreateElementIndex URIs/ elementIndex prefixIndexes/prefixIndex/prefixIndex.txt-r-00000 32
hadoop fs -getmerge elementIndex/Part2/ part2.txt
hadoop fs -put part2.txt elementIndex/Part1/
4. Lattice Measurements for Entities
hadoop jar LODsyndesis.jar gr.forth.ics.isl.latticeCreation.CreateDirectCounts elementIndex/Part1 directCounts 1
hadoop jar LODsyndesis.jar gr.forth.ics.isl.latticeCreation.CreateLattice directCounts lattice 32 100 15 2 5 0.05
For constructing all Indexes
Pre-Processing Steps: Download All Files from http://islcatalog.ics.forth.gr/dataset/lodsyndesis and upload them to HDFS.
hadoop fs -mkdir Triples/
Unrar all .rar files containing triples (6 different parts) and upload each file to HDFS hadoop fs -put <file> Triples/
hadoop fs -put entityEquivalenceCatalog.txt Triples/
hadoop fs -put propertyEquivalenceCatalog.txt
hadoop fs -put classEquivalenceCatalog.txt
1. Creation of Real World Triples
hadoop jar LODsyndesis.jar gr.forth.ics.isl.indexes.ReplaceSubjects Triples/ subjects 32 propertyEquivalenceCatalog.txt classEquivalenceCatalog.txt
hadoop fs -mv Triples/entityEquivalenceCatalog.txt subjects/object/
hadoop jar LODsyndesis.jar gr.forth.ics.isl.indexes.ReplaceObjects subjects/object objects/ 32
hadoop fs -mkdir realWorldTriples
hadoop fs -mv subjects/finished/* realWorldTriples/
hadoop fs -mv objects/* realWorldTriples/
2.Creation of Entity-Triples Index and Lattice Measurements for Triples
hadoop jar LODsyndesis.jar gr.forth.ics.isl.indexes.CreateEntityTriplesIndex realWorldTriples/ entityTriplesIndex 32 0 0
hadoop jar LODsyndesis.jar gr.forth.ics.isl.latticeCreation.CreateDirectCounts entityTriplesIndex/ dcTriples 1
hadoop jar LODsyndesis.jar gr.forth.ics.isl.latticeCreation.CreateLattice dcTriples latticeTriples 32 100 15 2 5 0.05
3. Creation of URI Indexes and Lattice Measurements for URIs
hadoop jar LODsyndesis.jar gr.forth.ics.isl.indexes.CreateEntityIndex realWorldTriples/ URI_Indexes 32 0
3a. Lattice for Entities
hadoop jar LODsyndesis.jar gr.forth.ics.isl.latticeCreation.CreateDirectCounts URI_Indexes/entities dcEntities 1
hadoop jar LODsyndesis.jar gr.forth.ics.isl.latticeCreation.CreateLattice dcEntities latticeEntities 32 100 15 2 5 0.05
3b. Lattice for Properties
hadoop jar LODsyndesis.jar gr.forth.ics.isl.latticeCreation.CreateDirectCounts URI_Indexes/properties dcProperties 1
hadoop jar LODsyndesis.jar gr.forth.ics.isl.latticeCreation.CreateLattice dcProperties latticeProperties 32 10 15 2 5 0.05
3c. Lattice for Classes
hadoop jar LODsyndesis.jar gr.forth.ics.isl.latticeCreation.CreateDirectCounts URI_Indexes/classes dcClasses 1
hadoop jar LODsyndesis.jar gr.forth.ics.isl.latticeCreation.CreateLattice dcClasses latticeClasses 32 10 15 2 5 0.05
4. Creation of Literals Index
hadoop jar LODsyndesis.jar gr.forth.ics.isl.indexes.CreateLiteralsIndex realWorldTriples/ literalsIndex 32 0
hadoop jar LODsyndesis.jar gr.forth.ics.isl.latticeCreation.CreateDirectCounts literalsIndex/ directCountsLiterals 1
hadoop jar LODsyndesis.jar gr.forth.ics.isl.latticeCreation.CreateLattice directCountsLiterals latticeLiterals 32 1000 10 2 5 0.05