Merge pull request uniovi-hepex#12 from vrbouza/master

Forward to upstream the master buena

README.md

@@ -18,6 +18,9 @@ Please never commit neither the build directory, nor the empty init.py files cre
 
     cd $CMSSW_BASE/src
     git clone https://github.com/cms-nanoAOD/nanoAOD-tools.git PhysicsTools/NanoAODTools
+    cd PhysicsTools/NanoAODTools
+    cmsenv
+    scram b
 
 ## General instructions to run the post-processing step
 
@@ -33,6 +36,7 @@ Here is a summary of its features:
 * the `-J`,`--json` option is used to pass the name of a JSON file that will be used to select events. It cannot be used in friend mode.
 * if run with the `--full` option (default), the output will be a full nanoAOD file. If run with the `--friend` option, instead, the output will be a friend tree that can be attached to the input tree. In the latter case, it is not possible to apply any kind of event selection, as the number of entries in the parent and friend tree must be the same.
 * the `-b`,`--branch-selection` option is used to pass the name of a file containing directives to keep or drop branches from the output tree. The file should contain one directive among `keep`/`drop` (wildcards allowed as in TTree::SetBranchStatus) or `keepmatch`/`dropmatch` (python regexp matching the branch name) per line, as shown in the [this](python/postprocessing/examples/keep_and_drop.txt) example file.
+  * `--bi` and `--bo` allows to specify the keep/drop file separately for input and output trees.  
 * the `--justcount` option will cause the script to printout the number of selected events, without actually writing the output file.
 
 Please run with `--help` for a complete list of options.
@@ -41,23 +45,29 @@ Please run with `--help` for a complete list of options.
 
 It is possible to import modules that will be run on each entry passing the event selection, and can be used to calculate new variables that will be included in the output tree (both in friend and full mode) or to apply event filter decisions.
 
-We will use `python/postprocessing/examples/mhtProducer.py` as an example.
-
-The module definition [file](python/postprocessing/examples/mhtProducer.py), containing the constructor:
-
-    mht = lambda : mhtProducer( lambda j : j.pt > 40, 
-                                lambda mu : mu.pt > 20 and mu.miniPFIso_all/mu.pt < 0.2,
-                                lambda el : el.pt > 20 and el.miniPFIso_all/el.pt < 0.2 ) 
-
+We will use `python/postprocessing/examples/exampleModule.py` as an example. The module definition [file](python/postprocessing/examples/exampleModule.py), containing a simple constructor
+```
+   exampleModuleConstr = lambda : exampleProducer(jetSelection= lambda j : j.pt > 30)
+```
 should be imported using the following syntax:
 
-`python scripts/nano_postproc.py -I PhysicsTools.NanoAODTools.postprocessing.examples.mhtProducer mht`
+```
+python scripts/nano_postproc.py outDir /eos/cms/store/user/andrey/f.root -I PhysicsTools.NanoAODTools.postprocessing.examples.exampleModule exampleModuleConstr
+```
 
-Let us now examine the structure of the `mhtProducer` module class. All modules must inherit from `PhysicsTools.NanoAODTools.postprocessing.framework.eventloop.Module`.
+Let us now examine the structure of the `exampleProducer` module class. All modules must inherit from `PhysicsTools.NanoAODTools.postprocessing.framework.eventloop.Module`.
 * the `__init__` constructor function should be used to set the module options.
 * the `beginFile` function should create the branches that you want to add to the output file, calling the `branch(branchname, typecode, lenVar)` method of `wrappedOutputTree`. `typecode` should be the ROOT TBranch type ("F" for float, "I" for int etc.). `lenVar` should be the name of the variable holding the length of array branches (for instance, `branch("Electron_myNewVar","F","nElectron")`). If the `lenVar` branch does not exist already - it can happen if you create a new collection, see an example [here](python/postprocessing/examples/collectionMerger.py)) - it will be automatically created.
 * the `analyze` function is called on each event. It should return `True` if the event is to be retained, `False` if it should be dropped.
 
+### Keep/drop branches
+See the effect of keep/drop instructions by running:
+```
+python scripts/nano_postproc.py outDir /eos/cms/store/user/andrey/f.root -I PhysicsTools.NanoAODTools.postprocessing.examples.exampleModule exampleModuleConstr -s _exaModu_keepdrop --bi scripts/keep_and_drop_input.txt --bo scripts/keep_and_drop_output.txt
+```
+comparing to the previous command (without `--bi` and `--bo`).
+The output branch created by _exampleModuleConstr_ produces the same result in both cases. But this one drops all other branches when creating output tree. It also runs faster.
+
 The event interface, defined in `PhysicsTools.NanoAODTools.postprocessing.framework.datamodule`, allows to dynamically construct views of objects organized in collections, based on the branch names, for instance:
 
     electrons = Collection(event, "Electron")
@@ -67,3 +77,14 @@ The event interface, defined in `PhysicsTools.NanoAODTools.postprocessing.framew
 and this will access the elements of the `Electron_someVar`, `Electron_pt` branch arrays. Event variables can be accessed simply by `event.someVar`, for instance `event.rho`.
 
 The output branches should be filled calling the `fillBranch(branchname, value)` method of `wrappedOutputTree`. `value` should be the desired value for single-value branches, an iterable with the correct length for array branches. It is not necessary to fill the `lenVar` branch explicitly, as this is done automatically using the length of the passed iterable.
+
+
+### mht producer
+Now, let's have a look at another example, `python/postprocessing/examples/mhtjuProducerCpp.py`, [file](python/postprocessing/examples/mhtjuProducerCpp.py). Similarly, it should be imported using the following syntax:
+
+```
+python scripts/nano_postproc.py outDir /eos/cms/store/user/andrey/f.root -I PhysicsTools.NanoAODTools.postprocessing.examples.mhtjuProducerCpp mhtju
+```
+This module has the same structure of its producer as `exampleProducer`, but in addition it utilizes a C++ code to calculate the mht variable, `src/mhtjuProducerCppWorker.cc`. This code is loaded in the `__init__` method of the producer.
+
+

crab/PSet.py

@@ -1,15 +1,17 @@
-#this fake PSET is needed for local test and for crab to figure the output filename
-#you do not need to edit it unless you want to do a local test using a different input file than
-#the one marked below
+# this fake PSET is needed for local test and for crab to figure the output
+# filename you do not need to edit it unless you want to do a local test using
+# a different input file than the one marked below
 import FWCore.ParameterSet.Config as cms
 process = cms.Process('NANO')
-process.source = cms.Source("PoolSource", fileNames = cms.untracked.vstring(),
-#	lumisToProcess=cms.untracked.VLuminosityBlockRange("254231:1-254231:24")
+process.source = cms.Source(
+    "PoolSource",
+    fileNames=cms.untracked.vstring(),
+    # lumisToProcess=cms.untracked.VLuminosityBlockRange("254231:1-254231:24")
 )
 process.source.fileNames = [
-	'../../NanoAOD/test/lzma.root' ##you can change only this line
+    '../../NanoAOD/test/lzma.root'  # you can change only this line
 ]
-process.maxEvents = cms.untracked.PSet(input = cms.untracked.int32(10))
-process.output = cms.OutputModule("PoolOutputModule", fileName = cms.untracked.string('tree.root'))
+process.maxEvents = cms.untracked.PSet(input=cms.untracked.int32(10))
+process.output = cms.OutputModule("PoolOutputModule",
+                                  fileName=cms.untracked.string('tree.root'))
 process.out = cms.EndPath(process.output)
-

crab/crab_cfg.py

@@ -1,26 +1,34 @@
 from WMCore.Configuration import Configuration
+from CRABClient.UserUtilities import config, getUsernameFromSiteDB
+
 config = Configuration()
 
 config.section_("General")
 config.General.requestName = 'NanoPost1'
-config.General.transferLogs=True
+config.General.transferLogs = True
 config.section_("JobType")
 config.JobType.pluginName = 'Analysis'
 config.JobType.psetName = 'PSet.py'
 config.JobType.scriptExe = 'crab_script.sh'
-config.JobType.inputFiles = ['crab_script.py','../scripts/haddnano.py'] #hadd nano will not be needed once nano tools are in cmssw
-config.JobType.sendPythonFolder	 = True
+# hadd nano will not be needed once nano tools are in cmssw
+config.JobType.inputFiles = ['crab_script.py', '../scripts/haddnano.py']
+config.JobType.sendPythonFolder = True
 config.section_("Data")
-config.Data.inputDataset = '/TTJets_DiLept_TuneCUETP8M1_13TeV-madgraphMLM-pythia8/RunIIFall17NanoAOD-PUMoriond17_94X_mcRun2_asymptotic_v2-v1/NANOAODSIM'
+config.Data.inputDataset = '/DYJetsToLL_1J_TuneCP5_13TeV-amcatnloFXFX-pythia8/RunIIFall17NanoAOD-PU2017_12Apr2018_94X_mc2017_realistic_v14-v1/NANOAODSIM'
+#config.Data.inputDBS = 'phys03'
 config.Data.inputDBS = 'global'
-#config.Data.splitting = 'FileBased'
-config.Data.splitting = 'EventAwareLumiBased'
-config.Data.unitsPerJob = 100
-config.Data.totalUnits = 2000
-config.Data.inputDBS='phys03'
-config.Data.outLFNDirBase = '/store/user/arizzi/NanoPost/'
-config.Data.publication = True
+config.Data.splitting = 'FileBased'
+#config.Data.splitting = 'EventAwareLumiBased'
+config.Data.unitsPerJob = 2
+config.Data.totalUnits = 10
+
+config.Data.outLFNDirBase = '/store/user/%s/NanoPost' % (
+    getUsernameFromSiteDB())
+config.Data.publication = False
 config.Data.outputDatasetTag = 'NanoTestPost'
 config.section_("Site")
-config.Site.storageSite = "T2_IT_Bari"
+config.Site.storageSite = "T2_DE_DESY"
 
+#config.Site.storageSite = "T2_CH_CERN"
+# config.section_("User")
+#config.User.voGroup = 'dcms'

crab/crab_script.py

@@ -1,14 +1,18 @@
 #!/usr/bin/env python
 import os
-from PhysicsTools.NanoAODTools.postprocessing.framework.postprocessor import * 
+from PhysicsTools.NanoAODTools.postprocessing.framework.postprocessor import *
 
-#this takes care of converting the input files from CRAB
-from PhysicsTools.NanoAODTools.postprocessing.framework.crabhelper import inputFiles,runsAndLumis
+# this takes care of converting the input files from CRAB
+from PhysicsTools.NanoAODTools.postprocessing.framework.crabhelper import inputFiles, runsAndLumis
 
-from  PhysicsTools.NanoAODTools.postprocessing.examples.exampleModule import *
-p=PostProcessor(".",inputFiles(),"Jet_pt>200",modules=[exampleModule()],provenance=True,fwkJobReport=True,jsonInput=runsAndLumis())
+from PhysicsTools.NanoAODTools.postprocessing.examples.exampleModule import *
+p = PostProcessor(".",
+                  inputFiles(),
+                  "Jet_pt>200",
+                  modules=[exampleModuleConstr()],
+                  provenance=True,
+                  fwkJobReport=True,
+                  jsonInput=runsAndLumis())
 p.run()
 
-print "DONE"
-os.system("ls -lR")
-
+print("DONE")

crab/crab_script.sh

@@ -5,7 +5,6 @@ if [ "`tty`" != "not a tty" ]; then
   echo "YOU SHOULD NOT RUN THIS IN INTERACTIVE, IT DELETES YOUR LOCAL FILES"
 else
 
-ls -lR .
 echo "ENV..................................."
 env 
 echo "VOMS"