{"url": "http://mondego.ics.uci.edu/projects/clonedetection/", "content": "\n\n\nCode Clone Detection \n\n\n\n\n\t\t\n\t\t\n\t\t\t\n\t\t\t\t\t\t\n\n\t\t\t\n\t\t\t\t\n \n \n\t\t
\n\t\t\t

Scaling Token-Based Code Clone Detection

\n\n\t\t\t\n\t\t\t\tThe Team
\n\t\t\t\tHitesh Sajnani, Vaibhav Saini, Cristina Lopes
\n\t\t\t\n\t\t
\n\t\t\n\t\t\t\n\t\t\t\t\n\n\t\n\n\n\n

Abstract

\nWe propose a new token-based approach for large scale code clone detection which is based on a filtering heuristic that reduces the number of token comparisons when the two code blocks are compared. We also present a MapReduce based parallel algorithm that uses the filtering heuristic and scales to thousands of projects. The filtering heuristic is generic and can also be used in conjunction with other token-based approaches. In that context, we demonstrate how it can increase the retrieval speed and decrease the memory usage of the index-based approaches.\n
\nIn our experiments on 36 open source Java projects, we found that: (i) filtering reduces token comparisons by a factor of 10, and thus increasing\nthe speed of clone detection by a factor of 1.5; (ii) the speed-up and scale-up of the parallel approach using filtering is near-linear on a cluster of\n2-32 nodes for 150-2800 projects; and (iii) filtering decreases the memory usage of index-based approach by half and the search time by a factor of 5.\u00e2\u20ac\u2039
\t\t\t\t\n

\n\t\t

Replicating the Experiment

\n\t\t

System Requirements

\n\t\tYou need a machine with at least 12GB ram running ubuntu or mac-osx, with Java-1.6 or higher installed. Please note that you need system with higher ram because the experiment involves very large subject systems. However, if you are running this on smaller systems (see Tools below), there is no such constraint.\n\t\t

Steps to replicate the experiment:

\n\t\t
    \n\t\t
  1. \n\t\tClick here to download the distribution (dist.zip). \n\t\t
    2. \n\n\t\t
  2. \n unzip dist.zip. Navigate to dist/ using terminal\n
    3. \n\n\t\t
  3. \n execute the command ./runexp.sh 1 \n\t\t    Please Note: this will run the experiment 1 time. To run it n times, change the command to ./runexp.sh n\n
    4. \n\t\t
\n\tYou can see the generated output in the ./output folder. Files with extension .txt will have computed clones and the files with .csv extension will have the performance analysis result. \n
\n
\t\t\n

Tools

\nIn order to run the tool on any arbitrary project, please follow the steps below:\n

\nA. Generating the input file of the project for which you want to detect clones\n
    \n
  1. Click here to download input generator for the code clone detector (ast.zip).
    2. \n
  2. Unzip ast.zip and import the project ast in your eclipse workspace.
    3. \n
  3. Run it as an \"Eclipse Application\". This should open another eclipse instance where you will import the projects for which you want to generate the input file.
    4. \n
  4. After importing the project in the workspace of the new eclipse instance, click on the \"Sample Menu\" in the top menu bar and then click on the \"Sample command\" to run. This should generate the output (desired input file) in the path specified by variable \"outputdirPath\".
    5. \n
  5. Please note that you will have to change the location of output directory on line 61 of SampleHandler.java.this.outputdirPath = \"/Users/vaibhavsaini/Documents/codetime/repo/ast/output/\"; to your desired output directory.
    6. \n
  6. The generated input file name will be of the format: <ProjectName>-clone-INPUT.txt. For example, if your project name is jython, then the generated input file name should be jython-clone-INPUT.txt
    7. \n
\n
\nB. Running the clone detection tool on the generated input file\n
    \n
  1. Click here to download the CloneDetector (tool.zip).
    2. \n
  2. Unzip tool.zip and navigate to tool/ using terminal
    3. \n
  3. Copy the input file generated above (<ProjectName>-clone-INPUT.txt) into input/dataset directory.
    4. \n
  4. Open cd.sh, and assign <ProjectName> as value to the variable arrayname (line #5). For example, If your generated input file is jython-clone-INPUT.txt, line #5 should be arrayname=(jython)
    5. \n
  5. Execute the command ./cd.sh
    6. \n
\n
\nC. Generated output\n
    \n
  1. The generated output will be in the ./output folder.
    2. \n
  2. Files with extension .txt will have the computed clones and the files with .csv extension will have the time taken to detect clones
    3. \n
\n\t\t

Data

\nThe table below describes all the subject systems, and the corresponding output. This data was used to calculate and report numbers in the paper. \nColumn 1 has links to the source code of each subject system. Column 2 is the input file that is generated by running INPUTGEN tool on the subject systems. \nThis input file is in turn used by NCCF and FCCD to compute clones. Column 4 has links to the computed clones. Since both the tools produced exactly the same output, you will see only one file per subject system. Column 4 has the final analysis results - runtime for tool, and total token comparisons done. These numbers are different for both the tools, so you will see two analysis files per project - one produced by NCCF, and another by FCCD. \n\n\t\n\t\t\n\t\n\t\t\n\t\t\n\t\t\n\t\t\n\t\n\t\t\n\t\t\n\t\t\n\t\t\n\n\t\n\t\n\t\t\n\t\t\n\t\t\n\t\t\n\t\n\t\n\t\t\n\t\t\n\t\t\n\t\t\n\t\n\t\n\t\t\n\t\t\n\t\t\n\t\t\n\t\n
Subject System\n\t\tGenerated Input for Tool\n\t\tClones Detected\n\t\tAnalysis\n\t
Ant-1.9.2ant.zipant-clones.zipant-analysis.zip
Cocoon-2.2.0cocoon.zipcocoon-clones.zipcocoon-analysis.zip
Maven-3.0.5maven.zipmaven-clones.zipmaven-analysis.zip
Lucene-4.4.0lucene.ziplucene-clones.ziplucene-analysis.zip
Hadoop-Rev:1531458hadoop.ziphadoop-clones.ziphadoop-analysis.zip
\t\n\t\t\n
\n\n", "encoding": "Windows-1252"}