Advertisement
Web Developer Resources
You Are Here: ESQSoft.com ¬ tools ¬ List Deduper:: Online tool to remove duplicates from a list
E.S.Q. Software Logo

List Deduper:: Online Tool for Removing Duplicates and Sorting Lists

This tool uses an HTML form and JavaScript to remove duplicates from your list, and sort it alphanumerically.

Brief instructions for using this tool:

  1. Grab your list into the buffer.
  2. Paste it into the "source" field below.
  3. Optional: choose what you want to delimit on (default is line feed).
  4. Press the "dedupe" button.
  5. Grab the deduped, sorted list from the "target" field below.
  6. Go about your business! No fuss, no muss.

Some helpful shortcuts to keep in mind if you deal with list manipulation regularly.

  • CTRL-A selects everything in the active field.
  • CTRL-C (copy), CTRL-X (cut), CTRL-V (paste)
  • Underlined letters in the field labels imply a shortcut access key. For example, pressing ALT-S (or CRTL-ALT-S in FireFox), will cause the cursor focus to jump to the Source field. Look for the other underlined letters to find other helpful keyboard shortcuts.
  • There are some test fields below. If you suspect an issue, you can sanity check the tool using a test list. (Lets face it, sometimes things become broken overtime--if you are seeing an issue, let me know and I'll try to get it fixed asap.)

List Deduper Tool:

source: paste your list here
target: deduped, sorted list output will appear here
Delimiter: (space) (comma) (line feed \n) (preface items)
example list (1): no dupes, unsorted
example list (2): dupes
example list (3): no dupes, unsorted (space delimited)
example list (4): dupes (space delimited)

Here is the source code from the tool to illustrate an example of how this can be done.

<script type="text/javascript"> function hError(msg){ var obj = document.getElementById('error') if(msg.length>0){ obj.innerHTML = msg; obj.style.display = 'block'; } else{ obj.innerHTML = ''; obj.style.display = 'none'; } } function dedupeIt(obj){ var arr = []; var html = ''; var fobj = obj.form var sobj = fobj.oSrc var tobj = fobj.oTgt /* find the delimiter */ var d = ''; for(var i=0; i<fobj.delimiter.length; i++){ if(fobj.delimiter[i].checked==true){ d = fobj.delimiter[i].value; i = fobj.delimiter.length; } } arr = eval('sobj.value.split(/'+d+'/);'); // array-ize the textarea data arr = arr.sort(); for(var i=0; i<arr.length; i++){ if(arr[i].match(/\n/)){ arr[i] = arr[i].replace(/\n/,''); // remove line feeds that made it into the array } for(var j=i; j<arr.length; j++){ if((i != j) && (arr[i] == arr[j])){ arr[j] = ''; // unset dupes } } } for(var i=0; i<arr.length; i++){ //alert('['+arr[i] +'] -- '+arr[i].length); if(arr[i].length<1) arr.splice(i,1); } // remove empties -- from the compare, and residue from the HTML textarea form input if( arr.length<2){ hError('The length of the array is suspicious. You might have selected the wrong delimiter.'); } else{ hError(''); // reset } var char = (fobj.prefaceItems.checked?'-':'') for(var i=0; i<arr.length; i++){ html += (i>0?'\n'+char:char) + arr[i] } tobj.value = html; return false } function hClick1(obj){ // scroll to the form var a = document.getElementById('myform'); window.scrollTo(0,document.getElementById('myform').offsetTop); obj.select(); } </script>
Advertisement
Advertisement
Statcounter