I've got a speed question. I have a bash script which parses information from TheTvDb.com. It downloads nearly 40,000 lines of data, then reduces it down to about 5000 lines of data which gets written to the hard disk. Then it reads the file and parses it into several files which are used later as a lookup table. It's basically taking all the information it sees before each "/Episode" and writing it to a specific file, then resetting for the next one.
It has to synchronize on the "/Episode" tag because there is a "FirstAired" tag outside of the episode tags. This ensures that the data is drawn in sequence rather then depending on each individual tag to be relating to a episode.
here is the code in question.
if [ -f "$mythicalLibrarian/$NewShowName/$NewShowName.xml" ]; then
Ename=""
actualEname=""
FAired=""
SeasonNr=""
EpisodeNr=""
recordNumber=0
echo "Parsing Downloaded information: $NewShowName.xml "
while read line
do
if [[ $line == \<\/Episode\> ]]; then
(( ++recordNumber ))
echo -ne "Building Record:$recordNumber ${actualEname:0:20} \r" 1>&2
echo "$actualEname" >> "$mythicalLibrarian/$NewShowName/$NewShowName.actualEname.txt"&
Ename=`echo "$actualEname" |sed 's/;.*//'`
echo "$Ename" >> "$mythicalLibrarian/$NewShowName/$NewShowName.Ename.txt"&
echo "$FAired" >> "$mythicalLibrarian/$NewShowName/$NewShowName.FAired.txt"&
echo "$SeasonNr" >> "$mythicalLibrarian/$NewShowName/$NewShowName.S.txt"&
echo "$EpisodeNr" >> "$mythicalLibrarian/$NewShowName/$NewShowName.E.txt"&
Ename=""
actualEname=""
FAired=""
SeasonNr=""
EpisodeNr=""
#Get actual show name
elif [[ $line == \<EpisodeName\>* ]]; then
actualEname=`echo "$line" | sed -e s/'<\/EpisodeName>'// -e s/'<EpisodeName>'// -e s/'\&\;'/'\&'/ -e s/'\"\;'/'\"'/ -e s/'\&\;'/'\&'/ -e s/'\&ndash\;'/'-'/ -e s/'\<\;'/'\<'/ -e 's/'\>\;'/'\>'/' |tr -d '|\?\*\<\"\:\>\+\\\[\]\/'`
#Get OriginalAirDate
elif [[ $line == \<FirstAired\>* ]]; then
FAired=`echo "$line" | sed -e s/'<FirstAired>'//g -e s/'<\/FirstAired>'//g`
#Get Season number
elif [[ $line == \<SeasonNumber\>* ]]; then
SeasonNr=`echo "$line" |sed -e s/'<SeasonNumber>'// -e s/'<\/SeasonNumber>'//`
#Get Episode number
elif [[ $line == \<EpisodeNumber\>* ]]; then
EpisodeNr=`echo "$line" |sed -e 's/<EpisodeNumber>//' -e 's/<\/EpisodeNumber>//'`
fi
done < "$mythicalLibrarian/$NewShowName/$NewShowName.xml"
chmod 777 "$mythicalLibrarian"/"$NewShowName"/"$NewShowName".actualEname.txt
chmod 666 "$mythicalLibrarian"/"$NewShowName"/"$NewShowName".Ename.txt
chmod 666 "$mythicalLibrarian/$NewShowName/$NewShowName".FAired.txt
chmod 666 "$mythicalLibrarian"/"$NewShowName"/"$NewShowName".S.txt
chmod 666 "$mythicalLibrarian/$NewShowName/$NewShowName".E.txt
GotNewInformation=1
elif [ ! -f "$mythicalLibrarian/$NewShowName/$NewShowName.xml" ]; then
echo "COULD NOT DOWNLOAD:www.thetvdb.com/api/$APIkey/series/$SeriesID/all/$Language.xml">>"$mythicalLibrarian"/output.log
fi
Here is some of the data it is processing
<?xml version="1.0" encoding="UTF-8" ?>
<Data><Series>
<Actors>|Fred Rogers|Adair Roth|Bert Lloyd|Bud Alder|Carol Saunders|Carole Switala|Deborah Neal Stampo|Don Brockett|Elsie Neal|Emilie Jacobson|Fred Michael|John Reardon|Jos|Judy Rubin|Keith David|Lenny Meledandri|Michael Horton|Robert Trow|Yoshi Ito|</Actors>
<Airs_DayOfWeek></Airs_DayOfWeek>
<Airs_Time></Airs_Time>
<ContentRating></ContentRating>
<FirstAired>1968-02-01</FirstAired>
<Genre>|Children|</Genre>
<Network>PBS</Network>
<NetworkID></NetworkID>
<Overview>"In a little toy neighborhood, a tiny trolley rolls past a house at the end of a street.
<Runtime>30</Runtime>
<SeriesID>6843</SeriesID>
<SeriesName>Mister Rogers' Neighborhood</SeriesName>
<Status>Ended</Status>
<added></added>
<addedBy></addedBy>
<banner>graphical/77750-g.jpg</banner>
<fanart>fanart/original/77750-1.jpg</fanart>
<poster></poster>
<zap2it_id>SH002930</zap2it_id>
</Series>
<Episode>
<EpisodeName>Change (1)</EpisodeName>
<EpisodeNumber>1</EpisodeNumber>
<FirstAired>1968-02-19</FirstAired>
<SeasonNumber>1</SeasonNumber>
</Episode>
<Episode>
<EpisodeName>Change (2)</EpisodeName>
<EpisodeNumber>2</EpisodeNumber>
<FirstAired>1968-02-20</FirstAired>
<SeasonNumber>1</SeasonNumber>
</Episode>
<Episode>
<EpisodeName>Change (3)</EpisodeName>
<EpisodeNumber>3</EpisodeNumber>
<FirstAired>1968-02-21</FirstAired>
<SeasonNumber>1</SeasonNumber>
</Episode>
<Episode>
<EpisodeName>Change (4)</EpisodeName>
<EpisodeNumber>4</EpisodeNumber>
<FirstAired>1968-02-22</FirstAired>
<SeasonNumber>1</SeasonNumber>
</Episode>
<Episode>
<EpisodeName>Change (5)</EpisodeName>
<EpisodeNumber>5</EpisodeNumber>
<FirstAired>1968-02-23</FirstAired>
<SeasonNumber>1</SeasonNumber>
</Episode>
<Episode>
<EpisodeName>Show 6</EpisodeName>
<EpisodeNumber>6</EpisodeNumber>
<FirstAired>1968-02-26</FirstAired>
<SeasonNumber>1</SeasonNumber>
</Episode>
<Episode>
<EpisodeName>Show 7</EpisodeName>
<EpisodeNumber>7</EpisodeNumber>
<FirstAired>1968-02-27</FirstAired>
<SeasonNumber>1</SeasonNumber>
</Episode>
<Episode>
<EpisodeName>Show 8</EpisodeName>
<EpisodeNumber>8</EpisodeNumber>
<FirstAired>1968-02-28</FirstAired>
<SeasonNumber>1</SeasonNumber>
</Episode>
<Episode>
<EpisodeName>Show 9</EpisodeName>
<EpisodeNumber>9</EpisodeNumber>
<FirstAired>1968-02-29</FirstAired>
<SeasonNumber>1</SeasonNumber>
</Episode>
<Episode>
<EpisodeName>Show 10</EpisodeName>
<EpisodeNumber>10</EpisodeNumber>
<FirstAired>1968-03-01</FirstAired>
<SeasonNumber>1</SeasonNumber>
</Episode>
<Episode>
<EpisodeName>Show 11</EpisodeName>
<EpisodeNumber>11</EpisodeNumber>
<FirstAired>1968-03-04</FirstAired>
<SeasonNumber>1</SeasonNumber>
</Episode>
<Episode>
<EpisodeName>Show 12</EpisodeName>
<EpisodeNumber>12</EpisodeNumber>
<FirstAired>1968-03-05</FirstAired>
<SeasonNumber>1</SeasonNumber>
</Episode>
<Episode>
<EpisodeName>Show 13</EpisodeName>
<EpisodeNumber>13</EpisodeNumber>
<FirstAired>1968-03-06</FirstAired>
<SeasonNumber>1</SeasonNumber>
</Episode>
<Episode>
<EpisodeName>Show 14</EpisodeName>
<EpisodeNumber>14</EpisodeNumber>
<FirstAired>1968-03-07</FirstAired>
<SeasonNumber>1</SeasonNumber>
</Episode>
<Episode>
<EpisodeName>Show 15</EpisodeName>
<EpisodeNumber>15</EpisodeNumber>
<FirstAired>1968-03-08</FirstAired>
<SeasonNumber>1</SeasonNumber>
</Episode>
<Episode>
<EpisodeName>Welcome Donkey Hodie (1)</EpisodeName>
<EpisodeNumber>16</EpisodeNumber>
<FirstAired>1968-03-11</FirstAired>
<SeasonNumber>1</SeasonNumber>
</Episode>
<Episode>
<EpisodeName>Welcome Donkey Hodie (2)</EpisodeName>
<EpisodeNumber>17</EpisodeNumber>
<FirstAired>1968-03-12</FirstAired>
<SeasonNumber>1</SeasonNumber>
</Episode>
<Episode>
<EpisodeName>Welcome Donkey Hodie (3)</EpisodeName>
<EpisodeNumber>18</EpisodeNumber>
<FirstAired>1968-03-13</FirstAired>
<SeasonNumber>1</SeasonNumber>
</Episode>
<Episode>
<EpisodeName>Welcome Donkey Hodie (4)</EpisodeName>
<EpisodeNumber>19</EpisodeNumber>
<FirstAired>1968-03-14</FirstAired>
<SeasonNumber>1</SeasonNumber>
</Episode>
<Episode>
<EpisodeName>Welcome Donkey Hodie (5)</EpisodeName>
<EpisodeNumber>20</EpisodeNumber>
<FirstAired>1968-03-15</FirstAired>
<SeasonNumber>1</SeasonNumber>
</Episode>
<Episode>
<EpisodeName>Show 21</EpisodeName>
<EpisodeNumber>21</EpisodeNumber>
<FirstAired>1968-03-18</FirstAired>
<SeasonNumber>1</SeasonNumber>
</Episode>
<Episode>
<EpisodeName>Show 22</EpisodeName>
<EpisodeNumber>22</EpisodeNumber>
<FirstAired>1968-03-19</FirstAired>
<SeasonNumber>1</SeasonNumber>
</Episode>
<Episode>
<EpisodeName>Show 23</EpisodeName>
<EpisodeNumber>23</EpisodeNumber>
<FirstAired>1968-03-20</FirstAired>
<SeasonNumber>1</SeasonNumber>
</Episode>
<Episode>
<EpisodeName>Show 24</EpisodeName>
<EpisodeNumber>24</EpisodeNumber>
<FirstAired>1968-03-21</FirstAired>
<SeasonNumber>1</SeasonNumber>
</Episode>
<Episode>
<EpisodeName>Show 25</EpisodeName>
<EpisodeNumber>25</EpisodeNumber>
<FirstAired>1968-03-22</FirstAired>
<SeasonNumber>1</SeasonNumber>
</Episode>
<Episode>
<EpisodeName>Show 26</EpisodeName>
<EpisodeNumber>26</EpisodeNumber>
<FirstAired>1968-03-25</FirstAired>
<SeasonNumber>1</SeasonNumber>
</Episode>
<Episode>
<EpisodeName>Show 27</EpisodeName>
<EpisodeNumber>27</EpisodeNumber>
<FirstAired>1968-03-26</FirstAired>
<SeasonNumber>1</SeasonNumber>
</Episode>
<Episode>
<EpisodeName>Show 28</EpisodeName>
<EpisodeNumber>28</EpisodeNumber>
<FirstAired>1968-03-27</FirstAired>
<SeasonNumber>1</SeasonNumber>
</Episode>
<Episode>
<EpisodeName>Show 29</EpisodeName>
<EpisodeNumber>29</EpisodeNumber>
<FirstAired>1968-03-28</FirstAired>
<SeasonNumber>1</SeasonNumber>
</Episode>
<Episode>
<EpisodeName>Show 30</EpisodeName>
<EpisodeNumber>30</EpisodeNumber>
<FirstAired>1968-03-29</FirstAired>
<SeasonNumber>1</SeasonNumber>
</Episode>
<Episode>
<EpisodeName>Red Monster (1)</EpisodeName>
<EpisodeNumber>31</EpisodeNumber>
<FirstAired>1968-04-01</FirstAired>
<SeasonNumber>1</SeasonNumber>
</Episode>
<Episode>
<EpisodeName>Red Monster (2)</EpisodeName>
<EpisodeNumber>32</EpisodeNumber>
<FirstAired>1968-04-02</FirstAired>
<SeasonNumber>1</SeasonNumber>
</Episode>
<Episode>
<EpisodeName>Red Monster (3)</EpisodeName>
<EpisodeNumber>33</EpisodeNumber>
<FirstAired>1968-04-03</FirstAired>
<SeasonNumber>1</SeasonNumber>
</Episode>
<Episode>
<EpisodeName>Red Monster (4)</EpisodeName>
<EpisodeNumber>34</EpisodeNumber>
<FirstAired>1968-04-04</FirstAired>
<SeasonNumber>1</SeasonNumber>
</Episode>
<Episode>
<EpisodeName>Red Monster (5)</EpisodeName>
<EpisodeNumber>35</EpisodeNumber>
<FirstAired>1968-04-05</FirstAired>
<SeasonNumber>1</SeasonNumber>
</Episode>
<Episode>
<EpisodeName>Show 36</EpisodeName>
<EpisodeNumber>36</EpisodeNumber>
<FirstAired>1968-04-08</FirstAired>
<SeasonNumber>1</SeasonNumber>
</Episode>
<Episode>
<EpisodeName>Show 37</EpisodeName>
<EpisodeNumber>37</EpisodeNumber>
<FirstAired>1968-04-09</FirstAired>
<SeasonNumber>1</SeasonNumber>
</Episode>
<Episode>
<EpisodeName>Show 38</EpisodeName>
<EpisodeNumber>38</EpisodeNumber>
<FirstAired>1968-04-10</FirstAired>
<SeasonNumber>1</SeasonNumber>
</Episode>
<Episode>
<EpisodeName>Show 39</EpisodeName>
<EpisodeNumber>39</EpisodeNumber>
<FirstAired>1968-04-11</FirstAired>
<SeasonNumber>1</SeasonNumber>
</Episode>
<Episode>
<EpisodeName>Show 40</EpisodeName>
<EpisodeNumber>40</EpisodeNumber>
<FirstAired>1968-04-12</FirstAired>
<SeasonNumber>1</SeasonNumber>
</Episode>
<Episode>
<EpisodeName>Show 41</EpisodeName>
<EpisodeNumber>41</EpisodeNumber>
<FirstAired>1968-04-15</FirstAired>
<SeasonNumber>1</SeasonNumber>
</Episode>
<Episode>
<EpisodeName>Show 42</EpisodeName>
<EpisodeNumber>42</EpisodeNumber>
<FirstAired>1968-04-16</FirstAired>
<SeasonNumber>1</SeasonNumber>
</Episode>
<Episode>
<EpisodeName>Show 43</EpisodeName>
<EpisodeNumber>43</EpisodeNumber>
<FirstAired>1968-04-17</FirstAired>
<SeasonNumber>1</SeasonNumber>
</Episode>
<Episode>
<EpisodeName>Show 44</EpisodeName>
<EpisodeNumber>44</EpisodeNumber>
<FirstAired>1968-04-18</FirstAired>
<SeasonNumber>1</SeasonNumber>
</Episode>
<Episode>
<EpisodeName>Show 45</EpisodeName>
<EpisodeNumber>45</EpisodeNumber>
<FirstAired>1968-04-19</FirstAired>
<SeasonNumber>1</SeasonNumber>
</Episode>
<Episode>
<EpisodeName>Show 46</EpisodeName>
<EpisodeNumber>46</EpisodeNumber>
<FirstAired>1968-04-22</FirstAired>
<SeasonNumber>1</SeasonNumber>
</Episode>
<Episode>
<EpisodeName>Show 47</EpisodeName>
<EpisodeNumber>47</EpisodeNumber>
<FirstAired>1968-04-23</FirstAired>
<SeasonNumber>1</SeasonNumber>
</Episode>
<Episode>
<EpisodeName>Show 48</EpisodeName>
<EpisodeNumber>48</EpisodeNumber>
<FirstAired>1968-04-24</FirstAired>
<SeasonNumber>1</SeasonNumber>
</Episode>
<Episode>
<EpisodeName>Show 49</EpisodeName>
<EpisodeNumber>49</EpisodeNumber>
<FirstAired>1968-04-25</FirstAired>
<SeasonNumber>1</SeasonNumber>
</Episode>
<Episode>
<EpisodeName>Show 50</EpisodeName>
<EpisodeNumber>50</EpisodeNumber>
<FirstAired>1968-04-26</FirstAired>
<SeasonNumber>1</SeasonNumber>
</Episode>
<Episode>
<EpisodeName>Show 51</EpisodeName>
<EpisodeNumber>51</EpisodeNumber>
<FirstAired>1968-04-29</FirstAired>
<SeasonNumber>1</SeasonNumber>
</Episode>
<Episode>
<EpisodeName>Show 52</EpisodeName>
<EpisodeNumber>52</EpisodeNumber>
<FirstAired>1968-04-30</FirstAired>
<SeasonNumber>1</SeasonNumber>
</Episode>
<Episode>
<EpisodeName>Show 53</EpisodeName>
<EpisodeNumber>53</EpisodeNumber>
<FirstAired>1968-05-01</FirstAired>
<SeasonNumber>1</SeasonNumber>
</Episode>
<Episode>
<EpisodeName>Show 54</EpisodeName>
<EpisodeNumber>54</EpisodeNumber>
<FirstAired>1968-05-02</FirstAired>
<SeasonNumber>1</SeasonNumber>
</Episode>
<Episode>
<EpisodeName>Show 55</EpisodeName>
<EpisodeNumber>55</EpisodeNumber>
<FirstAired>1968-05-03</FirstAired>
<SeasonNumber>1</SeasonNumber>
</Episode>
<Episode>
<EpisodeName>Show 56</EpisodeName>
<EpisodeNumber>56</EpisodeNumber>
<FirstAired>1968-05-06</FirstAired>
<SeasonNumber>1</SeasonNumber>
</Episode>
<Episode>
<EpisodeName>Show 57</EpisodeName>
<EpisodeNumber>57</EpisodeNumber>
<FirstAired>1968-05-07</FirstAired>
<SeasonNumber>1</SeasonNumber>
</Episode>
<Episode>
<EpisodeName>Show 58</EpisodeName>
<EpisodeNumber>58</EpisodeNumber>
<FirstAired>1968-05-08</FirstAired>
<SeasonNumber>1</SeasonNumber>
</Episode>
<Episode>
<EpisodeName>Show 59</EpisodeName>
<EpisodeNumber>59</EpisodeNumber>
<FirstAired>1968-05-09</FirstAired>
<SeasonNumber>1</SeasonNumber>
</Episode>
<Episode>
<EpisodeName>Show 60</EpisodeName>
<EpisodeNumber>60</EpisodeNumber>
<FirstAired>1968-05-10</FirstAired>
<SeasonNumber>1</SeasonNumber>
</Episode>
<Episode>
<EpisodeName>Show 61</EpisodeName>
<EpisodeNumber>61</EpisodeNumber>
<FirstAired>1968-05-13</FirstAired>
<SeasonNumber>1</SeasonNumber>
</Episode>
<Episode>
<EpisodeName>Show 62</EpisodeName>
<EpisodeNumber>62</EpisodeNumber>
<FirstAired>1968-05-14</FirstAired>
<SeasonNumber>1</SeasonNumber>
</Episode>
<Episode>
<EpisodeName>Show 63</EpisodeName>
<EpisodeNumber>63</EpisodeNumber>
<FirstAired>1968-05-15</FirstAired>
<SeasonNumber>1</SeasonNumber>
</Episode>
<Episode>
<EpisodeName>Show 64</EpisodeName>
<EpisodeNumber>64</EpisodeNumber>
<FirstAired>1968-05-16</FirstAired>
<SeasonNumber>1</SeasonNumber>
</Episode>
<Episode>
<EpisodeName>Show 65</EpisodeName>
<EpisodeNumber>65</EpisodeNumber>
<FirstAired>1968-05-17</FirstAired>
<SeasonNumber>1</SeasonNumber>
</Episode>
<Episode>
<EpisodeName>Show 66</EpisodeName>
<EpisodeNumber>66</EpisodeNumber>
<FirstAired>1968-05-20</FirstAired>
<SeasonNumber>1</SeasonNumber>
</Episode>
<Episode>
<EpisodeName>Show 67</EpisodeName>
<EpisodeNumber>67</EpisodeNumber>
<FirstAired>1968-05-21</FirstAired>
<SeasonNumber>1</SeasonNumber>
</Episode>
<Episode>
<EpisodeName>Show 68</EpisodeName>
<EpisodeNumber>68</EpisodeNumber>
<FirstAired>1968-05-22</FirstAired>
<SeasonNumber>1</SeasonNumber>
</Episode>
<Episode>
<EpisodeName>Show 69</EpisodeName>
<EpisodeNumber>69</EpisodeNumber>
<FirstAired>1968-05-23</FirstAired>
<SeasonNumber>1</SeasonNumber>
</Episode>
<Episode>
<EpisodeName>Show 70</EpisodeName>
<EpisodeNumber>70</EpisodeNumber>
<FirstAired>1968-05-24</FirstAired>
<SeasonNumber>1</SeasonNumber>
</Episode>
<Episode>
<EpisodeName>Show 71</EpisodeName>
<EpisodeNumber>71</EpisodeNumber>
<FirstAired>1968-05-27</FirstAired>
<SeasonNumber>1</SeasonNumber>
</Episode>
<Episode>
<EpisodeName>Show 72</EpisodeName>
<EpisodeNumber>72</EpisodeNumber>
<FirstAired>1968-05-28</FirstAired>
<SeasonNumber>1</SeasonNumber>
</Episode>
The problem is that on a i7 processor this takes 14.5 seconds. It is about 10x slower on my media center. I tried using a case statement which takes 15 seconds on the fast processor.
I would like to know about how to speed this process up. It seems that this is ridiculously slow for BASH which is supposed to be designed around data manipulation and file operations.