Thursday, 6 March 2025

MR2 Mk1 at 40 years old - Deep dive into the MOT Data

**** Work in progress march 2025 **** 

Following up on a recent article based on a survey of over 600 Toyota MR2 Mk1s, that included some data from downloaded DVLA MOT data, it was decided to have a deeper dive into the MOT data and see what can be found.  

Description of the data

The UK Driver and Vehicle  Licensing Agency is the statuary body that holds information on cars and drivers in UK. As part of its mission it provides controlled access some of its data. The annual car roadworthy test data ( MOT ) listing the tests and results is of most interest here. The MOT data, that includes some car details and MOT test results, can be downloaded directly from DVLA but together is about 370 Gigabytes of Data ( about 1/2 a domestic computer full).  Filtering the data on Make = Toyota and Model = MR2 within the years 1984 & 1990 shrinks the data down to a more managble 30 Megabytes. Some picking is then required.


The DVLA MOT data is partly historical and partly current with new MOT test results being added monthly. Also individual cars change as some are repainted a new colour, some have changed engines to replace the original equipment and some are older cars imported to the country from other markets (typically Japan some with superchargers). The data does not indicate when such changes are made and only provides a snapshot of the overall MOT register on a specific extraction date. The data in this study runs from approximately year 2005 up to near the end over December 2024. The last test included has 

 lastMotTestDate of 2024-12-28T17:41:51

Just to complicate matters the MOT test changed in XYZZ to have different failure codes and explanations.

Format of the data

The DVLA data is provided in a self describing .json format which looks in the raw like this..

{"_id":{"$oid":"67889a317b39617a2f7a7da4"},"registration":"G190OMX","firstUsedDate":"1989-08-01","registrationDate":"1989-08-01","manufactureDate":"1989-12-31","primaryColour":"White","secondaryColour":"Not Stated","engineSize":1587,"model":"MR2","make":"TOYOTA","fuelType":"Petrol","lastMotTestDate":"2024-08-28T15:17:05.000Z","motTests":[{"completedDate":"2008-05-10T09:24:15.000Z","testResult":"FAILED","odometerValue":40607,"odometerUnit":"MI","odometerResultType":"READ","defects":[{"dangerous":false,"text":"Front Windscreen wiper does not clear the windscreen effectively (8.2.2)","type":"FAIL"},{"dangerous":false,"text":"Front brake disc slightly pitted (3.5.1h)","type":"ADVISORY"},{"dangerous":false,"text":"Offside Front Windscreen washer provides insufficient washer liquid (8.2.3)","type":"PRS"},{"dangerous":false,"text":"Rear Anti-roll bar corroded but not seriously weakened (2.4.G.1)","type":"ADVISORY"},{"dangerous":false,"text":"Rear brake disc slightly pitted (3.5.1h)","type":"ADVISORY"},{"dangerous":false,"text":"Service brake:service brake efficiency only just met. It would appear that the braking system requires adjustment or repair. (3.7.B.7)","type":"ADVISORY"}]},{"completedDate":"2008-05-13T11:37:52.000Z","expiryDate":"2009-05-.....***SNIP**** ..28","testResult":"PASSED","odometerValue":54061,"odometerUnit":"MI","odometerResultType":"READ","defects":[]},{"completedDate":"2020-08-18T08:40:27.000Z","expiryDate":"2021-08-28","testResult":"PASSED","odometerValue":55384,"odometerUnit":"MI","odometerResultType":"READ","defects":[]},{"completedDate":"2021-08-19T14:49:01.000Z","expiryDate":"2022-08-08-28T15:17:05.000Z","expiryDate":"2025-08-28","testResult":"PASSED","odometerValue":58763,"odometerUnit":"MI","odometerResultType":"READ","defects":[]}],"lastUpdateTimestamp":"2024-08-28 15:17:05.000000","dataSource":"dvsa","lastUpdateDate":"2024-08-28","modification":"UPDATED"}


Which when expanded out looks like ...

{

  "_id": {

    "$oid": "67889a317b39617a2f7a7da4"

  },

  "registration": "G190OMX",

  "firstUsedDate": "1989-08-01",

  "registrationDate": "1989-08-01",

  "manufactureDate": "1989-12-31",

  "primaryColour": "White",

  "secondaryColour": "Not Stated",

  "engineSize": 1587,

  "model": "MR2",

  "make": "TOYOTA",

  "fuelType": "Petrol",

  "lastMotTestDate": "2024-08-28T15:17:05.000Z",

  "motTests": [

    {

      "completedDate": "2008-05-10T09:24:15.000Z",

      "testResult": "FAILED",

      "odometerValue": 40607,

      "odometerUnit": "MI",

      "odometerResultType": "READ",

      "defects": [

        {

          "dangerous": false,

          "text": "Front Windscreen wiper does not clear the windscreen effectively (8.2.2)",

          "type": "FAIL"

        },

        {

          "dangerous": false,

          "text": "Front brake disc slightly pitted (3.5.1h)",

          "type": "ADVISORY"

        },

        {

          "dangerous": false,

          "text": "Offside Front Windscreen washer provides insufficient washer liquid (8.2.3)",

          "type": "PRS"

        }, ....SNIP


but can be easily extracted with a Perl script and expanded into a more readable text ...

=> G190OMX TOYOTA MR2 NumberOfTests = 17

 c_lastMotTestDate > 2024-08-28T15:17:05.000Z

 c_engineSize > 1587

 c_registration > G190OMX

......snip

 c_model > MR2

 c_modification > UPDATED

 c_secondaryColour > Not Stated

 c_dataSource > dvsa

 t> FAILED on 2008-05-10T09:24:15.000Z with 40607 MI

   w> FAIL Front Windscreen wiper does not clear the windscreen effectively (8.2.2)

   w> ADVISORY Front brake disc slightly pitted (3.5.1h)

   w> PRS Offside Front Windscreen washer provides insufficient washer liquid (8.2.3)

   w> ADVISORY Rear Anti-roll bar corroded but not seriously weakened (2.4.G.1)

   w> ADVISORY Rear brake disc slightly pitted (3.5.1h)

   w> ADVISORY Service brake:service brake efficiency only just met. It would appear that the braking system requires adjustment or repair. (3.7.B.7)

 t> PASSED on 2008-05-13T11:37:52.000Z with 40607 MI

 t> FAILED on 2009-04-20T13:52:13.000Z with 41255 MI

   w> ADVISORY Front Shock absorber has a slightly reduced damping effect (2.7.5)

   w> ADVISORY Rear Anti-roll bar corroded but not seriously weakened (2.4.G.1)

   w> FAIL Rear Exhaust has a major leak of exhaust gases (7.1.2)

   w> ADVISORY Rear brake disc slightly pitted (3.5.1h)

   w> FAIL Nearside Front Tyre has a cut in excess of the requirements deep enough to reach the ply or cords (4.1.D.1a)

   w> FAIL Nearside Rear Tyre has a cut in excess of the requirements deep enough to reach the ply or cords (4.1.D.1a)

   w> FAIL Offside Rear Tyre has a cut in excess of the requirements deep enough to reach the ply or cords (4.1.D.1a)

 t> PASSED on 2009-04-21T16:25:38.000Z with 41255 MI

 t> PASSED on 2012-08-09T12:19:00.000Z with 41888 MI

 t> PASSED on 2013-07-17T20:16:43.000Z with 43155 MI

 t> PASSED on 2014-07-19T18:18:19.000Z with 45836 MI

.......snip

 t> PASSED on 2022-08-23T11:04:28.000Z with 57047 MI

 t> PASSED on 2023-08-29T14:16:36.000Z with 58086 MI

 t> PASSED on 2024-08-28T15:17:05.000Z with 58763 MI

or extracted to Excel ready format ( that does not include the individual test results ) 

registration,manufactureDate,registrationDate,firstUsedDate,primaryColour,secondaryColour,engineSize,make,model,fuelType,lastMotTestDate,testResult,odometerValue,odometerUnit,expiryDate,lastMOT+1

G190OMX,1989-12-31,1989-08-01,1989-08-01,White,Not Stated,1587,TOYOTA,MR2,Petrol,2024-08-28T15:17:05.000Z,PASSED,58763,MI,2025-08-28,17

Looking at the readable text format the structure of the data can be seen.

=> Registration number

c_CarDetailName ( multiple values )

t> MOT Test and result ( Multiple ) each failed test has ...

     w> MOT test failure reasons and description ( Multiple Entries )

The original .json data can be directly loaded into a MongoDB or similar for further processing. The structure of the data appears quite easy to access but the content of the data fields have a few wrinkles. Notably seen in the MR2 data not all the information is present for all records. The first MR2 were 20 years old when MOTs went from paper based processes the digital format that can be read today. From the data it does look like many of the early MR2s have a record but no further MOT information indicating that the car was scrapped or exported before reaching 20 years old. There are about 13,800 MR2 records with about 6400 ( 46 %) having some MOT history.

In a further wrinkle, when down loading all the MOT data you end up with "bulk-" files and much smaller "delta-" files.  The delta files seem to contain a much small number of updated records presumably these are updates to the bulk files that are only generated once a month or so. 

Mk1 or Mk2 or other ?

The MOT data makes no consistant distinction between Mk1 and Mk2 & Mk3 version of the MR2 but there are some indicators that can be used to split the MR2 into the three types. The three date fields, (ManufactureDate, RegistrationDate, FirstUsedDate) and engine size should be enough to tell the difference between the models. The data does not entirely help with this distinction.

The model field gives us inconsistent data, shown here with a < on the end of the data. Most cars 58,819 are recorded as model MR2 but others have some trailing extra information. It's hard to see but a very few of the model names have an extra trailing space after the name.  It is these sort of inconsistencies that when used in combination with an equality search rather than pattern match search could result in recall or car identification look up failures.

Count, Make Model


58819 TOYOTA MR2 <

   1      TOYOTA MR2  <

   1      TOYOTA MR2 2.0GTI 16V 2 DOOR MANUAL COUPE <

 166     TOYOTA MR2 AUTO <

2192     TOYOTA MR2 GT <

 139     TOYOTA MR2 GT 16V <

   1     TOYOTA MR2 GT TURBO <

   1     TOYOTA MR2 GT TURBO  <

   1     TOYOTA MR2 ROADSTER <

   1     TOYOTA MR2 ROADSTER  <

   3     TOYOTA MR2 ROADSTER TF300 VVT-I <

   3     TOYOTA MR2 ROADSTER VVT-I <

 332    TOYOTA MR2 ROADSTER VVTI <

  20     TOYOTA MR2 ROADSTER VVTI AUTO SMT <

   7     TOYOTA MR2 VVT-I RED <


ManufactureDate, RegistrationDate, FirstUsedDate

The ManufactureDate appears to only indicate the year defaulting to 1/December/Year in most cases. The RegistrationDate appears to be more accurate peaking in August of most years but usually shows as before the ManufactureDate that is set to 1st December of the same year. The FirstUsedDate is not set for many cars and when it is either the same as the RegistrationDate for (5582 cars) or a long way before or after. 

The cars having FirstUsedDate a long (over 500 days) after the RegistrationDate appear to be cars that were possibly imported carrying in the original Registration date from the country of origin.  Car with a FirstUsedDate a long way before the RegistrationDate are when the FirstUsedDate was carried in from country of origin and the RegistrationDate should be read as first registered in the UK. This appears to be a large inconsistency in the dating of cars of foreign origin. There may also be some cars that were bought then stored in unregistered condition or moved between countries adding to this confusion.

The most interesting part of this date analysis is noticing that there are couple of clusters where the FirstUsedDate is about 150/151 days ( for 100 cars) or 360 days, ( for 12 cars) after the registration date. It is thought that these are "Show room" cars that were registered then displayed for 6 or 12 months then sold off. [ See R2u_2 tab ]

Mk1 or Mk2 or Mk3 or other ( again )

There appears to be some variance in the recorded CC of the cars even when thought to be standard size. The CC field is apparently input by the MOT testers as all the registration records without MOT data have a blank Engine size field. This would make sense as some engine swaps occur. The only exception to this is car D21FPW that has MOT data but happens to have fuelType set as Electric.

As a side note this engine conversion must have happened after 2015 as the test for that year reports ..
"FAILED on 2015-02-21T10:16:21.000Z with 111350 MI

   ADVISORY Exhaust has a minor leak of exhaust gases (7.1.2)"


MR2 TypeOfficial production datesUsual Engine size
MK11984 - 19891587cc
MK21989 - 1991 August1990cc
MK3 Roadster1991 Oct - 2007 July1800cc

Looking now at the more reliable RegistrationDate and engine sizes. Will need all the MR2 Data to complete this as we have see some cars being registered well after the apparent ManufactureDate.

After a small bit of data cleaning we have some first results from the 56k Tribe numbers
This is RegistrationDate V Engine size

There are 10,214 cars not showing here as they have a blank engine size indicating that they never got an MOT recorded after 2005. Removing those from the picture highlights the engine differences. The 1800-1899 column is also a bit ambiguous between MK2 & Mk3 but on closer examination most cars are recorded as 1800cc ( being the engine size in a MK3 ).

There were 10 cars listed with 9999cc and 19901cc engine sizes. These are bad data as no 10 or 20 litre engines are available for fitment. The two with engineSize 1200->1500 cc have probably had a motorbike engine replacements.



Red = MK1
Blue = MK2
Purple = MK3
Black are probable engine swaps
The unboxed date requires further investigation as may be incorrectly entered data or imports / late registrations.
The bad data cc < 200 or cc > 9000





MOT Details pass / fail rate and repeat tests


Most likely to fail on 



Acknowledgements

Steve Earle for MOT Data Extraction and detailed assistance.

Data From Neil Jones at Classic MR2 Parts and MR2 Drivers Club

Contact Neil Jones for spares via email on Mk1mr2@live.co.uk or message on on Instagram @neil.Jones.mr2






Wednesday, 5 March 2025

Warp shell

After watching a Youtube by the very respected Dave on a new command line shell called Warp, I gave its go. After describing a task to be done it built the commands needed for that task and executed them. In this case I asked it to "create a page of thumbnails images from a directory of pictures". As it progressed it provided explanations and checked before running any actual commands. The task was achieved by generating a command script incorporating the subtasks to be done.  The script was left behind and can be adapted and deployed in other similar situations.

Very impressive I can see we will have to explore this tool further. 

Available on Mac, Windows, Linux from Warp.dev


...........



The script created .. Might need to add .png and parameterise a few things before redeployment

#!/bin/bash


# Create thumbnails directory if it doesn't exist

mkdir -p CEP_Thumbnails


# Remove any existing thumbnails to start fresh

rm -f CEP_Thumbnails/*.jpg CEP_Thumbnails/*.jpeg


# Counter for thumbnails

count=0


# Process all JPG and JPEG files

for img in CEP_AllPics/*.jpg CEP_AllPics/*.jpeg CEP_AllPics/*.JPG CEP_AllPics/*.JPEG; do

    # Check if file exists (to handle case when no matches are found)

    if [ -f "$img" ]; then

        # Extract filename without path

        filename=$(basename "$img")

        echo "Creating thumbnail for $filename"

        

        # Create thumbnail using sips

        sips --resampleWidth 150 "$img" --out "CEP_Thumbnails/$filename"

        

        # Increment counter

        ((count++))

    fi

done


echo "Created $count thumbnails"


# Generate HTML file

cat > thumbnails.html << EOF

<!DOCTYPE html>

<html lang="en">

<head>

    <meta charset="UTF-8">

    <meta name="viewport" content="width=device-width, initial-scale=1.0">

    <title>Image Thumbnails</title>

    <style>

        body {

            font-family: Arial, sans-serif;

            margin: 20px;

            background-color: #f5f5f5;

        }

        h1 {

            text-align: center;

            color: #333;

        }

        .thumbnail-grid {

            display: grid;

            grid-template-columns: repeat(auto-fill, minmax(170px, 1fr));

            gap: 15px;

            padding: 20px;

        }

        .thumbnail-container {

            background: white;

            box-shadow: 0 2px 5px rgba(0,0,0,0.1);

            padding: 10px;

            border-radius: 5px;

            text-align: center;

            transition: transform 0.3s ease;

        }

        .thumbnail-container:hover {

            transform: scale(1.05);

        }

        .thumbnail-container img {

            max-width: 150px;

            max-height: 150px;

            display: block;

            margin: 0 auto 8px auto;

        }

        .thumbnail-container p {

            margin: 5px 0 0 0;

            font-size: 12px;

            color: #666;

            overflow: hidden;

            text-overflow: ellipsis;

            white-space: nowrap;

        }

        .timestamp {

            text-align: center;

            margin-top: 30px;

            color: #777;

            font-size: 14px;

        }

    </style>

</head>

<body>

    <h1>Image Thumbnails</h1>

    <div class="thumbnail-grid">

EOF


# Add each thumbnail to the HTML

for thumb in CEP_Thumbnails/*.jpg CEP_Thumbnails/*.jpeg CEP_Thumbnails/*.JPG CEP_Thumbnails/*.JPEG; do

    if [ -f "$thumb" ]; then

        filename=$(basename "$thumb")

        echo "        <div class=\"thumbnail-container\">" >> thumbnails.html

        echo "            <a href=\"CEP_AllPics/$filename\" target=\"_blank\">" >> thumbnails.html

        echo "                <img src=\"$thumb\" alt=\"$filename\">" >> thumbnails.html

        echo "                <p>$filename</p>" >> thumbnails.html

        echo "            </a>" >> thumbnails.html

        echo "        </div>" >> thumbnails.html

    fi

done


# Close the HTML file

cat >> thumbnails.html << EOF

    </div>

    <div class="timestamp">

        Generated on $(date)

    </div>

</body>

</html>

EOF


echo "HTML gallery created: thumbnails.html"

echo "Open this file in your web browser to view the thumbnails."


# Make the script executable

chmod +x create_thumbnails.sh