Automatically Scraping Workout Data with Playwright + Alibaba Cloud OCR and Showing It in Hexo

目录

Recently I needed to sync my personal training records from a certain sports website (RQ Run) into my Hexo blog. Because the site requires login to view detailed data and the login involves a captcha, I wrote an automation script with the Node.js version of Playwright, used the Alibaba Cloud OCR service to recognize the captcha, achieved automatic data scraping, and displayed it on the blog via an EJS template.

This article walks through the whole implementation in detail.

1. Tech Stack

• Playwright: Microsoft’s open-source automation testing tool, supporting Chromium, Firefox and WebKit. Compared with Puppeteer, it is more modern, more stable, and supports multiple browsers.
• Alibaba Cloud OCR: used to recognize the image captcha at login.
• Hexo: the static blog framework; its _data data-file feature stores the scraped JSON data.

2. Environment Setup

First initialize the project and install the necessary dependencies:

npm install playwright @alicloud/ocr-api20210707 @alicloud/openapi-client @alicloud/tea-util @alicloud/darabonba-stream
npx playwright install chromium

3. Writing the Scraper Script (rq_scraper.js)

The script’s core logic has three steps: simulate login, recognize the captcha, scrape the data.

3.1 Initializing the Browser

To avoid being detected as a bot, we need to configure the UserAgent and HTTP headers. Also, Playwright’s headless: true mode lets the script run silently in the background.

const browser = await chromium.launch({
    headless: true // Enable headless mode
});

const context = await browser.newContext({
    userAgent: 'Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/144.0.0.0 Safari/537.36',
    viewport: { width: 1920, height: 1080 },
    extraHTTPHeaders: {
        'sec-fetch-dest': 'document',
        'sec-fetch-mode': 'navigate',
        'sec-fetch-site': 'same-origin',
        // Note: don't set 'accept': 'application/json', or normal page visits will error
    }
});

3.2 Captcha Recognition and Auto-Retry

The login page has a captcha image that refreshes on click. We first screenshot and save the captcha, then call the Alibaba Cloud OCR API to recognize it. If the recognized captcha isn’t 4 digits, or the page doesn’t navigate after submitting (meaning the captcha was wrong), the script automatically refreshes the captcha and retries.

// Alibaba Cloud OCR recognition function
async function recognizeCaptchaAliyun(filePath) {
    // ... configure the Alibaba Cloud client ...
    const bodyStream = Stream.default.readFromFilePath(filePath);
    const recognizeAllTextRequest = new ocr_api20210707.RecognizeAllTextRequest({
        body: bodyStream,
        type: 'Advanced'
    });
    // ... call the API and return the result ...
}

// Login retry logic
let loginSuccess = false;
let attempts = 0;
while (!loginSuccess && attempts < 5) {
    // 1. Screenshot the captcha
    await captchaElement.screenshot({ path: 'captcha.png' });
    
    // 2. OCR recognition
    let code = await recognizeCaptchaAliyun('captcha.png');
    
    // 3. Fill in the captcha and submit
    if (code.length === 4) {
        await page.fill('#login_code', code);
        await loginBtn.click();
        
        // 4. Wait for navigation to verify login succeeded
        try {
            await page.waitForNavigation({ timeout: 10000 });
            loginSuccess = true;
        } catch (e) {
            console.log('Login timed out or failed, retrying...');
        }
    } else {
        // Wrong length recognized, refresh the captcha
        await captchaElement.click();
    }
}

3.3 Data Scraping (API Interception vs Page Parsing)

At first I tried parsing the HTML table directly, but it was inefficient and easily broken by page-structure changes. After analyzing the network requests, I found the site fetches JSON via an API. So I switched to calling that API directly in the browser context.

Here I use page.evaluate to run the fetch request inside the browser, which lets us leverage the browser’s already-logged-in cookie state without extracting cookies manually.

const apiData = await page.evaluate(async () => {
    const allData = [];
    let page = 1;
    const limit = 50;
    let totalPages = 1;

    while (page <= totalPages) {
        const response = await fetch(`https://www.rq.run/Dc/Api?_=User/Record/get_list&page=${page}&limit=${limit}...`);
        const json = await response.json();
        
        if (page === 1) {
            // Compute how many pages to turn based on the total
            totalPages = Math.ceil(json.data.total / limit);
        }
        
        allData.push(...json.data.list);
        page++;
        await new Promise(r => setTimeout(r, 200)); // Avoid requesting too fast
    }
    return allData;
});

3.4 Saving the Data

The scraped data is saved into Hexo’s data directory at source/_data/rq_data.json. Hexo automatically loads that file as site.data.rq_data for templates to use.

4. Front-End Display (Hexo + EJS)

In the Hexo layout file (layout/running.ejs), I added a “Workout Records” tab. Because the dataset is large (possibly several hundred entries), rendering it all at once would be very long, so I implemented pagination on the front end.

4.1 Data Preprocessing

In EJS, I first cleaned and sorted the data (in reverse chronological order).

<%
  var rqData = site.data.rq_data || [];
  // Sort in reverse chronological order
  rqData.sort(function(a, b) {
    return new Date(b.training_at).getTime() - new Date(a.training_at).getTime();
  });
%>

4.2 Client-Side Paginated Rendering

For a better user experience, I inject the data into the HTML via a data- attribute and use JavaScript for paginated rendering.

// Example front-end JS logic
const records = JSON.parse(document.getElementById('raw-data').textContent);
const pageSize = 10;

function renderPage(page) {
    const start = (page - 1) * pageSize;
    const end = start + pageSize;
    const pageData = records.slice(start, end);
    
    // Generate HTML and insert it into the container
    const html = pageData.map(item => `
        <div class="record-card">
            <div class="date">${item.training_at}</div>
            <div class="distance">${item.distance} km</div>
            <div class="pace">${item.avg_speed}</div>
        </div>
    `).join('');
    
    container.innerHTML = html;
}

5. Summary

With Playwright and Alibaba Cloud OCR, we successfully tackled the challenge of simulated login with a captcha; by calling the backend API directly, we improved the efficiency and stability of scraping. Finally, combined with Hexo’s data-driven capability, we built an automated personal workout-data display page.

If you have similar needs, give this approach a try!

中英对照

文章作者：阿文

文章链接： https://www.awen.me/post/c01cfe3f.html

版权声明：本博客所有文章除特别声明外，均采用 CC BY-NC-SA 4.0 许可协议。转载请注明来自 阿文的博客！