Screen scraping

What is screen scraping?

Screen scraping is reading data straight from what an application shows on screen, instead of getting it through a proper data channel such as an API, an export, or a database query. The automation uses the same text and pixels a person would look at.

It is mostly a fallback. When a system has no supported way to hand over its data, and the information you need is only visible in its windows, screen scraping lets a robot pull it out anyway. This is common with old in-house software and remote desktop sessions.

Screen scraping shows up a lot in RPA, where a flow has to work with applications that were never built to be automated. It reaches data that nothing else can, and it breaks easily, which is the trade-off at the centre of the technique.

How screen scraping reads a screen

Some applications expose their text and controls through an accessibility or UI-automation layer. A scraper can then read values directly, which is the most reliable route.

When only an image is available, such as a graphics canvas, a remote stream, or a scanned page, the tool falls back to OCR. It captures a region of the screen and recognises the text inside it. Pixel- and coordinate-based reading is sensitive to resolution, zoom, font, theme, and window position, so structured elements are preferred wherever they exist.

Screen scraping versus an API integration

An API gives you a defined contract: named fields, stable formats, and a version the vendor commits to. Reading through it is predictable, and a change is usually announced ahead of time.

Screen scraping reads the presentation layer, which was designed for human eyes, not for machines. Nothing promises the layout will stay put. A cosmetic redesign that a person would barely notice can break a scraper overnight. When a system offers an API, an export, or a custom connector, that route almost always beats scraping the screen. Keep scraping for the systems that leave you no other option.

A worked example

An old finance system shows a daily balance report and has no export button. An unattended flow opens the report and reads account, date, and amount from fixed rows.

Before trusting the numbers, the flow confirms the report title and the reporting date, parses amounts with the right locale, and compares the row count against a total shown on screen. If a column looks unexpected or OCR confidence is low, it stops rather than saving a half-read report as if it were complete.

What to watch out for with screen scraping

Layout changes are the main risk. An application update can move columns, rename labels, or add a dialog. Monitor for extraction failures and for values that look wrong, and test across resolutions, scaling settings, languages, and data variations.

Validate what you read. Use an anchor such as a title or a label to confirm you are on the right screen and reading the right area. Check data types, ranges, and totals. A wrong but plausible number can carry high OCR confidence, so confidence alone is not proof.

Mind privacy and access rules. A scraper often sees the same sensitive data a human user does. Secure its credentials and any screenshots it keeps for diagnostics, mask sensitive parts, and set a retention limit. Check the terms of use as well; information being visible does not make it free to collect automatically.

Plan the exit. Write down why scraping is needed and when it can be retired. A temporary workaround has a way of quietly becoming a business-critical integration that no one dares touch.