Progress 4GL / OpenEdge

Progress OpenEdge (formerly Progress 4GL) is an application development platform and database management system widely used for business applications. DataMeans reads Progress data from the text dumps the Data Dictionary writes: a .df schema file and .d data files.

What You Can Upload

• A .df schema dump (Admin > Dump Data and Definitions > Data Definitions)
• .d data dumps, one per table (Admin > Dump Data and Definitions > Table Contents, or the ABL EXPORT statement)
• A ZIP archive containing the .df and .d files

The binary database can't be read: the .db file, its extents (.d1, .b1 and so on) and the .st structure file. Note the difference between a .d text dump, which DataMeans reads, and a .d1 data extent, which it can't.

What You Get Out

DataMeans converts your data into multiple modern formats:

How to Export / Obtain Files

1. In the Data Administration tool, dump your schema: Admin > Dump Data and Definitions > Data Definitions (.df file)
2. Dump your table data: Admin > Dump Data and Definitions > Table Contents (.d file). In ABL, the equivalent for one table is OUTPUT TO "table.d". FOR EACH table-name NO-LOCK: EXPORT table-name. END. OUTPUT CLOSE.
3. Create a ZIP archive containing the .df and .d text files
4. Upload the ZIP archive to DataMeans

Supported Features

• Tables and fields (names, data types and field order) from the .df schema
• Rows from each table's .d dump, with numbers and dates read the way the dump says it wrote them (European decimals such as 119,5, day-month-year or month-day-year dates) and text in the code page the dump names
• Array fields (EXTENT) become one column per element
• A preview of every table before you convert: its columns, the number of records Progress counted in the dump, and the first rows, read the same way as the conversion. Tables in the .df that have no .d in the upload are shown as schema only

Known Limitations

• The binary database and its extents can't be read: dump to .df and .d first
• Very large tables are read whole and held in memory while they convert: a single table of a few hundred megabytes or more may not fit, in which case dump it in parts (one table, or one range of records, per file)
• Indexes, sequences, triggers and stored procedures aren't converted
• Application logic (4GL / ABL code) isn't migrated, only the data

Troubleshooting

Note on the Excel formats. An .xlsx sheet holds at most 1,048,575 rows of data and 16,384 columns, and a legacy .xls sheet 65,535 rows and 256 columns. Rather than ship a file that looks complete but is silently cut short, DataMeans leaves a table out of any Excel format it does not fit and notes it in the job report. The .xlsx copy is also left out for tables of more than 8,000,000 cells (1,048,575 rows of 12 columns is 12.6 million), which take too long and too much memory to write, and the .xls copy for tables of more than 150,000 cells, which the legacy format takes far too long to write. The CSV, JSON and SQL outputs always contain every row.

Overview

Progress 4GL (now OpenEdge ABL) is a fourth-generation programming language and database platform developed by Progress Software Corporation. It combines a relational database with an English-like programming language for rapid business application development. The database uses a proprietary format optimized for transactional processing and complex business logic.

History and Background

• 1981: Progress Software founded as Data Language Corporation (DLC) by MIT graduates including Joseph Alsop, Clyde Kessel, and Chip Ziering.
• 1984: First commercial version of Progress ships for Unix.
• 1987: Company renamed Progress Software, after its flagship product.
• 1993: Progress Version 7 adds GUI development for Windows and Motif.
• 1995: Progress Version 8 introduces component-based visual development.
• 1998: Progress Version 9 introduces Type I storage areas.
• 2003: OpenEdge 10 released, the first OpenEdge-branded version, introducing Type II storage areas.
• 2005: OpenEdge 10.1 adds object-oriented language extensions.
• 2006: Progress 4GL renamed OpenEdge ABL (Advanced Business Language).
• 2009: OpenEdge 10.2 released; release 10.2B adds Transparent Data Encryption for data at rest.
• 2011: OpenEdge 11 introduces multi-tenant database tables.
• 2014: OpenEdge 11.5 introduces the Pacific Application Server for OpenEdge (PASOE).
• 2017: OpenEdge 11.7, a long-term support release, introduces Change Data Capture.
• 2019: OpenEdge 12 adds a multi-threaded database server.
• 2022: OpenEdge 12.5 adds an ABL API for Apache Kafka messaging.
• 2024: OpenEdge 12.8 adds Dynamic Data Masking and OpenTelemetry-based application observability.
• 2026: OpenEdge 13.0 released, resolving the year-2038 problem by widening on-disk time values from 32 to 64 bits.

File Format Specifications

Database Files:

• .db: Database control area, a binary table of contents listing every area and extent
• .d1, .d2, etc.: Data extents holding table data, indexes, and LOBs
• .b1, .b2, etc.: Before-image extents used to undo incomplete transactions
• .a1, .a2, etc.: After-image extents for roll-forward recovery
• .lg: Text log file recording significant database events
• .lk: Lock file created at session startup, recording whether the database is open in single-user or multi-user mode
• .st: Structure description file, a text file defining the database's areas and extents
• .tn: Transaction log extents (fixed-length only) listing committed two-phase commit transactions

Storage Architecture:

• Block-based storage with configurable block sizes (1KB, 2KB, 4KB, or 8KB)
• B-tree indexes for fast lookups
• Record-level locking for concurrency
• Default block size of 4KB on Windows and Linux, 8KB on UNIX
• Databases divided into storage areas composed of one or more extents
• Up to 32,000 storage areas per database, of which 31,994 can hold application data
• Up to 1,024 extents per area and 1TB per extent; Type II areas can reach about 1PB with large files enabled
• Records per block configurable in powers of two from 1 to 256, defaulting to 64 for 8KB blocks and 32 for other block sizes

File Structure:

• Schema stored within the database's schema area
• Compiled ABL code in .r files (r-code)
• Data dictionary for metadata management
• Audit trails and security settings
• Replication and high-availability configurations
• Limits of 32,767 table definitions and 32,767 indexes per database
• Up to 1,000 fields per table through ABL, or 500 through the SQL engine
• Index entries of at most 16 fields each

Data Types and Structures

Database Objects:

• Tables: Primary data containers
• Indexes: B-tree structures for performance
• Sequences: Auto-incrementing number generators (32-bit and 64-bit)
• Triggers: ABL schema triggers and Java-based SQL triggers
• Stored Procedures: Java routines executed by the SQL engine
• Views: Virtual tables defined through the SQL engine

Version Differences

Compatibility Notes:

• Databases can be upgraded but not downgraded; reverting requires a restore from backup
• The database format changes between major releases; PROUTIL conversion utilities (CONV89, CONV910, CONV1011, CONV1112) upgrade databases in place
• Minor releases share a format; a database created under any Release 11.x can be opened by any other 11.x release without conversion
• Some data types require minimum versions; INT64 fields require OpenEdge 10.1B or later
• Single-volume Version 8 databases must be converted to multi-volume structure before upgrading to Version 9
• Migration utilities are provided for major-version upgrades

Technical References

• Progress Documentation
• Wikipedia: Progress Software
• Wikipedia: OpenEdge Advanced Business Language
• OpenEdge Database Essentials: Database File Structure
• OpenEdge Life Cycle