Journaling: Unraveling the mysteries of sporadic growth of Journal receivers
Web Doc
Note: This is publication is now archived. For reference only.
Abstract
Do you like a good mystery? If you are a person who likes to keep an eye on the size of your journal receivers, then you might have noticed a somewhat mysterious phenomenon. From time to time, journal receivers grow beyond their designated size threshold and then suddenly seem to shrink in size. How can that be? Are critical journal entries disappearing? In this technote, we explain what you have witnessed and put your mind at ease.
Written by Larry Youngren, Software Engineer
IBM Systems &Technology Group, Development
Development iSeries Journaling
Contents
A little background
Journal receivers serve multiple missions. In a sense, they wear multiple hats. On one side, they identify and record every alteration you make to the rows residing within your database files. Such data (often in the form of “after” images) is employed to serve as an audit trail of the changes made to your files. High Availability products often capitalize on the presence of such after images to replay matching changes to replica database files on a target machine.
On the other side, the journal also has a decidedly different role. It is the place to which the IPL and recovery code turn in order to repair and recover your database files and matching keyed access paths that reside on the production machine, if the machine goes down abruptly. "Before" images are among the kinds of journal entries which help serve this purpose. They help i5/OS™ start recovery from a known, concise starting point. This is especially true for the before images deposited on behalf of system-managed access path protection (SMAPP). That is the automated protection mechanism which helps insure that the duration of your IPL and matching recovery steps for any access paths open at the time of the crash is limited. After all, no one ever sends us thank-you letters for long duration IPLs.
Two zones are better than one
If you could look inside a journal receiver, you would discover that it is divided into two physical zones. Let us call them zones “A” and “B”.
Zone “A” is the normal residence of visible journal entries. These are the before and after images which show up because you elect to provide journal protection for a database physical file or SQL table. It is this portion of a journal receiver that houses journal entries such as R/PT = record “put” and R/DL = record “delete”. This portion of a journal receiver grows in size as more journal entries arrive. It never shrinks. This is also the portion of the journal receiver that is written to tape if you elect to issue a SAVOBJ request against a journal receiver. In addition, zone "A" is the only portion which is sent to a remote journal, and the only portion from which high availability software harvests and replays database actions. Every journal receiver has a zone “A”.
So, what is the purpose of zone “B”
The other zone, zone “B”, is the optional area of a journal receiver. It is present ONLY if you elect to specify RcvSizOpt(*RmvIntEnt) as a parameter on your CHGJRN command. The *RmvIntEnt option says “periodically remove and retire the internal journal entries”.
And what are these “internal” journal entries that can be removed? They are the “before” images deposited by SMAPP on behalf of keyed access paths that i5/OS has decided need protection. Their presence helps cap the duration of recovery time if you crash.
For journals without a zone “B”, the hidden SMAPP-induced journal entries are forced to flow into zone “A” instead, and that can be a bad thing. This rapid influx into zone “A” can chew up a lot of space. Yet, such entries (those on behalf of SMAPP) are transient in nature and their usefulness quickly fades, often within minutes of arrival. Do you think it would be good if there were a mechanism by which these transient journal entries could be separated from the rest and the space that they occupy could be recycled on a regular basis? That is precisely what the use of the *RmvIntEnt setting lets you do. Therefore, it is a wise choice.
When *RmvIntEnt is the attribute that you have selected for your journal, then the optional zone “B” is carved out and any SMAPP-induced entries are routed to zone “B”. We like to think of the journal receiver as having a fork in the road. Along comes a new journal entry desiring to be deposited. When these entries reach the fork in the road, the visible journal entries take the path to zone “A” while the normally hidden transient journal entries take the path leading to zone “B”. Zone “A” and zone “B” are often found on separate disk units. Among journal experts, zone “B” is known as the “transient” area. It gets that name because journal entries flowing into this area linger only until there is no one relying upon their presence. At that point, they are discarded. That is correct, i5/OS actually throws away such journal entries at appropriate times and recycles the space that they formerly occupied.
Time for some housekeeping
But is this practice safe? Yes, absolutely. You see, periodically i5/OS does a bit of hidden housekeeping. During this housekeeping, i5/OS flushes the access path’s recently changed pages from main memory to disk. Once this has ensued, the journal “before” images (those residing in zone “B”) that correspond to the same pages which were written to disk, are no longer of any interest to the IPL. In fact, if present, IPL code actually skips over such entries. Therefore, once the housekeeping has ensued, such journal entries residing in zone “B” are chaff. They serve no useful purpose. Their moment in the sun has passed. They are simply consuming disk space.
In order to clean up and recycle such disk space, earlier releases of i5/OS periodically physically freed up such space, removed it from the journal receiver, and handed back such space. When this ensued, you would see the total size of your journal receiver shrink. Therefore, occasional oscillation in journal receiver size was a perfectly normal and expected phenomena, especially if you had SMAPP enabled and had also designated use of zone “B” through the *RmvIntEnt option.
For performance reasons (physical return of space is more expensive than mere logical freeing), recent releases no longer behave that way. That is, during the housekeeping, they refrain from physically returning such space and, instead, merely set it aside (unpopulated) temporarily within the journal receiver, thereby, keeping such space around as the residence for the next batch of hidden journal entries about to arrive.
You can think of this like a motel which has dog kennels out back for guests who bring along their pets. In the past (former releases), each time a guest (a journal entry) with a pet (a SMAPP-induced transient "before" image) arrived, we sent out the carpenters to construct a new kennel (fresh space in zone “B”). When the guest checked out and housekeeping ensued, we dismantled the kennel (triggering zone “B” space shrinkage).
Today, by contrast, we behave differently and allow the kennels to persist for a while unpopulated, hoping that the next guest will want to rent some of the freed up space for their pet. However, the kennels do not last forever. Instead, once we are ready to sell the motel to the next owner (that is, CHGJRN), we finally tear down the kennels (thereby physically freeing the transient space of zone “B”). Hence, the populated area of zone “B” rises and falls as housekeeping ensues, but the oscillations are not quite as frequent or as dramatic as they were in the past.
A related mystery
There is a related mystery that we are often called upon to explain. It too is related to the presence of zone “B”.
Some shops elect to gain tight control over the quantity of space consumed by their journal receivers by establishing a size cap. Most do this not so much because they are extra tight on disk space and at risk of using it all up, but rather because they do not want the currently attached journal receiver to grow too bulky before a new receiver is attached. In effect, this is their means of assuring that CHGJRN operations ensue on a regular basis, thereby assuring that no single journal receiver becomes unwieldy in size, and that from a recovery perspective, they do not have too many entries in one journal receiver. Hence, they attempt to cap the total size of each journal receiver.
People, who want such a cap, combine the system-managed CHGJRN behavior with a system-provided trigger mechanism (influenced by total journal receiver size) to assure that a CHGJRN ensues on a regular basis. It is just that simple. Thereafter, the system monitors the journal receivers and uses size to manage CHGJRN behavior automatically for you.
In fact, there is even a matching journal receiver size threshold parameter: THRESHOLD, that you can set either on the CRTJRNRCV command or (beginning in V5R4) on the CHGJRN command. Normal intuition would lead you to believe that if you selected such a threshold, such as 1 GB, and instructed the system to serve as your housekeeper, that no journal receiver attached to this journal would ever exceed the designated size, right? Wrong! How can that be?
The answer again relates to zone “B” and how the space residing within that zone is managed, monitored, and tracked.
It turns out that the threshold enforcement mechanism normally monitors only what resides within zone “A” and generally ignores what resides within zone “B”. The obvious reason is that what resides within zone “A” is not going anywhere. It is permanent and persistent. However, what resides within zone “B” is transient: Here today and gone tomorrow. Hence, rather than exactly insisting on a CHGJRN operation as soon as the combined total size of zone “A” plus zone “B” exceeds the designated size threshold, i5/OS acts more conservatively and only concludes that your size threshold has been crossed if zone “A” is about to increase in size and the total space (the sum of the size of zone “A” plus zone “B”) at that instant exceeds your designated threshold.
That is, actions, which trigger an increase in the size purely of zone “B”, do not monitor the threshold. They are oblivious to it. Rather, it is only actions that trigger the enlargement of zone “A”. These actions take a peek at the threshold and realize that the total sum of space consumed by both zone “A” plus zone “B” now exceeds the designated threshold. Hence, increases in size affecting purely zone “B” can cause your total journal receiver size to creep above the threshold, and you do not know about it.
If we think back to our motel analogy, it is as if the tax assessor (think of the tax assessor as our size enforcer) shows up to determine your tax bill only when you enlarge the main motel (which would happen if you added a new wing), not when you merely assemble another kennel out back. Your motel is zone “A” and it is monitored, while your kennels represent zone “B” and they are largely ignored: out of sight, out of mind. The tax assessor generally only watches what you do to zone “A” but factors in zone “B” when you apply for a new zone “A” building permit.
Hence, you can (and probably will) occasionally see a journal receiver whose total size (the sum of zone “A” plus zone “B”) has exceeded your designated journal receiver threshold. This is a perfectly normal phenomena and more likely to ensue if you tend to have lots of large keyed logical files and/or SQL indexes built over your journaled physical files (because these indexes are the objects aggressively populating zone “B”). In fact, this phenomena is especially prevalent if the logical page size employed by your access paths is 64k or larger (as is often the case for indexes created through SQL).
If you tend to have lots of these “large-leaf” indexes and populate them aggressively, zone “B” fills at a torrid rate and keeps growing in size. Meanwhile, the size of zone “A” barely budges. The result: Total journal receiver size exceeds your threshold.
The asynchronous nature of responding to a threshold
In addition, there is yet another phenomenon, which makes it likely that you will occasionally see a journal receiver whose total size is somewhat larger than the designated size threshold, and that is the fact that the size threshold is monitored at the SLIC (microcode) level while the matching response action (attaching new journal receivers through the CHGJRN command) is performed at the i5/OS level.
When SLIC detects that the size threshold is about to be exceeded, it sends up an asynchronous notification (in the form of an event) to the i5/OS level. So you might have application job number 1 cranking away adding new database rows with each such row generating a new journal entry. When the last few bytes of the designated desired maximum size of zone “A” of your journal receiver (as evidenced by your threshold) have been populated, the SLIC layer of i5/OS (operating under application job number 1) sends a notification as an event (you can think of this as a message) to a queue being monitored by a background housekeeping job of i5/OS (generally, the QDBSRV01 job or one of its cousins).
This i5/OS housekeeping job might have lots of other things it has to do and eventually gets around to noticing the request to attach a new journal receiver. In the meantime, application job number 1 has continued to pour new journal entries into zone “A” (and maybe zone “B” as well!) of the journal receiver.
The net of this is while waiting for the background housekeeping job to respond, the total size of your journal receiver creeps above the designated size.
Here today, gone tomorrow: rapid shrinkage at detach time
There is one final mystery worthy of note: Detached journal receivers are often smaller in size than they were just seconds ago while attached. How can that be? Why would a journal receiver suddenly shrink in size merely because you set it aside and ceased populating it?
The explanation involves what happens during a CHGJRN operation. Not only does CHGJRN attach a new, fresh, empty journal receiver and set aside the former receiver, but it also tidies things up a bit. It reaches back to the detached journal receiver and trims excess space, thereby aggressively pruning both zone “A” and zone “B”. Unneeded transient entries residing within zone “B” might well be discarded, and unpopulated growth space (I like to think of it as extra room) at the end of zone “A” could be as well. The result is that a journal receiver, whose total combined size (zone “A” plus zone “B”) exceeded the designated size threshold a moment ago, but now appears to exceed it no longer.
Conclusion
Well, there you have it: Mystery solved! Yes, journal receivers can (and do) exceed their designated size threshold. And yes, in the past, it was not that your eyes were going bad or that your memory was failing, and no, you were not hallucinating. You really did see a journal receiver’s size shrink. Both phenomena are normal and expected behaviors.
Journal receivers serve multiple missions. In a sense, they wear multiple hats. On one side, they identify and record every alteration you make to the rows residing within your database files. Such data (often in the form of “after” images) is employed to serve as an audit trail of the changes made to your files. High Availability products often capitalize on the presence of such after images to replay matching changes to replica database files on a target machine.
On the other side, the journal also has a decidedly different role. It is the place to which the IPL and recovery code turn in order to repair and recover your database files and matching keyed access paths that reside on the production machine, if the machine goes down abruptly. "Before" images are among the kinds of journal entries which help serve this purpose. They help i5/OS™ start recovery from a known, concise starting point. This is especially true for the before images deposited on behalf of system-managed access path protection (SMAPP). That is the automated protection mechanism which helps insure that the duration of your IPL and matching recovery steps for any access paths open at the time of the crash is limited. After all, no one ever sends us thank-you letters for long duration IPLs.
Two zones are better than one
If you could look inside a journal receiver, you would discover that it is divided into two physical zones. Let us call them zones “A” and “B”.
Zone “A” is the normal residence of visible journal entries. These are the before and after images which show up because you elect to provide journal protection for a database physical file or SQL table. It is this portion of a journal receiver that houses journal entries such as R/PT = record “put” and R/DL = record “delete”. This portion of a journal receiver grows in size as more journal entries arrive. It never shrinks. This is also the portion of the journal receiver that is written to tape if you elect to issue a SAVOBJ request against a journal receiver. In addition, zone "A" is the only portion which is sent to a remote journal, and the only portion from which high availability software harvests and replays database actions. Every journal receiver has a zone “A”.
So, what is the purpose of zone “B”
The other zone, zone “B”, is the optional area of a journal receiver. It is present ONLY if you elect to specify RcvSizOpt(*RmvIntEnt) as a parameter on your CHGJRN command. The *RmvIntEnt option says “periodically remove and retire the internal journal entries”.
And what are these “internal” journal entries that can be removed? They are the “before” images deposited by SMAPP on behalf of keyed access paths that i5/OS has decided need protection. Their presence helps cap the duration of recovery time if you crash.
For journals without a zone “B”, the hidden SMAPP-induced journal entries are forced to flow into zone “A” instead, and that can be a bad thing. This rapid influx into zone “A” can chew up a lot of space. Yet, such entries (those on behalf of SMAPP) are transient in nature and their usefulness quickly fades, often within minutes of arrival. Do you think it would be good if there were a mechanism by which these transient journal entries could be separated from the rest and the space that they occupy could be recycled on a regular basis? That is precisely what the use of the *RmvIntEnt setting lets you do. Therefore, it is a wise choice.
When *RmvIntEnt is the attribute that you have selected for your journal, then the optional zone “B” is carved out and any SMAPP-induced entries are routed to zone “B”. We like to think of the journal receiver as having a fork in the road. Along comes a new journal entry desiring to be deposited. When these entries reach the fork in the road, the visible journal entries take the path to zone “A” while the normally hidden transient journal entries take the path leading to zone “B”. Zone “A” and zone “B” are often found on separate disk units. Among journal experts, zone “B” is known as the “transient” area. It gets that name because journal entries flowing into this area linger only until there is no one relying upon their presence. At that point, they are discarded. That is correct, i5/OS actually throws away such journal entries at appropriate times and recycles the space that they formerly occupied.
Time for some housekeeping
But is this practice safe? Yes, absolutely. You see, periodically i5/OS does a bit of hidden housekeeping. During this housekeeping, i5/OS flushes the access path’s recently changed pages from main memory to disk. Once this has ensued, the journal “before” images (those residing in zone “B”) that correspond to the same pages which were written to disk, are no longer of any interest to the IPL. In fact, if present, IPL code actually skips over such entries. Therefore, once the housekeeping has ensued, such journal entries residing in zone “B” are chaff. They serve no useful purpose. Their moment in the sun has passed. They are simply consuming disk space.
In order to clean up and recycle such disk space, earlier releases of i5/OS periodically physically freed up such space, removed it from the journal receiver, and handed back such space. When this ensued, you would see the total size of your journal receiver shrink. Therefore, occasional oscillation in journal receiver size was a perfectly normal and expected phenomena, especially if you had SMAPP enabled and had also designated use of zone “B” through the *RmvIntEnt option.
For performance reasons (physical return of space is more expensive than mere logical freeing), recent releases no longer behave that way. That is, during the housekeeping, they refrain from physically returning such space and, instead, merely set it aside (unpopulated) temporarily within the journal receiver, thereby, keeping such space around as the residence for the next batch of hidden journal entries about to arrive.
You can think of this like a motel which has dog kennels out back for guests who bring along their pets. In the past (former releases), each time a guest (a journal entry) with a pet (a SMAPP-induced transient "before" image) arrived, we sent out the carpenters to construct a new kennel (fresh space in zone “B”). When the guest checked out and housekeeping ensued, we dismantled the kennel (triggering zone “B” space shrinkage).
Today, by contrast, we behave differently and allow the kennels to persist for a while unpopulated, hoping that the next guest will want to rent some of the freed up space for their pet. However, the kennels do not last forever. Instead, once we are ready to sell the motel to the next owner (that is, CHGJRN), we finally tear down the kennels (thereby physically freeing the transient space of zone “B”). Hence, the populated area of zone “B” rises and falls as housekeeping ensues, but the oscillations are not quite as frequent or as dramatic as they were in the past.
A related mystery
There is a related mystery that we are often called upon to explain. It too is related to the presence of zone “B”.
Some shops elect to gain tight control over the quantity of space consumed by their journal receivers by establishing a size cap. Most do this not so much because they are extra tight on disk space and at risk of using it all up, but rather because they do not want the currently attached journal receiver to grow too bulky before a new receiver is attached. In effect, this is their means of assuring that CHGJRN operations ensue on a regular basis, thereby assuring that no single journal receiver becomes unwieldy in size, and that from a recovery perspective, they do not have too many entries in one journal receiver. Hence, they attempt to cap the total size of each journal receiver.
People, who want such a cap, combine the system-managed CHGJRN behavior with a system-provided trigger mechanism (influenced by total journal receiver size) to assure that a CHGJRN ensues on a regular basis. It is just that simple. Thereafter, the system monitors the journal receivers and uses size to manage CHGJRN behavior automatically for you.
In fact, there is even a matching journal receiver size threshold parameter: THRESHOLD, that you can set either on the CRTJRNRCV command or (beginning in V5R4) on the CHGJRN command. Normal intuition would lead you to believe that if you selected such a threshold, such as 1 GB, and instructed the system to serve as your housekeeper, that no journal receiver attached to this journal would ever exceed the designated size, right? Wrong! How can that be?
The answer again relates to zone “B” and how the space residing within that zone is managed, monitored, and tracked.
It turns out that the threshold enforcement mechanism normally monitors only what resides within zone “A” and generally ignores what resides within zone “B”. The obvious reason is that what resides within zone “A” is not going anywhere. It is permanent and persistent. However, what resides within zone “B” is transient: Here today and gone tomorrow. Hence, rather than exactly insisting on a CHGJRN operation as soon as the combined total size of zone “A” plus zone “B” exceeds the designated size threshold, i5/OS acts more conservatively and only concludes that your size threshold has been crossed if zone “A” is about to increase in size and the total space (the sum of the size of zone “A” plus zone “B”) at that instant exceeds your designated threshold.
That is, actions, which trigger an increase in the size purely of zone “B”, do not monitor the threshold. They are oblivious to it. Rather, it is only actions that trigger the enlargement of zone “A”. These actions take a peek at the threshold and realize that the total sum of space consumed by both zone “A” plus zone “B” now exceeds the designated threshold. Hence, increases in size affecting purely zone “B” can cause your total journal receiver size to creep above the threshold, and you do not know about it.
If we think back to our motel analogy, it is as if the tax assessor (think of the tax assessor as our size enforcer) shows up to determine your tax bill only when you enlarge the main motel (which would happen if you added a new wing), not when you merely assemble another kennel out back. Your motel is zone “A” and it is monitored, while your kennels represent zone “B” and they are largely ignored: out of sight, out of mind. The tax assessor generally only watches what you do to zone “A” but factors in zone “B” when you apply for a new zone “A” building permit.
Hence, you can (and probably will) occasionally see a journal receiver whose total size (the sum of zone “A” plus zone “B”) has exceeded your designated journal receiver threshold. This is a perfectly normal phenomena and more likely to ensue if you tend to have lots of large keyed logical files and/or SQL indexes built over your journaled physical files (because these indexes are the objects aggressively populating zone “B”). In fact, this phenomena is especially prevalent if the logical page size employed by your access paths is 64k or larger (as is often the case for indexes created through SQL).
If you tend to have lots of these “large-leaf” indexes and populate them aggressively, zone “B” fills at a torrid rate and keeps growing in size. Meanwhile, the size of zone “A” barely budges. The result: Total journal receiver size exceeds your threshold.
The asynchronous nature of responding to a threshold
In addition, there is yet another phenomenon, which makes it likely that you will occasionally see a journal receiver whose total size is somewhat larger than the designated size threshold, and that is the fact that the size threshold is monitored at the SLIC (microcode) level while the matching response action (attaching new journal receivers through the CHGJRN command) is performed at the i5/OS level.
When SLIC detects that the size threshold is about to be exceeded, it sends up an asynchronous notification (in the form of an event) to the i5/OS level. So you might have application job number 1 cranking away adding new database rows with each such row generating a new journal entry. When the last few bytes of the designated desired maximum size of zone “A” of your journal receiver (as evidenced by your threshold) have been populated, the SLIC layer of i5/OS (operating under application job number 1) sends a notification as an event (you can think of this as a message) to a queue being monitored by a background housekeeping job of i5/OS (generally, the QDBSRV01 job or one of its cousins).
This i5/OS housekeeping job might have lots of other things it has to do and eventually gets around to noticing the request to attach a new journal receiver. In the meantime, application job number 1 has continued to pour new journal entries into zone “A” (and maybe zone “B” as well!) of the journal receiver.
The net of this is while waiting for the background housekeeping job to respond, the total size of your journal receiver creeps above the designated size.
Here today, gone tomorrow: rapid shrinkage at detach time
There is one final mystery worthy of note: Detached journal receivers are often smaller in size than they were just seconds ago while attached. How can that be? Why would a journal receiver suddenly shrink in size merely because you set it aside and ceased populating it?
The explanation involves what happens during a CHGJRN operation. Not only does CHGJRN attach a new, fresh, empty journal receiver and set aside the former receiver, but it also tidies things up a bit. It reaches back to the detached journal receiver and trims excess space, thereby aggressively pruning both zone “A” and zone “B”. Unneeded transient entries residing within zone “B” might well be discarded, and unpopulated growth space (I like to think of it as extra room) at the end of zone “A” could be as well. The result is that a journal receiver, whose total combined size (zone “A” plus zone “B”) exceeded the designated size threshold a moment ago, but now appears to exceed it no longer.
Conclusion
Well, there you have it: Mystery solved! Yes, journal receivers can (and do) exceed their designated size threshold. And yes, in the past, it was not that your eyes were going bad or that your memory was failing, and no, you were not hallucinating. You really did see a journal receiver’s size shrink. Both phenomena are normal and expected behaviors.
Special Notices
The material included in this document is in DRAFT form and is provided 'as is' without warranty of any kind. IBM is not responsible for the accuracy or completeness of the material, and may update the document at any time. The final, published document may not include any, or all, of the material included herein. Client assumes all risks associated with Client's use of this document.