# xpdf project in PDFedit

Pdfedit project uses &xpdf_link_index; code for low level pdf content operations, such as pdf object parsing (with quite good matching to the Adobe pdf specification ver. 1.6. ), indirect object resolving, generation of page and text output devices, streams decoding and so on. We have tried to reuse the most of functionality which is not somehow directly related to xpdf application special logic.

To prevent from errors comming from xpdf code as well as to be less depended on xpdf in whole project, this is used just in few places (namely CXref, XRefWriter, CObject, CPdf and CPage classes) and rest of our pdfedit code uses just our classes and special objects. This means that substitution of xpdf code by something different is possible with changes on concrete places and rest of the code doesn't know about that. Project currently uses xpdf in 3.01 version.

## Changes needed for code reuse

Code of XPDF project couldn't have been reused without modifications because it is not prepared for making changes to its objects very well (code assumes data reading and using them - not to use and change them). Our code changes can be divided to 3 cathegories:

- Syntactic - these are changes related to functions/methods signature (const modificators, private methods changed to protected, new parameters, non virtual methods are changed to virtual in some classes).
- New features - these are changes which produce new functionality required by our project (e. g. clone support for Object and all used values inside).
- Design - these are changes in xpdf object hierarchy or meaning of some components so they better fit our usage.

For more information see following detailed description.

### Object.h and Object.cc

`Object` class is used as value keeper in whole xpdf code. Design is not very good, because all value types are stored in this one object (even more in one union in Object class) and real value type is then identified by enumeration Object type (returned by getType method). We consider this design not very good because this doesn't prevent user from bad usage and different value type can be obtained than really stored (no internal checking is done and so on). Nevertheless this behavior is kept, because change would require whole xpdf code reorganization. We have focused just to Syntactic and new features changes here.

Xpdf code uses kind of optimization for objects copying and so complex values (such as dictionaries or arrays) are not copied by copy method at all and reference counting is used instead. Our Object usage (used primary in CXref class) requires deep copying and so cloning support is neccessary. We have added new clone method:

    	Object::clone()const;

which creates new Object instance with deep copy of value held by original.

Clone is considered to be same typed object as original with equal value (deep copy). Following contracts has to be fullfiled (if there is not special limitation which is not possible/required to keep - in such situation this must be explicitly described).

    	Object * clone=original->clone();
    	clone!=original;				// instances must be different
    	clone->getType()==original->getType();	// type must be same
    	equals(clone, original)==true;		// content is same
    	equals(change(clone), original)==false;	// change to clone or orginal doesn't influnce second one
    	equals(clone, change(original))=false;
    	destroy(clone) && exists(original) == true;	// destroying clone or original keeps seconf one existing
    	destroy(original) && exists(clone) == true;

Returned Object instance is change independant on original and so they don't influence each other when one is changed. Cloning of complex value types are delegated directly to specialized clone method implemented on such type.

Syntactic changes simply corrects parameters modificators and all methods with pointer parameters which can have const modificators are changed to have it. This change is just cosmetic and should prevent from bad xpdf code usage.

### XRef.h and XRef.cc

XRref class is responsible for pdf objects fetching from stream. Pdf defines so called . This objects are identified by their object and generic numbers. XRef keeps and maintains which contains mapping from indirect object identifiers to file offset where object is stored. Internally uses Parser and Lexer classes to parse file stream content to Object.

#### Design changes

In first XRef had to be prepared for transparent wrapper usage (see , so all public methods were changed to virtual and private elements to protected (to enable access to and manipulation with them in inheritance hierarchy). XRef class is then wrapped by CXref (see ) class and rest of xpdf code doesn't know difference and can be used without changes (with respect to XRef usage).

CXref reopen This method is essential for revision changing, because it forces XRef supertype to do the clean up for current revision and init all internals for revision staring at given position (so all document content before this position). functionality requires correct chaning of XRef internal state (which includes entries array reinitialization, trailer creation and so on). This everything was done in construtor in original implementation. Clean up was done in destructor. We have added new protected

    	void initInternals(Guint pos);
    	void destroyInternals();

methods, which use same code as original one but separated to enable such internal state change anytime during XRef instance's life.

#### New features

XRefWriter (see ) (descendant of CXref class which inherits directly from XRef) needs to know where it is safe to put data not to destroy original document data when changes are written to the document (as an ). To enable this, XRef has new

    	Guint eofPos;

field which contains position of %%EOF marker or end of document. Value is set in constructor because it has to be found out anyway and XRefWriter doesn't have to this work again.

XRef class didn't provide information whether pdf reference (object and generation number pair) is known Information whether there is indirect object with this reference. and so it wasn't possible to find out whether object value is null object or it is not present. To solve this problem, we have added new public

    	virtual RefState knowsRef(Ref ref);

method which returns state of given reference. State is integer value with predefined constants which may hold:

- UNUSED_REF - if there is no indirect object with given reference.
- RESERVED_REF - if reference is reserved to be used, but no indirect object is registered yet. This state is used by CXref class to mark that reference is planned to be used and we are just waiting for some object to be used for it.
- INITIALIZED_REF - if indirect object with given reference exists. This objects are considered when number of objects is required.

CXref and XRefWriter descendants override this method to reflect object added/reserved by their interface and additional logic (e. g. current revision and so on).

#### Implementation changes

XRef's getNumObjects returned the size of allocated entries array. This is not very clean, because entries array contains also free and unused entries. Even more array is allocated by blocks and so there are more entries than real objects. This method is not used in xpdf code at all, so it could be reimplemented to return just really used objects (those with state INITIALIZED_REF).

### Array.h and Array.cc

Array class represents pdf array object. It is one of complex value type. It may contain number of Object instances. To enable Object cloning, new

    	Array * clone()const;

is implemented. It returns new Array instance with same number of elements where each one (Object instance) is cloned (by Object::clone() method).

### Dict.h and Dict.cc

Dict class represents pdf dictionary data type. It is one of complex value type holding association of objects with their names (key, value pair where key is name object and value is Object instance).

#### Design changes

DictEntry used as entry (key, value pair association) kept value (Object instance) as normal instance. This was changed to pointer to instance to enable simpler value updating.

Original code didn't use const modificator for key (char * typed) parameter and so it wasn't clear whether it uses given value and stores it (and so parameter can't be deallocated after method returns) or just use it to get information (so it can be deallocated). This could possibly lead to memory leaks or worse to duplicate deallocation od same memory. To solve this potential problems, all methods which don't store key have `const char * key` parameter.

#### New features

Dict as complex object stored in general Object data keeper has to support cloning, so new

    	Dict * clone()const;

is added. It returns new Dict instance with same number of entries where each entry is deep copied - name string and associated object (Object instance) is cloned (by Object::clone() method).

New method for simpler updating value has been added:

    	Object * update(char * key, Object * val);

This method will add new entry if no such entry is in dictionary or replaces an old by given value and original is returned.

Original implementation didn't contain any method for entry removing and so new

    	Object * del(const char * key);

has been added. This will remove entry with given key and returns associated value.

### Stream.h and Stream.cc

Xpdf code defines Stream hierarchy to describe pdf stream objects. Streams (as pdf data types) define container of pdf objects. This container is associated with dictionary which contains information about its length and filters which were used for stream encoding. XRef class reads data from stream or object is based on stream.

#### Design changes

Stream is base class for both normal streams represented by BaseStream (Stream descendant) base class and FilterStream (also direct Stream descendant) base class used for all filered streams. This stream objects hierarchy is strictly specialized for reading and can't be used for making changes to stream data. CXref and XRefWriter however needs to make transparent modifications to stream with pdf content (so that xpdf code using Streams doesn't have to be changed very much). This is the reason for some changes in Stream hierarchy design.

Problem with stream modification is solved by new abstract class (base class for all specialized stream modificators) StreamWriter. This defines interface for stream writing (in same way as Stream defines operations for reading). However, implementation of concrete writer requires (such as FileStreamWriter) multiple inheritance, because it needs interface from StreamWriter and also access to concrete BaseStream (in FileStreamWriter it is FileStream) fields. So original inheritance of all direct descendants of Stream and BaseStream had to be changed to virtual (to prevent ambiguity). This model enables transparent usage of StreamWriter as Stream typed instances in xpdf code and as StreamWriter typed instances in our higher level classes (like FileStreamWriter) in pdfedit code for writing.

FilterStream hierarchy is untouched in design way, because our project doesn't change filtered streams directly. It works just with base stream, because FilterStream hierarchy is hard to be reused for encoding. So just decode functionality is used.

Stream hierarchy

#### New features

Stream object as one of complext value data type which is stored in Object (as all other data types) has to to provide cloning support. We have added abstract

    	virtual Stream * clone()=0;

method in Stream base class. Each specific stream implementator has to provide its clone implementation. No default implementation is written in Stream directly to force all specific filters provide one. If any of filters is not able to create clone, this method should return NULL. This should not happen, however clone implementation has to be aware of it (and has to check whether filter stream has cloned correctly).

`FileStream` is stream which reads data directly from FILE stream and so cloning has to copy all data (from stream start to the end - if stream is limited, then just first length bytes) somewhere else. Creation of new file, just for temporarily created clone is not very effective and may produce several problems (not enough free place, creation and removing of temporary file, etc.). We have solved this problem by creating `MemStream` with buffer containing same data as FileStream. This brakes contract of clone meaning a bit, because cloned stream is not precisely the same as original, because it is represented by another Stream class. Nevertheless it keeps the most important contract, that user of Stream interface doesn't know the difference and clone and original don't affect each other when one is changed.

`MemStream` represent stream stored in buffer in the memory. So cloning is straightforward and just buffer is copied for new MemStream. All other attributes are set according copied buffer. Buffer copying starts from start field position and lenght bytes are copied. So final MemStream will contain just data used in original one. Finaly needFree field is always set to true, because we have allocated new buffer for clone and so it has to be deallocated when MemStream is destroyed. Note that MemStream may contain buffer but this buffer is not used whole. This is controled by start field, which says where is the first byte of this concrete stream. This is original behaviour of MemStream's implementation and the reason is that in many situation MemStream is used to manipulate with buffers without need to copy the buffer. So xpdf code just reuses some buffer and says that new MemStream starts from here and its lenght is such value. Such MemStream is marked not to be deallocated in destructor by needFree field.

`EmbedStream` is special stream which is stored in some other stream. Clonig is also very simple, because this stream just holds one Stream pointer field and some attributes which doesn't change during instance life cycle. Cloning is then just delegation to cloning of stream field and creating new EmbedStream with cloned value and same parameters which were used for given instance.

`FilterStream` stream branch represents different types of filters which can be used to encode stream data (Pdf specification describes which filters can be used). Hierarchy and design of filters follows and each filter works with underlaying stream (which is stored as pointer) field which is typed as Stream (so it can be either stream with data - `MemStream`, `FileStream` or `EmbedStream` - or another filter stream).

FileStream is cloned in similar way as EmbedStream. Each filter implemenetator holds Stream pointer (as already mentioned). This is cloned by clone method (defined in super type). When underlaying stream is cloned (and clone method returned wih non NULL value - which means that this stream supports clonning), current stream creates new same filter stream instance with same configuration parameters (these are usually parameters which were given to it in constructor - but when such attributes can change in time they have to be stored somewhere else in constructor specially for this purpose). As an expample, FlatStream stream can't reconstruct original parameters (uses parameters for StreamPredictor initialization in constructor but doesn't store them to internal fields). We have added additional data structures to store such parameters. This kind of work around enables to create filter with same attributes. General implementation for all filter streams is as follows:

    	// clones underlying stream
    	Stream * cloneStream=str->clone();
    	if(!cloneStream)
    	// if underlying stream doesn't support cloning, it will fail too
    	return NULL;

    	// creates same typed filter stream with same parameters and cloned
    	// stream
    	return new ...(cloneStream[, specific_parameters]);

As mentioned above, some filters are not able to reconstruct parameters given them as constructor parameters and so it is hard to reconstruct same filter. Specially all filters which holds StreamPredictor field has additional field with PredictorContext (added by us):

    	struct PredictorContext
    	{
    	  int predictor;
    	  int width;
    	  int comps;
    	  int bits;
    	};

where all parameters needed for StreamPredictor creation are stored. This structure is initialized in constructor and never changed. It is just used for cloning.

### Parser.h and Stream.cc

Xpdf reads pdf files using two level mechanism. The lowest level, called Lexer, decodes streams if necessary and reads byte after byte. The second level, called Parser, reads from Lexer byte after byte until one complete object is read. This can be applied recursively. Then it initializes xpdf Object class with type and data of the read object. Parser object can read objects either from a single stream or from an array of streams (simply reading one stream after another could result in incomplete objects returned at the end of a stream). Parser is used to parse all objects including content stream tags.

xpdf Parser class

xpdf Parser class

#### New features

Content streams can consist of more streams. Decoding and then concatenating of these streams must form a valid content stream. The problem is, that the content stream can be split into streams at almost arbitrary place.

Added feature which was missing in the Parser object is the information when it has finished reading one stream and started reading another. Added method

    	// End of actual stream
    	bool eofOfActualStream () const { return (1 == endOfActStream); }

where endOfActStream is new variable indicating how many objects have been buffered from current stream.

Content streams can consist of many small valid content streams. When splitted correctly, user can easily delete/add small content streams. Changes made by PdfEditor can be considered as such small valid content streams. After saving our changes we want to see these changes separately to existing content streams. This new feature is used to split many streams (which create one page content stream) to many small content streams.

Because of the object buffering done by Parser the new feature had to be implemented specially this way.