Home        The Boat        Photo Essay        Contact

Photography and Videography

Photography, the art of fixing an image for later viewing, has become one of the main drives of mankind. Photography is a form of communication, and we as humans are absolutely obsessed with communicating ideas to each other (or anything else we think might be listening). Being able to show someone else a representation of what we have ourselves seen is a powerful and compelling form of communication, and the relative ease of conveying a realistic image afforded by photography has lead to an ever increasing interest in the art.

Digital Sensors
File Storage

Film - Size, detail and color.

Early photography used huge format film sizes simply because the chemical processes were slow. Larger format sizes allowed more light to be captured and reduced exposure times down to fractions of a second that people could easily hold still for. As improvements were made in both the manufacture of film and in the chemical processes themselves exposure times got a whole lot shorter, and smaller format cameras worked much better. The old standard of 8 inch by 10 inch film however remained popular simply because the huge level of detail it afforded was desirable to many photographers.

The other factor in film size was enlargement. Going back to the late 1800's photographers were much more likely to have access to good lenses for the initial exposure than to have access to the huge lenses required for printing enlargements. In the absence of good enlargement lenses the direct prints from 8x10 film provided a marketable product that consistently provided good results.

By the 1920's widespread prevalence of professional print shops with good enlargement equipment meant that there was no longer much reason for any serious photographers to stick with direct prints, and the standard four by five inch film became nearly universally the largest format used. Smaller film formats were also used, but the dawn of color photography introduced new considerations for film format size as well.

Ultimately the only firm limitation on how good of photographs can be taken with small format cameras has to do with the physical size of the waves of light. Red colored visible light is 700 nanometers in wavelength, which means that an enlargement ratio of 120:1 could theoretically yield 300dpi resolution in a black and white print. This high level of detail has for the most part not been attainable. Most of the difficulty has centered around a wide spread refusal to acknowledge the fact that different colors of light are different sizes. The best example of this is black and white archival film that was used for storing pages of books, magazines, newspapers and other documents. The small roll film had an exposable area less than a half an inch wide but was able to store an entire news paper sheet across this width. This approximately 100:1 enlargement ratio was able to reproduce amazingly good fine detail in the black and white photographs and perfectly formed easily legible text characters at the smallest 10 point sizes sometimes used for captions on photographs. The secret of these high levels of detail on small film sizes was the use of high frequency ultraviolet light for the exposure of the high performance black and white film. In an industrial setting exposing film with ultraviolet light was easy to do. With a sealed enclosure high intensity ultraviolet light can be used for rapid exposure times, or longer exposure times can be used with a low intensity ultraviolet source to minimize damage to the source material.

For photography the small film sizes did not work out so well because the frequency of light available is inconsistent and generally of a much longer wavelength. Small format black and white film can be used outside where ultraviolet light is available, but the high intensity of other wavelengths of light present tends to interfere with maximum detail reproduction. Filters to isolate the ultraviolet light can help, but this complicates photography for a number of reasons. Indoors or at night a short wavelength flash unit can be used, but the high intensity ultraviolet light is not suitable for head on photographs of people. What this means is that even for black and white photography keeping enlargement ratios down to about 40:1 is absolutely necessary.

When color film was first introduced it tended to require extremely large formats because it was both slow and unable to capture fine detail. As color processes were perfected over the decades the performance of small film improved immensely to the point where 20:1 enlargement ratios were possible. A 24mm wide exposable film width was able to consistently make good eighteen inch wide prints and even larger prints looked pretty good if everything went just perfectly in the exposure, development and printing processes. This 24mm wide film was known as 35mm film for two by three aspect ratio cameras. Lens limitations however continued to make larger format cameras popular for demanding photographic applications.

Lenses - Size and adjustability.

The main problem with photographic lenses is that a high level of adjustability is required. A simple lens system with just a few individual lens elements can be made fairly distortion free for use at a single focus distance, a single aperture size and of course a fixed focal length. The problems begin to arise when one or more of these three basic lens system parameters need to be made adjustable. The parameter that nearly always needs to be adjustable is the focus distance. For anything other than studio photography it is absolutely necessary that the focus distance be adjustable over a fairly wide range of values. For overall flexibility adjustable aperture sizes are also nearly universally considered mandatory so that a range of exposure times can be used over a range of lighting conditions. The most difficult parameter to make adjustable is the focal length, and zoom lenses have only become popular with the development of very sophisticated lens systems with fairly large numbers of individual lens elements. Careful design of lens systems with about 10 or 20 individual lens elements can well control distortion and provide good image reproduction over a generous rang of focus distances, aperture sizes and focal lengths. All of those individual lens elements do however create a new problem. Refraction off of all of the surfaces messes up the final image, and any imperfections in the glass or in the surface finish are also much larger problems when there are so many pieces of glass that the light has to pass through. Imperfections in the glass can be reduced with more precise manufacturing processes, and refraction problems can be somewhat reduced with careful selection of materials with a more optimal index of refraction value. Ultimately though the only thing that really eliminates refraction problems is to use a single wavelength light source for black and white photography. Since a single wavelength light source is normally only practical in a studio setting this does little good, as a large number of lenses of limited adjustability are quite easy to use in a studio. For color photography refraction is always a problem under all conditions as reproduction of color inherently requires a range of frequencies of light to pass through the lens system.

Somewhat ironically the only real solution to these lens problems is larger format sizes. A larger format allows the pieces of glass to be larger, which dramatically reduces problems both with refraction and glass imperfections. It is easy to see that larger lens elements dwarf small imperfections in the glass. One of the really big advantages or larger format sizes is simply that keeping the front element clean enough for good results is much easier. Refraction problems diminish with larger format sizes just because there is more of everything to work with. It is not that larger format sizes do anything dramatic to eliminate refraction problems, but rather it is a case of the overall design of the lens system being easier because other problems that would tend to crop up can much more easily be controlled. This is because refraction is about angles as opposed to being tied into size alone.

Digital Sensors - Sensitivity and detail.

With the introduction of high resolution digital sensors the format size versus detail and color relationships became relevant in new ways. The same basic limitations of the physical size of waves of light and lack of adjustability of lens systems still apply to digital photography, but the elimination of the chemical processes of exposure, development and printing means that perfect results are universally attainable. Color film with an exposable width of just 6mm was quite commonly used for many decades, and reasonably good results up to 8x10 print sizes was attainable under some conditions. With digital sensors the performance of extremely small format cameras improved so dramatically that the physical size of the light has become the main limitation for color photography.

The standard shirt pocket camera format has for many decades been a 4.5mm by 6mm digital sensor, which is under some conditions good for fairly nice 6 inch by 8 inch prints and 8x10 prints for hanging on walls are also sometimes attainable. The level of detail attainable in these print sizes is however severely limited by the extremely small format. The limitation is simply that red light is no less than 700 nanometers in size. If each of the three individual light sensors required for color imaging is exactly 700 nanometers across then there is only room for 8600 of them across a 6mm wide sensor for a total of 2860 lines of horizontal resolution. For a three by four aspect ratio compact camera this means a theoretical maximum resolution of nine megapixels. Color photography results at this capture site size does not however yield all that good of results. In order to get good color fidelity and pleasing reproduction of red, orange, yellow and brown tones considerably larger capture site sizes are in fact required.

This layout with each of the three individual color photo sensors being three times as long as they are wide is what most people are familiar with having looked closely at the layout of individual color blocks on traditional electronic display screens, but it is not the only way that the individual photo sensors can be laid out on a digital sensor. If the individual photo sensors are round in shape then they can be crammed more tightly onto a sensor for a theoretical maximum of 3840 lines of horizontal resolution for a 6mm wide color digital sensor. This minimum size individual photo sensor just does not work though, regardless of what shape it is. If the individual photo sensors are exactly the same distance across as the wavelength of the light then an individual photon must "land" squarely in the middle of the photo sensor to generate an electric signal. To get any kind of reasonable low light performance the individual photo sensors have to be large enough that a large portion of the photons coming through the lens system are actually registered as electric signals.

There are actually two different ways that the individual photo sensors on a digital sensor can be laid out. If equal size individual photo sensors are used then the layout is octagonal or round shaped individual photo sensors where each three individual photo sensors makes up one roughly triangular shaped block. This octagonal and triangular arrangement yields the highest theoretical resolution for a small color sensor, but very ironically this layout is better suited to larger size sensors where the individual photo sites are much larger. For cramming the highest resolution possible onto a small sensor getting one of the three individual photo sensors larger than the other two is of great importance for generating good detain in the red, orange and brown tones. Unequal size individual photo sensors requires an uneven sort of layout, and the theoretical maximum resolution works out to be considerably lower.

It is interesting to note that a 6mm wide digital sensor for black and white photography would have a theoretical maximum resolution of 8600 lines of horizontal resolution. At this photo sensor size though red light would interfere with the maximum attainable resolution. Just as with high performance black and white film for archival purposes exposure would have to be done with filters or under artificial ultraviolet light if the full 8600 lines of horizontal resolution was to be obtained.

At this point it is absolutely necessary to go into the biggest lie in the modern world, and that is the advertised resolution of digital cameras. The reality is that all digital cameras available for more than a decade have been what I like to call "two into three interpolators". That is the output files are interpolated up to a higher resolution where each two lines of horizontal and vertical resolution on the sensor is converted up to three lines of horizontal or vertical resolution in the output file. This two into three interpolation yields an output file with a resolution of two and a quarter times the pixel count of the sensor. This means that the 9mp of 4.5mm by 6mm digital sensor capable of doing color photography would be marketed as a 20mp digital camera.

Such a camera has in fact been built by Sony for the past several years, and can be had at many stores for about $120 to $150. These cheap toy cameras have fallen out of favor because they are nearly totally useless for real color photography. The technical achievement of the 20mp output from a 6mm sensor packaged in an extremely thin shirt pocket camera with a zoom lens is however no trifling mater and should not be so quickly dismissed as insignificant. Yes they are useless toy cameras, but what they are capable of doing is mind boggling. Perhaps a useless toy in the hands of children and casual photographers but they are also quite intriguing for anyone who has a deep technical understanding of photography. These toy cameras should come with a warning label indicating that they are not capable of true color photography under any conditions, but they are fun to play with because they can so easily make quite high resolution images under all sorts of conditions while fitting into a shirt pocket instead of a bulky camera bag.

So this of course raises the question as to how large the photo sites in fact need to be for good color fidelity. Many of the 6mm sensor compact cameras were able to do fairly good color fidelity under all conditions with advertised resolutions of about 7 to 9mp, or 3 to 4mp actual sensor resolution. This seems like a huge difference between a 7mp camera and a 20mp camera, and the difference in detail is in fact quite huge. What it comes down to is that reliably good color fidelity even in somewhat reduced lighting conditions requires much larger individual photo sites than the theoretical maximum. A 6mm sensor camera with a 7mp advertised resolution and a 3mp actual sensor resolution (2000 lines of horizontal resolution) has individual light sensors that are only 1.7 micrometers across. That is really not much bigger than the 700 nanometer individual light sensors of the theoretical maximum resolution for color photography (3840 lines of horizontal resolution for a 6mm wide sensor). One way that these high resolution compact cameras are made to function better is with the use of larger sensors for red with the other two sensors being somewhat smaller. The little bit of extra size on the red sensors goes a long way to getting good color fidelity under reduced lighting conditions. Another trick used is software corrections that guess at the color of red areas of the photograph. This software guessing can be made to work well enough to get passable color fidelity for blue, green and yellow tones, but large areas of red color still come out looking quite horrible on the 20mp advertised resolution 6mm sensors. Ultimately there is no substitute for larger photo sites.

The digital cameras that do work for large prints at high resolution are the standard 24mm wide APS-C sized sensors that have been standard in real digital cameras all the way back to the dawn of digital photography in the late 20th century. It is all about basic proportionality. If a 6mm wide sensor is good for 6x8 prints and can sometimes do 8x10 prints then a 24mm sensor is good for 24 inch by 36 inch prints and somewhat larger sizes are possible with really good lenses.

On the 24mm sensor cameras the main limitation tends to be the cheap, light and mediocre performing lenses that have been prevalent for many decades, but very low resolution sensors also severely cut into overall performance. The original Canon 1D APS-C sized digital camera was advertised as a 4mp camera, and it in fact had a 4mp sensor. These cameras required later interpolation of the output file for large print sizes because the resolution was very low and quite choppy "pixilated" prints resulted from direct printing of the output files. Other early digital cameras were advertised at 6mp, 8mp and 10mp and had actual sensor resolutions of 2.7mp 3.6mp and 4.4mp respectively. Particularly the 6mp two into three interpolators yielded quite poor results because the resolution was way too low for prints larger than about 6x8. For this reason the standard digital camera became the 10mp two into three interpolator, and these cameras worked quite well even though print sizes of larger than 8x12 were still not a good idea.

The desire to get performance back up to what the 35mm film cameras had done started a steady climb in sensor resolutions. This escalation of sensor resolutions quite quickly yielded 14mp cameras in 2008, and within a few years 16mp and 18mp cameras were common. These increased resolutions allowed the 11x17 prints that most photographers desired, but was by no means the limit of development of the 24mm sensor standard size. At the same time the larger 35mm wide "full frame" digital cameras were also available at 18mp and 24mp advertised resolutions for the discerning photographer with deep pockets. The larger, heavier, more expensive and generally somewhat higher quality lenses for the full frame cameras did yield better results, but better lenses specifically for the 24mm sensor cameras also became available. As it turns out 24mp advertised resolution (10.7mp sensor resolution) on a 16mm by 24mm camera works quite well. At this resolution lens limitations are of course a huge problem, but when a good lens for the desired conditions is available the output is better than was ever possible with 35mm film cameras. The digital sensors are in fact able to be made to work somewhat better than the best color film that was ever produced.

File Storage - Compression and editing.

Most digital files are stored with a "bit depth" of 24 bits per pixel, which is called 24 bit color. A 10.7 megapixle image with 24 bit color stored with no image compression is 32 megabytes in size. That is a rather large file, unnecessarily large really. Much smaller file sizes can be obtained with vector compression of the image. The basic idea of vector compression is that areas of smooth color gradients can be represented with just a fraction of the data that would be required to store the color of each individual pixel. How much a 10.7 megapixle image can be compressed without losing detail or image quality depends on the level of detail in the image. With essentially no loss of detail or loss of image quality what so ever vector compression down to about 15 megabytes is possible. In practice the best lenses available will not support anywhere near a perfect image at that resolution, and much smaller file sizes are possible. Just how much the real life images can be compressed depends on many things, paramount among them being how much extra file size is desired so that the image can be opened and recompressed one or two times during printing without large losses of detail and image quality.

Storing the file with two into three interpolation for a 24mp interpolated up image means that the compressed file has to be slightly larger, but the difference is really not all that dramatic. A 15mb file for a 10.7 megapixle image stored at the interpolated up 24mp size leaves quite a bit of extra information for perfect color fidelity and near perfect reproduction of detail if the image has to be opened and recompressed one or two times. Much smaller file sizes are possible for final compression if no additional opening and compression of the file will be required. About five or eight megabytes will usually do the trick for keeping the loss of detail to a low level on a 10.7mp (24mb interpolated up) image. Because lens limitations are so severe many mediocre but still usable photographs that will print out well at the 8x10 size can be compressed all the way down to one or two megabytes with only rather slight loss of detail and image quality. For this maximum level of compression it is quite useful to strip the image back down to the native resolution of the sensor.

A common misconception is that heavily compressed files have poor color fidelity, but this really is not the case. Going down to 16 bit color totally ruins all photographs and saves only one third of the file size. In reality the millions of colors supported by 24 bit color is more than is required for perfect color fidelity, and it is possible to go down to 20 bit color with good results. Saving 20% on the file size though is mostly insignificant, and there is essentially no reason to use less than 24 bit color under any circumstances. The point here is that 20 or 24 bit color is the critical threshold for bit depth. Less bit depth than this is not possible and higher bit depth only adds to the file size without increasing color fidelity or image quality.

Vector compression of images is a one way process, so any editing of the image itself requires that the file be opened and then recompressed with some loss of detail and quality. There are however some things that should be adjustable on images without requiring recompression. These parameters that should be adjustable are overall brightness and the overall color. Sharpening and contrast can of course also be applied at the time of opening of the file, but this requires much more processing power in the viewing device than storing the sharpening and contrast settings as part of the compressed image. The use of more processing power on the viewing end can also allow more sophisticated brightness corrections such as lightening the dark areas and/or darkening the light areas to be applied without recompression of the image. The way that using processing power at the viewing end can work is for the file to be opened and modified with sharpening, contrast and selective darkening and lightening but not recompressed. The parameters for the sharpening, contrast adjustment and selective darkening or lightening are instead stored as external parameters. When the image is later opened these same external parameters can then automatically be applied to get the same finished image that was seen when the external parameters were created. For purposes of file storage for later printing there is not much of a problem with extra processing power requirement for applying these external parameters because the delay is extremely short compared to the time required for the physical printing. The key to getting an external parameter system to work well of course relies on being able to chose whether to apply those external parameters or not. Opting out of applying the external parameters allows fast file opening times just to get a quick look at the original image.

Video - Frame Rate, compression and size.

Video requires large amounts of information storage, but there are substantial amounts of compression possible for digital video. Film movies required at least 24 frames be exposed each second for realistic motion at sedate speeds of action. To reproduce fast action 60 frames per second was used, and it is no coincidence that alternating current electricity operates at 60 cycles per second (60Hz). Computer monitors and most television sets have always operated at a refresh rate of 60Hz. It is interesting and important to note that the human visual system also operates with a refresh rate, but the refresh rate for humans is not constant. A typical refresh rate for a person at rest and casually observing moving objects is 100Hz, but somewhat higher refresh rates are also possible.

Even though human visual systems typically operate well above 60Hz it still remains true that quite good electronic video reproduction of fast action can be attained at 60Hz. For film movies 60 frames per second meant that a foot of 35mm film (24mm exposable width) played for only a third of a second. That is four miles of film for a two hour feature length movie. A whole big bulky reel of film that could easily become an onerous task to load into the projecting machine.

What digital video requires is rather fast processing equipment (a computer) for playback and sophisticated compression of the files. Early television was of course not computerized, but that early black and white television was not true digital video even though it operated at a 60Hz refresh rate. The entire screen refreshed 60 times per second, but the scanning of each line was analogue. Just over 500 lines 60 times per second for an image with 480 lines of vertical resolution. The horizontal resolution was however not divided up into discrete segments. Instead the radio signal was modulated as the light intensity changed along each line. With the switch to color television in the mid to late 1950's this analogue reproduction was eliminated, and color television has always been based on digital images. The reason that broadcast television was considered an analogue system up to the switch over to digital television in 2007 was that the radio transmissions that carried the color picture and sound information were still very similar to the radio transmissions that had been used for true analogue black and white television. The new digital television system introduced in 2007 is entirely computerized, meaning that the transmission of information is in the form of compressed data that can be stored on any digital information storage system. And it is the ability to compress both the audio and the video that is so important for digital television. The audio portion is however insignificantly small, less than one percent of the data required for the video.

The main concept in video compression is the frame to frame compression. The individual images are of course compressed with vector image compression similar to that used for still photographs, but huge additional reductions in file size are attained with frame to frame compression. The basic idea of frame to frame compression is that portions of the image that do not change from frame to frame do not need to be stored over and over again. For most types of real life video this represents an absolutely huge reduction in file size. Think of a person talking on camera while the background remains the same for many seconds in a row. During the entire time that the background remains constant only the movement of the person talking has to be represented in the bit rate of the video. This is how an average bit rate of 19 megabits per second can reproduce 1080 by 1920 video at a 60Hz refresh rate. When the background does not change the bit rate drops down to a much lower value, and when the entire frame changes with fast action the bit rate skyrockets up to much higher levels.

Of course frame to frame compression is not as simple as just using fixed backgrounds. There are all sorts of sophisticated processing techniques that can be used for frame to frame compression. The goal of frame to frame compression is to dramatically reduce the total file size of a segment of video without interfering with the observed smoothness and lifelike nature of the video. Even during segments of panning and zooming portions of the frame that change only because of the movement of the camera can be heavily compressed. This is tricky to attain when the video is compressed by the camera during the shooting of a video clip, but high levels of frame to frame compression are nevertheless possible. Much more powerful and seamless fame to frame compression techniques can be applied when the final compression is done later in the production process. The key to good in camera frame to frame compression is a large buffer to store many seconds of sensor output before it is compressed and written to the storage media, as well as quite a bit of processing power. Once a large buffer and a large amount of processing power are available software can be written that analyses the video over a rather long period of many seconds to decide which compression algorithms to apply. And of course as with any computer system that is to be compact, lightweight and energy efficient hardware that is designed for a particular type of software pays big dividends.

The standard 19 megabit per second average data rate for 1080x1920 60Hz video means that a half hour of video fits on a DVD. When movies were first available in HD on DVD around 2004 it was typically two double sided DVDs for a movie and some special features. This would also have fit on a single double sided dual layer disk, but dual layer disks were not popular at that time even though the standard had existed at least as far back as the early 1990's. When Blue Ray took over the home theater industry dual layer disks became universal, but an entire movie being crammed onto just one single sided disk also became standard practice. A whole feature length movie does not however fit on just one single sided disk. If the special features are on a second disk then a short 90 minute feature length movie can do pretty well on just one single sided dual layer disk, but it is then a stretch to call the resulting product HD 1080x1920. The sales tool of calling the standard disk "Blue Ray" has however been quite successful because the name implies the concept of using a shorter wavelength laser to read data so that the data can be crammed more tightly on the same size disk. The thing is though that the cramming of more data onto a standard 12cm laser disk was already done once, and that was when audio CD's of 650MB (8 bit storage) were beefed up to become DVD's of 4.5GB (8 bit storage). Theoretically 4.5GB on a single sided single layer 12cm optical disk should be possible with a 500 nanometer wavelength laser, but it is entirely unclear what frequency of electromagnetic radiation is actually required in practice to attain this density of data. Going down to 400 nanometer blue color light would theoretically allow 7.8GB per single sided single layer disk, which is a very substantial increase in data density for a modest increase in the frequency of the light. There also seems to be the possibility that a higher frequency laser could allow triple layer disks instead of just dual layer disks. If the frequency of the laser was increased by one and a half times so that three layers could fit in the space currently required for two layers then the data density of a standard size 12cm disk would increase by over 300%. That would be more than 25GB on a single sided disk, which would allow nearly three hours of play time at the standard 19 megabit per second average data rate for 1080x1920 HD video.

Back to Index to Technical Articles

©2014 Michael Traum