This is G o o g l e's cache of http://www.nikonians.org/dcforum/DCForumID86/5745.html as retrieved on Oct 18, 2005 15:29:36 GMT.
G o o g l e's cache is the snapshot that we took of the page as we crawled the web.
The page may have changed since that time. Click here for the current page without highlighting.
This cached page may reference images which are no longer available. Click here for the cached text only.
To link to or bookmark this page, use the following url: http://www.google.com/search?q=cache:Mho36QRqmMIJ:www.nikonians.org/dcforum/DCForumID86/5745.html+site:www.nikonians.org+bclaff&hl=en


Google is neither affiliated with the authors of this page nor responsible for its content.
These search terms have been highlighted: bclaff 


Discussions @ Nikonians

Subject: "The Naked Truth (Raw Facts)" Archived thread - Read only
 
  Previous Topic | Next Topic
Printer-friendly copy    
Conferences NIKON PRODUCTS FORUMS D70 Users Group Topic #5745
Reading Topic #5745
bclaff
Member since 26-Oct-04
1883 posts
10-May-05, 03:19 AM (GMT+2)
Click to EMail bclaff Click to send private message to bclaff Click to view user profileClick to add this user to your buddy list  
"The Naked Truth (Raw Facts)"
 
   LAST EDITED ON 10-May-05 AT 09:06 PM (GMT 2) by tfhoey (moderator)
 
My intention is to present technical facts about certain information that is stored in the files written by our Nikon DSLRs.

Nikon DSLRs write Jpg and/or Nef files.

The Nef file is a Tagged Image File Format (TIFF) file.
TIFF files have variable sized blocks of data each marked with a tag.
There are a number of standardized TIFF tags, among these are the EXchangeable Image File (EXIF) tags.
One of the EXIF tags is the Makernote, intended for manufacturer specific information.
Nikon places additional information about the image in the Makernote.
Nikon also places standard information in normal TIFF tags.
(An annoying exception is the ISO).
The Nikon Makernote is essentially a TIFF file embedded in a TIFF file.

The Jpg file also has variable sized blocks marked with a "name".
The Nikon Jpg has an "APP1" EXIF block that contains the same information as the Nef Makernote.
The Makernote information is not unique to the Nef file.

There are in excess of 50 tags in the Nikon Makernote.
The Nikon Makernote tags are not publicly documented.
The layout of the data within the tags is not publicly documented.
Some Nikon Makernote tags and layout are obvious by simple inspection of the file.
Some additional information has been reverse engineered and is publicly available.
The camera model is not stored in the Makernote.
Complex tags start with a 4 byte version so they can be revised between camera models.

The D70 'serial number' is tag xA0, the D2X 'serial number' is tag x1D.
(The leading 'x' indicates a hexadecimal number)
The 'serial number' in an internal one and does not match what is stamped on the body.
(Sure about D70, not 100% sure about D2x)
The shutter counter for both is tag xA7.

Starting with the introduction of the D2X Nikon chose to encrypt certain tags.
Makernote tags x91, x97, and x98 were obfuscated.
On the D2X this is over 1,200 bytes of information.
The 'As Shot' White Balance values are 8 bytes in the x97 tag.
Many other useful data has also been encrypted.

Respectfully,
Bill

Visit me, info and galleries at: Nikon Photos and Information


  Printer-friendly page | Top

  Subject     Author     Message Date     ID  
The Naked Truth (Raw Facts) [View All] bclaff 10-May-05 TOP
  RE: The Naked Truth (Raw Facts) AQS1974 10-May-05 1
     RE: The Naked Truth (Raw Facts) bclaff 10-May-05 2
         RE: The Naked Truth (Raw Facts) AlanC 10-May-05 12
  RE: The Naked Truth (Raw Facts) baliphile 10-May-05 3
     RE: The Naked Truth (Raw Facts) bclaff 10-May-05 4
     RE: The Naked Truth (Raw Facts) bclaff 10-May-05 5
         RE: The Naked Truth (Raw Facts) baliphile 10-May-05 6
             RE: The Naked Truth (Raw Facts) bclaff 10-May-05 7
                 RE: The Naked Truth (Raw Facts) baliphile 10-May-05 8
                     RE: The Naked Truth (Raw Facts) bclaff 10-May-05 9
                         RE: The Naked Truth (Raw Facts) baliphile 10-May-05 10
                             RE: The Naked Truth (Raw Facts) bclaff 10-May-05 14
                                 RE: The Naked Truth (Raw Facts) baliphile 11-May-05 21
                                 RE: The Naked Truth (Raw Facts) Kerry Pierce 12-May-05 24
                                     RE: The Naked Truth (Raw Facts) bclaff 12-May-05 25
                                         RE: The Naked Truth (Raw Facts) Kerry Pierce 12-May-05 27
  RE: The Naked Truth (Raw Facts) RDW 10-May-05 11
     RE: The Naked Truth (Raw Facts) bclaff 10-May-05 13
  RE: The Naked Truth (Raw Facts) sfbillm 10-May-05 15
     RE: The Naked Truth (Raw Facts) bclaff 10-May-05 16
         RE: The Naked Truth (Raw Facts) sfbillm 11-May-05 22
             RE: The Naked Truth (Raw Facts) bclaff 11-May-05 23
  RE: The Naked Truth (Raw Facts) avm247moderator 10-May-05 17
     RE: The Naked Truth (Raw Facts) bclaff 10-May-05 18
         RE: The Naked Truth (Raw Facts) avm247moderator 10-May-05 19
             RE: The Naked Truth (Raw Facts) bclaff 10-May-05 20
  RE: The Naked Truth (Raw Facts) bclaff 12-May-05 26
     RE: The Naked Truth (Raw Facts) jbloom 12-May-05 28
         RE: The Naked Truth (Raw Facts) bclaff 12-May-05 29

Home | Conferences | Forums | Topics | Previous Topic | Next Topic
AQS1974
Member since 15-Feb-05
321 posts
10-May-05, 04:04 AM (GMT+2)
Click to EMail AQS1974 Click to send private message to AQS1974 Click to view user profileClick to add this user to your buddy list  
1. "RE: The Naked Truth (Raw Facts)"
In response to message #0
 
   Cool,

I never realized that a NEF is a type of TIFF.

Are D70 NEF files 12 bit?

A NEF file is concidered RAW data, correct? Would it be true to say that a RAW image is NOT a bitmap?

What about if I save a file I have been working on in PhotoShop as a TIFF? I think it ends up being 16 bit. Isn't it a Bit Map at that point?


Neato stuff about markernotes and other information imbedded in these files Bill. I look forward to an interesting thread here

Alex

from Pacific Northwest USA
"The End of the Oregon Trail"


  Printer-friendly page | Top
bclaff
Member since 26-Oct-04
1883 posts
10-May-05, 04:20 AM (GMT+2)
Click to EMail bclaff Click to send private message to bclaff Click to view user profileClick to add this user to your buddy list  
2. "RE: The Naked Truth (Raw Facts)"
In response to message #1
 
   Alex,

The camera does a 12-bit Analog to Digital Conversion (ADC) of the voltages accumulated by the CCD.
These 12-bit values are stored in the Nef file. This is the Raw data.
If it's a compressed Nef file, like the D70, some bits are lost to compression.
For more info on that see Nikon D70 compressed NEF files are lossy

Yes, the Nef file contains the Raw data, and more; as explained in the original post.

Bitmap is an oxymoron that goes back to monochrome images.
Nowadays bitmap has come to mean a format with values for individual pixels as opposed to a vector or other format.
So yes, I consider a Raw file to be a bitmap; it's just that each pixel is only red, green, or blue.

FWIW, the Nef file also has a small TIFF bitmap and two Jpegs embedding in it in addition to the Raw bitmap.

Because the computers we use store 8-bits in a byte and it's very awkward to split bytes; 12-bit data is stored using 16-bits, 4-bits are unused.

Bill

Visit me, info and galleries at: Nikon Photos and Information


  Printer-friendly page | Top
AlanC
Charter Member
2667 posts
10-May-05, 11:23 AM (GMT+2)
Click to send private message to AlanC Click to view user profileClick to add this user to your buddy list  
12. "RE: The Naked Truth (Raw Facts)"
In response to message #2
 
   >Because the computers we use store 8-bits in a byte and it's
>very awkward to split bytes; 12-bit data is stored using
>16-bits, 4-bits are unused.

Small correction here - NEF files actually pack two 12-bit values into three bytes to keep the size of the files down.

Alan.


  Printer-friendly page | Top
baliphile
Member since 22-Feb-05
43 posts
10-May-05, 05:39 AM (GMT+2)
Click to EMail baliphile Click to send private message to baliphile Click to view user profileClick to add this user to your buddy list  
3. "RE: The Naked Truth (Raw Facts)"
In response to message #0
 
   Bill, x97 is not encrypted on the D70 and all but the most recent Nikon DSLRs. Why is it that some software programs, Photoshop and RSE, misread (or read differently) the temperature in that tag. I think that you pointed this out elsewhere on Adobe and I'm now noticing that cloudy -3, 6600, is read by RSE as 7900. Is the tag content somehow open to variable interpretations?

George

Bali, ID


  Printer-friendly page | Top
bclaff
Member since 26-Oct-04
1883 posts
10-May-05, 06:02 AM (GMT+2)
Click to EMail bclaff Click to send private message to bclaff Click to view user profileClick to add this user to your buddy list  
4. "RE: The Naked Truth (Raw Facts)"
In response to message #3
 
   LAST EDITED ON 10-May-05 AT 02:18 PM (GMT 2)
 
George,

That's correct, right now no D70 tags are encrypted.

The value is not stored as in Kelvins but rather as ratios of red/green and blue/green.
Mapping this to a Correlated Color Temperature (CCT) is non-trivial so discrepancies are to be expected.
If I were reverse engineering this I would be trying to duplicate the Nikon Capture (NC) values; since this would be the "gold standard".

(Edited to remove an erroneous statement)

Bill

Visit me, info and galleries at: Nikon Photos and Information


  Printer-friendly page | Top
bclaff
Member since 26-Oct-04
1883 posts
10-May-05, 06:05 AM (GMT+2)
Click to EMail bclaff Click to send private message to bclaff Click to view user profileClick to add this user to your buddy list  
5. "RE: The Naked Truth (Raw Facts)"
In response to message #3
 
   George,

It also occurs to me that you might be talking about the discrepancy in the D70 manual page 51 and actual color temperatures.
That subject is covered here Real Color Temperature Values for White Balance

Bill

Visit me, info and galleries at: Nikon Photos and Information


  Printer-friendly page | Top
baliphile
Member since 22-Feb-05
43 posts
10-May-05, 06:45 AM (GMT+2)
Click to EMail baliphile Click to send private message to baliphile Click to view user profileClick to add this user to your buddy list  
6. "RE: The Naked Truth (Raw Facts)"
In response to message #5
 
   LAST EDITED ON 10-May-05 AT 06:52 AM (GMT 2)
 
Bill, Let me run this one past you again:

"Your selected WB and the 'As Shot' WB are different."

In Capture's WB dialogue, on a Mac, a RAW image taken with WB of cloudy -3 shows "Camera WB:Cloudy -3". In the "New WB" drop down option list, selecting the "Recorded Value" option will restore WB to Cloudy -3.

From this I'm assuming that Recorded Value and Camera WB are the same. Does this mean that "As Shot" is another variable? If so, the "Recorded Value" or "CameraWB" must be stored in another tag - has that tag also been encypted?

Thanks for the link and yes that's the thread I was referring to.

BTW Another current discussion on WB is showing that on some systems Capture displays this same WB setting as "CameraWB:6600"

George

Bali, ID


  Printer-friendly page | Top
bclaff
Member since 26-Oct-04
1883 posts
10-May-05, 07:04 AM (GMT+2)
Click to EMail bclaff Click to send private message to bclaff Click to view user profileClick to add this user to your buddy list  
7. "RE: The Naked Truth (Raw Facts)"
In response to message #6
 
   George,

You're right! I was confused.
I corrected post#4 so people who read it later won't get wrong information.

Thanks!
Bill

Visit me, info and galleries at: Nikon Photos and Information


  Printer-friendly page | Top
baliphile
Member since 22-Feb-05
43 posts
10-May-05, 07:22 AM (GMT+2)
Click to EMail baliphile Click to send private message to baliphile Click to view user profileClick to add this user to your buddy list  
8. "RE: The Naked Truth (Raw Facts)"
In response to message #7
 
   LAST EDITED ON 10-May-05 AT 07:29 AM (GMT 2)
 
Thanks for clearing that up Bill. I was not neccessarily disagreeing with you as storing the red/green and blue/green ratios is more consistent with measured data than a user setting.

George

Bali, ID


  Printer-friendly page | Top
bclaff
Member since 26-Oct-04
1883 posts
10-May-05, 07:28 AM (GMT+2)
Click to EMail bclaff Click to send private message to bclaff Click to view user profileClick to add this user to your buddy list  
9. "RE: The Naked Truth (Raw Facts)"
In response to message #8
 
   George,

Except for 'Auto' these values come from a table in the firmware.
So another fact is that only the 'Auto' WB is problematic if you respect the encryption.
The other values can be looked up given WhiteBalance and WhiteBalanceBias (tags x05 and x0B respectively, not encrypted).

Bill

Visit me, info and galleries at: Nikon Photos and Information


  Printer-friendly page | Top
baliphile
Member since 22-Feb-05
43 posts
10-May-05, 08:37 AM (GMT+2)
Click to EMail baliphile Click to send private message to baliphile Click to view user profileClick to add this user to your buddy list  
10. "RE: The Naked Truth (Raw Facts)"
In response to message #9
 
   LAST EDITED ON 10-May-05 AT 08:49 AM (GMT 2)
 
Bill,

That makes sense. The camera doesn't yet have enough oomph to deduce the CCT from an auto setting so the ratios are stored for computation elsewhere. Capture must be using x05 and x0b as default for WB and only looking at x97 when auto WB is set. That still doesn't explain how RSE, and presumably Adobe, manage to come up with a consistent offset against the camera user's set WB. They can't be using x97 because it will vary from shot to shot and RSE consistently reads a camera set 6600K as 7900K

...scratching head ...unless the firmware table look up, when the user sets WB, also stores associated red/green blue/green ratios in x97, rather than those actually measured. Do you know whether that is the case?

George

George

Bali, ID


  Printer-friendly page | Top
bclaff
Member since 26-Oct-04
1883 posts
10-May-05, 02:05 PM (GMT+2)
Click to EMail bclaff Click to send private message to bclaff Click to view user profileClick to add this user to your buddy list  
14. "RE: The Naked Truth (Raw Facts)"
In response to message #10
 
   George,

x97 always has the 'As Shot' WB.
In 'Auto' it is computed by the camera, otherwise it comes from the table in the firmware.
If you don't have access to x97 then you can use x05 and x0b and a table like the one in the firmware to "get" the 'As Shot' WB value; unless x05 is 'Auto'.

Does this make sense?

Bill

Visit me, info and galleries at: Nikon Photos and Information


  Printer-friendly page | Top
baliphile
Member since 22-Feb-05
43 posts
11-May-05, 05:43 AM (GMT+2)
Click to EMail baliphile Click to send private message to baliphile Click to view user profileClick to add this user to your buddy list  
21. "RE: The Naked Truth (Raw Facts)"
In response to message #14
 
   Thanks Bill and yes it does make sense. I doubt the 3rd party software vendors would go beyond x97 as the WB information they are seeking will always be located there. Explains also how RSE produce a "consistently" incorrect WB offset ...at least when the user sets WB to any non-auto value.

George

Bali, ID


  Printer-friendly page | Top
Kerry Pierce
Member since 23-Apr-05
61 posts
12-May-05, 02:44 AM (GMT+2)
Click to EMail Kerry%20Pierce Click to send private message to Kerry%20Pierce Click to view user profileClick to add this user to your buddy list  
24. "RE: The Naked Truth (Raw Facts)"
In response to message #14
 
   >George,
>
>x97 always has the 'As Shot' WB.
>In 'Auto' it is computed by the camera, otherwise it comes
>from the table in the firmware.
>If you don't have access to x97 then you can use x05 and x0b
>and a table like the one in the firmware to "get" the 'As
>Shot' WB value; unless x05 is 'Auto'.

Hi Bill,

Thanks for posting this stuff, it's quite interesting. I have a question on custom WB. If one sets a custom WB with a gray card, Expodisc, etc, then I assume that x05 will report 'Custom' and the true value will be recorded in x97. I also assume that there's no way to use the table lookup in the same manner that you appear to be saying will apply to the camera presets. If that is true, then the WB setting with either Auto or Custom are lost, correct?

As a side note, I've seen reports from Australian users that the new D70s appears to have this encryption as well. Have you any information on that?

Thanks.

Kerry


  Printer-friendly page | Top
bclaff
Member since 26-Oct-04
1883 posts
12-May-05, 03:04 AM (GMT+2)
Click to EMail bclaff Click to send private message to bclaff Click to view user profileClick to add this user to your buddy list  
25. "RE: The Naked Truth (Raw Facts)"
In response to message #24
 
   Kerry,

The x97 'As Shot' can only hold one value so if your setting is not 'Custom' then it will not be there.
However, I don't know what the vast majority of the other data in x97 is.
Specifically, I don't know one way or the other whether the 'Custom' setting isn't stored there as well in a different field.

As to the D70s, look elsewhere in this thread; I'm about to make a post

Bill

Visit me, info and galleries at: Nikon Photos and Information


  Printer-friendly page | Top
Kerry Pierce
Member since 23-Apr-05
61 posts
12-May-05, 03:14 AM (GMT+2)
Click to EMail Kerry%20Pierce Click to send private message to Kerry%20Pierce Click to view user profileClick to add this user to your buddy list  
27. "RE: The Naked Truth (Raw Facts)"
In response to message #25
 
   Okay, thanks for the info, Bill. I saw your post on the d70s not being encrypted, so at least that is some good news.

Kerry


  Printer-friendly page | Top
RDW
Member since 20-Sep-02
109 posts
10-May-05, 11:15 AM (GMT+2)
Click to EMail RDW Click to send private message to RDW Click to view user profileClick to add this user to your buddy list  
11. "RE: The Naked Truth (Raw Facts)"
In response to message #0
 
   'The D70 'serial number' is tag xA0, the D2X 'serial number' is tag x1D.
(The leading 'x' indicates a hexadecimal number)
The 'serial number' in an internal one and does not match what is stamped on the body.
(Sure about D70, not 100% sure about D2x)'

x1D seems to contain the actual 'stamped' serial number in D2X files, or at least a number in the correct 50xxxxx range reported elsewhere (I don't have NEF files generated with a D2X of known serial number to test, so I'm just relying on downloaded NEFs from various sites). See my post at http://www.nikonians.org/dcforum/DCForumID36/1640.html#38
This has implications for anyone uploading D2X NEFs to public sites, since it looks like they're effectively broadcasting their serial number as well. Perhaps this is one tag that _ought_ to be encrypted!


  Printer-friendly page | Top
bclaff
Member since 26-Oct-04
1883 posts
10-May-05, 02:00 PM (GMT+2)
Click to EMail bclaff Click to send private message to bclaff Click to view user profileClick to add this user to your buddy list  
13. "RE: The Naked Truth (Raw Facts)"
In response to message #11
 
   Richard,

I've heard that the D2X is hand assembled so it's certainly possible the the innards and shell could match.
But for a mass produced camera, like the D70; that would be hard to achieve.

Thanks for contributing!
Bill

Visit me, info and galleries at: Nikon Photos and Information


  Printer-friendly page | Top
sfbillm
Member since 15-Jun-04
171 posts
10-May-05, 05:09 PM (GMT+2)
Click to EMail sfbillm Click to send private message to sfbillm Click to view user profileClick to add this user to your buddy list  
15. "RE: The Naked Truth (Raw Facts)"
In response to message #0
 
   Starting with the introduction of the D2X Nikon chose to encrypt certain tags.
Makernote tags x91, x97, and x98 were obfuscated.
On the D2X this is over 1,200 bytes of information.
The 'As Shot' White Balance values are 8 bytes in the x97 tag.
Many other useful data has also been encrypted.

Bill, thank you for the most informative post so far on the business of encrypted data in NEF files. Now, please, post the info on what other data is being encrypted in the D2X NEF files. I'm sure that is of great general interest.


SantaFeBill


  Printer-friendly page | Top
bclaff
Member since 26-Oct-04
1883 posts
10-May-05, 06:10 PM (GMT+2)
Click to EMail bclaff Click to send private message to bclaff Click to view user profileClick to add this user to your buddy list  
16. "RE: The Naked Truth (Raw Facts)"
In response to message #15
 
   Bill,

x98 is lens information, such as information from your CPU lens.
I'm currently clueless about x91 and the balance of x97.
Maybe someone out there will provide the answer for those!

Bill

Visit me, info and galleries at: Nikon Photos and Information


  Printer-friendly page | Top
sfbillm
Member since 15-Jun-04
171 posts
11-May-05, 06:10 PM (GMT+2)
Click to EMail sfbillm Click to send private message to sfbillm Click to view user profileClick to add this user to your buddy list  
22. "RE: The Naked Truth (Raw Facts)"
In response to message #16
 
   x98 is lens information, such as information from your CPU lens.

So, then to summarize.

Photoshop (and any other sw that doesn't decrypt the encrypted data) won't be able to read the 'as shot' WB, lens info (type, focal length setting for zooms - I assume aperture is from the camera body), and some other as yet undetermined info in D2XD2Hs NEF files.

Am I understanding this correctly?

SantaFeBill


  Printer-friendly page | Top
bclaff
Member since 26-Oct-04
1883 posts
11-May-05, 07:19 PM (GMT+2)
Click to EMail bclaff Click to send private message to bclaff Click to view user profileClick to add this user to your buddy list  
23. "RE: The Naked Truth (Raw Facts)"
In response to message #22
 
   Bill,

Yes, for the D2X.
If unsure about the D2Hs, I have not yet acquired any D2Hs Nefs for testing.
BTW, not all lens information is encrypted; only details regarding CPU lenses.
So LensType, FocalLength, etc. are not encrypted.

Bill

Visit me, info and galleries at: Nikon Photos and Information


  Printer-friendly page | Top
avm247moderator
Charter Member
9326 posts
10-May-05, 07:57 PM (GMT+2)
Click to EMail avm247 Click to send private message to avm247 Click to view user profileClick to add this user to your buddy list  
17. "RE: The Naked Truth (Raw Facts)"
In response to message #0
 
   Bill,

Where did you get your facts? Can you point me to some documentation to support your statements?

Thanks!

Anthony

The Moderator Page and My Gallery
"Eliminate the unnecessary and focus on the substance." (Tommy Ramone of the Ramones)


  Printer-friendly page | Top
bclaff
Member since 26-Oct-04
1883 posts
10-May-05, 08:36 PM (GMT+2)
Click to EMail bclaff Click to send private message to bclaff Click to view user profileClick to add this user to your buddy list  
18. "RE: The Naked Truth (Raw Facts)"
In response to message #17
 
   Anthony,

Can you be specific about which fact(s) you want documented?

To anticipate...

I have gathered a great deal of "publicly" available information and have also done some of my own investigations.
This information is encapsulated in a "super Exif" program that I use personally.
The output of my program seems correct therefore I believe the underlying facts are correct.

I will not disseminate information that I don't clearly have the right to disseminate.
For example, telling you tag x98 is lens information is benign.
Telling you the layout and meaning of the data in tag x98 might violate Nikon's intellectual property rights.
(Just because someone else already posted it doesn't mean I will).

I hope you will understand.
If I were you I'd be disappointed but that's my position.
(This might also be the Nikonians Terms of Use position depending on how you read it.)

All I can suggest is web searchs for "makernote" AND "Nikon" etc.
And also a reminder than not all the information you find will be correct.

If you want to discuss this further I suggest e-mail rather than posting here.

Regretfully,
Bill

Visit me, info and galleries at: Nikon Photos and Information