Using HttpSendRequestEx for large POST requests

Such a pain to use WinINet in Asynchronous mode, isn't it?

The WinINet section in MSDN documentation seems incomplete once you start to look for things you want to know about. The sample codes provided in MSDN documents and Technical Support pages are too simple to know the exact behavior of WinINet. However, by trial and error, I have learned a big deal about WinINet. As I am still in the process of learning how WinINet works and how I'm supposed to use it, I would like to share something about WinINet that could be really tricky.

As I am writing this post I will assume that  whoever is reading this already knows how to use the WinINet Windows API in some sort of way. All the details that I consider irrelevant to the following topic will not be explained here.

So the thing that I am going to talk about is...
Making a POST request with large data

More specifically, if you are stuck in a problem where you do a post operation typically with a sufficiently large sized file and you never get a response back see if this is your problem.

Check what your code's behavior is when InternetWriteFile() returns false, and GetLastError returns ERROR_IO_PENDING. An experienced programmer might be able to get things right, but without much knowledge about asynchronous IO the above situation was very confusing.

What happens when InternetWriteFile() return false, and the reason was because IO operation was already pending? What is your guess??? The problem that I had resulted from not knowing the exact behavior for such situation.

Here is what I thought would happen.
Since InternetWriteFile returned false, the operation failed... nothing is going to be written. Besides the last paramter for InternetWriteFile, which is an out parameter that indicates the number of bytes written to the file when the function returns results in the value 0 when the InternetWriteFile returns... Therefore, nothing happened, and nothing is going to happen until the pending IO operation completes and INTERNET_STATUS changes to INTERNET_STATUS_REQUEST_COMPLETE and InternetWriteFile() is called again.

From what I have learned yesterday, here is what really happens. However, this is not a confirmed fact, but my own understanding of the behavior.
When InternetWriteFile() returns false due to ERROR_IO_PENDING, the write operation takes place after the current pending operation(maybe it's the read operation on the buffer for the actual writing operation on the network that is pending) is finished. When that write operation is finished after the pending operation is finished, the out parameter which indicates the written number of bytes is updated.

For this reason, this is why all the WinINet examples on MSDN and tech support pages are written in the following manner. During the initialization, or the setup phase of WinINet, an event handle is created. When InternetWriteFile() returns false for a pending IO request, then it waits for the event handle to be signaled, using WaitForSingleObject(). Then the pending IO operation would eventual finish and the write operation will take place. After the write operation is finished, the INTERNET_STATUS will change to INTERNET_STATUS_REQUEST_COMPLETE and the Internet Status Callback Function will be called. SetEvent() is called on the event handle in the callback function, and the blocked thread will be able to resume. Since the requested write operation completed, the out parameter which in indicates the number of bytes written of InternetWriteFile() us updated properly, and everything is good !!!

When posting large data, InternetWriteFile() must be called several times in order to send data in portions. If you want to get things done right, the calculation of how much data has been sent so far must be accurate. And the calculation cannot be accurate if the information of number of bytes written resulting from InternetWriteFile() is inaccurate. And since the information is not updated accurately until the operation has finished, it has to wait.

How lame... you have to block your thread to get things right. So, I came up with an alternative solution. Make a flag which will be set when an ERROR_IO_PENDING occurs. Make a global(not literally, just has to be visible from the appropriate scope) variable, lets say something like dwBytesWritten, which will be used to get the bytes written information. Pass &dwBytesWritten as the last parameter to InternetWriteFile()  in order to retrieve the number of bytes that has been written. On INTERNET_STATUS_REQUEST_COMPLETE, check the flag. If the flag is set, then it means that the write request which was blocked previously because of another IO operation has finished, and the number of bytes written is updated on the vairable dwBytesWritten. So, you can now update the writing progress of data being posted.

Do you get it? It's really hard to put it into words...

Reference - http://support.microsoft.com/kb/177188/en-us