Byte per byte download is slow!

Blitz3D Forums/Blitz3D Programming/Byte per byte download is slow!

bytecode77

(Posted 2008) [#1]

hey;)
since no-one could help me with this problem yet, i thought i ask you how do download fast using blitz tcp.

i usualy do something like this:

for i = 1 to bytes
    writebyte file, readbyte(tcp)
next

which gets me 100 kb/s maximum.
i also tried readline/writeline which works only with html files and not with image files (600 kb/s)

do you have any idea how to read whole blocks to accelerate the whole process?
thanks in advance :)

SebHoll

(Posted 2008) [#2]

It's been a while since I used BlitzPlus/Blitz3D networking commands, but you can read multiple bytes at once using ReadBytes() into a bank previously created with CreateBank(). Note, that with ReadBytes(), you also have to specify how many bytes to read which can be determined with ReadAvail(). I haven't tested it, but something like the following should work:

Local tmpBank = CreateBank(bytes)
Local i% = 0, tmpAvailBytes% = 0
Repeat
    If Eof(tcp) Then RuntimeError "TCP Connection terminated before all bytes could be downloaded. Progress: " + i + "/" + bytes + " bytes."
    tmpAvailBytes = Min(ReadAvail(tcp), bytes-i)
    ReadBytes( tmpBank, tcp, i, tmpAvailBytes )
    i=i+tmpAvailBytes
Until i>=bytes

Hope this helps. ;-)

bytecode77

(Posted 2008) [#3]

readbytes and writebytes seems to be for banks only or i didnt understand your idea correctly...

my code is based on BlitzGet:

Graphics 640, 480, 32, 2
SetBuffer BackBuffer()

DeleteFile "1_google_logo.jpg"
BlitzGet("http://www.tgdaily.com/picturegalleries/20070423/1_google_logo.jpg")
If FileType("1_google_logo.jpg") <> 1 Then RuntimeError "Download error!"

DrawImage LoadImage("1_google_logo.jpg"), 0, 0
Flip
WaitKey()
End

Function BlitzGet(path$)
If Left(path$, 7) = "http://" Then path$ = Right(path$, Len(path$) - 7)
inst = Instr(path$, "/")
If inst Then
	host$ = Left(path$, inst - 1)
	path$ = Right(path$, Len(path$) - inst + 1)
Else
	host$ = path$
	path$ = "/"
EndIf

If out$ = "" Then
	If path = "/" Then
		out$ = "Unknown file.txt"
	Else
		For findSlash = Len(path$) To 1 Step - 1
			testForSlash$ = Mid(path$, findSlash, 1)
			If testForSlash$ = "/" Then
				out$ = Right(path$, Len(path$) - findSlash)
				Exit
			EndIf
		Next
		If out$ = "" Then out$ = "Unknown file.txt"
	EndIf
EndIf

tcp = OpenTCPStream(host$, 80)
If tcp Then
	WriteLine tcp, "GET " + path$ + " HTTP/1.1"
	WriteLine tcp, "Host: " + host$
	WriteLine tcp, "User-Agent: unknown"
	WriteLine tcp, "Accept: */*"
	WriteLine tcp, ""
	Repeat
		header$ = ReadLine(tcp)
		reply$ = ""
		inst = Instr(header$, ": ")
		If inst Then reply$ = Left(header$, inst + 1)
		If Lower(reply$) = "content-length: " Then toread = Right(header$, Len(header$) - Len(reply$))
	Until header$ = "" Or Eof(tcp)
	If toread > 0 Then
		file = WriteFile(out$)
		For readpath = 1 To toread
			;==========================================================================================
			If Eof(tcp) = False Then WriteByte file, ReadByte(tcp)
			;==========================================================================================
		Next
		CloseFile file
		If readpath - 1 <> toread Then RuntimeError "Download failed!"
	EndIf
	CloseTCPStream tcp
Else
	RuntimeError "Failed to connect"
EndIf
End Function

I also tried reading and writing integers (=4 bytes) instead of bytes, which is WAY FASTER, but it adds 0-3 empty bytes at the end of the file which i dont really like to see...

SebHoll

(Posted 2008) [#4]

You can still use the method - in fact, at present you are writing byte by byte directly to the hard-disk which is horribly slow. Instead, write the data to a memory bank, and then save the bank directly to disk. Again, not tested (as I don't have BlitzPlus/Blitz3D installed), but here is the BlitzGet() function tweaked:

Function BlitzGet2(path$)
	If Left(path$, 7) = "http://" Then path$ = Right(path$, Len(path$) - 7)
	inst = Instr(path$, "/")
	If inst Then
		host$ = Left(path$, inst - 1)
		path$ = Right(path$, Len(path$) - inst + 1)
	Else
		host$ = path$
		path$ = "/"
	EndIf
	
	If out$ = "" Then
		If path = "/" Then
			out$ = "Unknown file.txt"
		Else
			For findSlash = Len(path$) To 1 Step - 1
				testForSlash$ = Mid(path$, findSlash, 1)
				If testForSlash$ = "/" Then
					out$ = Right(path$, Len(path$) - findSlash)
					Exit
				EndIf
			Next
			If out$ = "" Then out$ = "Unknown file.txt"
		EndIf
	EndIf
	
	tcp = OpenTCPStream(host$, 80)
	If tcp Then
		WriteLine tcp, "GET " + path$ + " HTTP/1.1"
		WriteLine tcp, "Host: " + host$
		WriteLine tcp, "User-Agent: unknown"
		WriteLine tcp, "Accept: */*"
		WriteLine tcp, ""
		Repeat
			header$ = ReadLine(tcp)
			reply$ = ""
			inst = Instr(header$, ": ")
			If inst Then reply$ = Left(header$, inst + 1)
			If Lower(reply$) = "content-length: " Then toread = Right(header$, Len(header$) - Len(reply$))
		Until header$ = "" Or Eof(tcp)
		If toread > 0 Then
			Local tmpBank = CreateBank(toread)
			Local i% = 0, tmpReadAvail%
			Repeat
				;==========================================================================================
				If Eof(tcp) Then
					ResizeBank(tmpBank, i)
				Else
					tmpReadAvail = Min(ReadAvail(tcp),toread-i)
					ReadBytes( tmpBank, tcp, i, tmpReadAvail)
					i = i + tmpReadAvail
				EndIf 
				;==========================================================================================
			Until i >= BankSize(tmpBank)
			file = WriteFile(out$)
			WriteBytes( tmpBank, file, 0, BankSize(tmpBank) )
			CloseFile file
			If BankSize(tmpBank) <> toread Then RuntimeError "Download failed!"
			FreeBank( tmpBank )
		EndIf
		CloseTCPStream tcp
	Else
		RuntimeError "Failed to connect"
	EndIf
End Function

Function Min( intOne, intTwo )
	If intOne < intTwo Then Return intOne Else Return intTwo
End Function

Edit: Updated! I've downloaded and installed Blitz3D for you, and have tested the above code. It seems to work far faster than the old code.

bytecode77

(Posted 2008) [#5]

ok. this is fairly fast... 1.2 MB/s!!!

EDIT: the other problem is solved, too. thanks for your help buddy ;)
i'm telling you for what i needed all this: i'm writing a program, which downloads a html file and is seeking links and downloads them, too. this way i'm able to 'download the internet'
bye ;)

Panno

(Posted 2008) [#6]

devils child have you enough CD's for your cool project ?
if not i will send some ......

bytecode77

(Posted 2008) [#7]

no, but i have enough webspace to host my project as soon as it is finished.

bytecode77

(Posted 2008) [#8]

ok. it is now working perfectly. thank you for your help :)