Remote File Engine

Community Forums/Showcase/Remote File Engine

Andres

(Posted 2006) [#1]

Use the commands as you would reading local files :) Except that RemoteFileSize%(file%) can be used only on opened files.

* 19.09.2006: Added Post request possibility
If POSTvariables$ has a value then the request will be automatically turned into post one and the variables will be added too Example: POSTvariables$ = "name=Andres&age=18"

* 20.09.2006: Added HTTP header changing
You can now add cookies and stuff to header request: HTTPheader$ = "Cookie: username=my_name; password=my_pass"
Also many lines of header is addable by using line break: chr(13) + chr(10)

* 29.09.2006: Multi protocol
Now supporting FTP protocol. Usage:
file% = OpenRemoteFile%("ftp://www.domain.com/file", 21, "username", "password")

* 12.10.2006: Better server detection and File info funcs
Now opening TCP streams to host too. Added Some file info functions: RemoteFileName$(file%), RemoteFileCode%(bank%), RemoteFileProtocol$(bank%). If the file name is sent in header then you can receive it with RemoteFileName$(file%) because the file name isn't always in URL. Improved EORF()

Any suggestion or criticism is welcome!

Function list:
file% = OpenRemoteFile%(url$[, port%, POSTvariables$, HTTPheader$])
CloseRemoteFile(file%)
EORF(file%)
RemoteFileSize%(file%) - if no size known then it's -1
RemoteFileName$(file%)
RemoteFileCode%(file%) - Server response code (HTTP only)
RemoteFileProtocol$(file%)
ReadRemoteLine$(file%)
ReadRemoteString$(file%)
ReadRemoteInt%(file%)
ReadRemoteShort%(file%)
ReadRemoteByte%(file%)
WriteRemoteBytes(file%, save%, offset%, count%)
RemoteReadAvail(file%)

Example:

Include "transporter.bb"

file% = OpenRemoteFile("http://battleempire.kylm.com/files/clip.avi")
save% = WriteFile("received.avi")

Graphics 400, 60, 32, 2
SetBuffer BackBuffer()

While Not KeyHit(1)
	Cls
	
	Text 10, 10, "Downloaded: " + total% + "/" + RemoteFileSize(file%) + "(" + (Int 100 * (Float total% / RemoteFileSize(file%))) + "%)"
	
	If Not EORF(file%)
		N% = WriteRemoteBytes(file%, save%, 0, RemoteReadAvail(file%))
		total% = total% + N%
	Else
		Text 10, 30, "FINISHED!"
	EndIf
	
	Flip
Wend

CloseFile save%
CloseRemoteFile(file%)

Engine to include:

Global UserAgent$ = "Transporter"	; What ever you'd like to name your application
Global ResponseDelay% = 2000		; How many millisecs to wait for HTTP response
Global NewLine$ = Chr(13) + Chr(10)	; Line break may differ
Const RFDebugMode% = False				; Debug mode

Type transporter
	Field id%
	Field code%
	Field filename$
	Field protocol$
End Type

Function OpenRemoteFile%(path$, port% = 80, variables$ = "", httpheader$ = "")
	Local protocol$ = Lower(Sector$(path$, ":", 0))
	Local host$ = Sector$(path$, "/", 2)
	Local ip$ = "www." + Sector$(host$, ".", Sectors%(host$, ".") - 1) + "." + Sector$(host$, ".", Sectors%(host$, "."))
	path$ = "/" + Sector$(path$, "/", 3, True)
	Local header%, bank%, tim%, occ%
	
	
	Local Stream% = OpenTCPStream(host$, port%)
	If Not Stream% Then Stream% = OpenTCPStream(ip$, port%)
	
	If Stream
		Select protocol$
			Case "http", "https" ; ---------------------------------------------------------------------------------------------------------------------------------------
				; Send request
				
				If Not Len(variables$) Then
					WriteLine(Stream, "GET " + path$ + " HTTP/1.1")
				Else
					WriteLine(Stream, "POST " + path$ + " HTTP/1.1")
				EndIf
				WriteLine(Stream, "Host: " + host$)
				WriteLine(Stream, "User-Agent: " + UserAgent$)
				If Len(variables$) Then
					WriteLine(Stream, "Content-Type: application/x-www-form-urlencoded")
					WriteLine(Stream, "Content-Length: " + Len(variables$))
				EndIf
				If Len(httpheader$) Then WriteLine(Stream, httpheader$)
				WriteLine(Stream, "Connection: Close")
				WriteLine(Stream, "")
				If Len(variables$) Then WriteLine(Stream, variables$)
				
				; Debug mode
				If RFDebugMode% Then
					If Not Len(variables$) Then DebugLog ">>> GET " + path$ + " HTTP/1.1" Else DebugLog ">>> POST " + path$ + " HTTP/1.1"
					DebugLog ">>> Host: " + host$
					DebugLog ">>> User-Agent: " + UserAgent$
					If Len(variables$) Then
						DebugLog ">>> Content-Type: application/x-www-form-urlencoded"
						DebugLog ">>> Content-Length: " + Len(variables$)
					EndIf
					If Len(httpheader$) Then DebugLog ">>> " + httpheader$
					DebugLog ">>> Connection: Close"
					DebugLog ">>> "
					If Len(variables$) Then DebugLog ">>> " + variables$
				EndIf
				
				; Wait for response
				tim = MilliSecs()
				Repeat
				Until (MilliSecs() - tim) => ResponseDelay% Or ReadAvail(Stream)
				
				txt$ = ReadLine$(Stream%)
				If RFDebugMode Then DebugLog "<<< " + txt$
				If Sector$(txt$, " ", 0) = "HTTP/1.1" Or Sector$(txt$, " ", 0) = "HTTP/1.0"
					code% = Int Sector$(txt$, " ", 1)
					If (code% => 300) Then
						CloseTCPStream Stream%
						Return False
					EndIf
				EndIf
				
				occ% = 0
				header% = CreateBank(0)
				bank% = CreateBank(12); Bank Int, Stream Int, Size Int
				
				PokeInt(bank%, 0, header%)
				PokeInt(bank%, 4, Stream%)
				PokeInt(bank%, 8, -1)
				
				this.transporter = New transporter
					this\id% = bank%
					this\code% = code%
					this\filename$ = Sector$(path$, "/", Sectors%(path$, "/"))
					this\protocol$ = "http"
				
				; Received HTTP Header
				Repeat
					txt$ = ReadRemoteLine$(bank%)
					value$ = Mid$(Sector$(txt$, ":", 1, True), 2)
					
					If RFDebugMode Then DebugLog "<<< " + txt$
					
					Select Sector$(txt$, ":", 0)
						Case ""
							Return bank%
						Case "Content-Disposition"
							For i = 0 To Sectors(value$, ";")
								subtxt$ = Trim$(Sector$(value$, ";", i))
								subvalue$ = Sector$(subtxt$, "=", 1)
								
								Select Sector$(subtxt$, "=", 0)
									Case "filename"
										this\filename$ = subvalue$
																End Select
							Next
						Case "Content-Length"
							PokeInt bank%, 8, Int(value$)
					End Select
				Forever
			Case "ftp" ; ---------------------------------------------------------------------------------------------------------------------------------------
				header% = CreateBank(1)
				bank% = CreateBank(16); Bank Int, Stream Int, Size Int
				
				PokeInt(bank%, 0, header%)
				PokeInt(bank%, 4, Stream%)
				PokeInt(bank%, 8, -1)
				PokeInt(bank%, 12, stream%)
				
				this.transporter = New transporter
					this\id% = bank%
					this\code% = "0"
					this\filename$ = Sector$(path$, "/", Sectors(path$, "/"))
					this\protocol$ = "ftp"
				
				; Wait for response
				tim = MilliSecs()
				Repeat
				Until (MilliSecs() - tim) => ResponseDelay% Or ReadAvail(Stream)
				
				Repeat
					If ReadAvail(stream%)
						txt$ = ReadLine(stream%)
						code% = Int Sector$(txt$, " ", 0)
						value$ = Mid$(Sector$(txt$, " ", 1, True), 1)
						If RFDebugMode Then DebugLog "<<< " + txt$
					Else
						txt$ = "":code% = 0:value$ = "":cmd$ = ""
					EndIf
					
					If Not ReadAvail(stream%)
						Select code%
							Case 220
								cmd$ = "USER " + variables$
							Case 331
								cmd$ = "PASS " + httpheader$
							Case 230
								cmd$ = "SIZE " + path$
							Case 213
								PokeInt(bank%, 8, Int value$)
								cmd$ = "PASV"
							Case 227
								; Connect to PASV mode
								txt$ = Sector(txt$, "(", 1)
								host$ = Sector(txt$, ",", 0) + "." + Sector(txt$, ",", 1) + "." + Sector(txt$, ",", 2) + "." + Sector(txt$, ",", 3)
								port% = (Int Sector(txt$, ",", 4)) * 256 + (Int Left$(Sector(txt$, ",", 5), Len(Sector(txt$, ",", 5)) - 1))
								pasv_stream% = OpenTCPStream(host$, port%)
								
								; Update stream
								PokeInt(bank%, 4, pasv_stream%)
								
								cmd$ = "RETR " + path$
							Case 150
								Return bank%
						End Select
						
						; ERROR
						If code% => 400
							If stream% Then CloseTCPStream stream%
							If pasv_stream% Then CloseTCPStream pasv_stream%
							If header% Then FreeBank header%
							If bank% Then FreeBank bank%
							Return 0
						EndIf
					EndIf
					
					If Len(cmd$) Then
						If RFDebugMode% Then DebugLog ">>> " + cmd$
						WriteLine stream%, cmd$
					EndIf
				Forever
				
				Return bank%
		End Select
	EndIf
End Function

Function CloseRemoteFile(bank%)
	If PeekInt(bank%, 0) Then FreeBank PeekInt(bank%, 0)
	If PeekInt(bank%, 4) Then CloseTCPStream PeekInt(bank%, 4)
	If BankSize(bank%) = 18 Then If PeekInt(bank%, 12)
		WriteLine PeekInt(bank%, 12), "BYE"
		If RFDebugMode% Then DebugLog ">>> BYE"
		CloseTCPStream PeekInt(bank%, 12)
	EndIf
	FreeBank bank%
	
	For this.transporter = Each transporter
		If this\id% = bank%
			Delete this
			Exit
		EndIf
	Next
End Function

Function EORF(bank%)
	If Not PeekInt(bank%, 4)
		If BankSize(PeekInt(bank%, 0)) = 0 Then Return True
	Else
		If Eof(PeekInt(bank%, 4)) Then Return True
	EndIf
End Function

Function RemoteFileSize%(bank%)
	Return PeekInt(bank%, 8)
End Function

Function RemoteFileName$(bank%)
	For this.transporter = Each transporter
		If this\id% = bank% Return this\filename$
	Next
End Function

Function RemoteFileCode%(bank%)
	For this.transporter = Each transporter
		If this\id% = bank% Return this\code%
	Next
End Function

Function RemoteFileProtocol$(bank%)
	For this.transporter = Each transporter
		If this\id% = bank% Return this\protocol$
	Next
End Function

Function ReadRemoteLine$(bank%)
	Local avail%, rbank% = PeekInt(bank%, 0), stream% = PeekInt(bank%, 4)
	
	; Update bank
	UpdateRemoteFile(bank%)
	
	; Read line
	txt$ = ""
	For i = 0 To BankSize(rbank%) - 1
		char% = PeekByte(rbank%, i)
		txt$ = txt$ + Chr(char%)
		If Right$(txt$, Len(NewLine$)) = NewLine$ Then Exit
	Next
	
	If Right$(txt$, 2) = NewLine$ Then
		txt$ = Mid$(txt$, 1, Len(txt$) - Len(NewLine$))
	ElseIf Right$(txt$, 1) = Right$(NewLine$, 1) Then
		txt$ = Mid$(txt$, 1, Len(txt$) - (Len(NewLine$) - 1))
	EndIf
	
	; Resize bank
	size% = BankSize(rbank%) - (Len(txt$) + Len(NewLine$))
	If size% < 0 Then size% = 0
	ResizeRemoteBank(bank%, size%)
	
	Return txt$
End Function

Function ReadRemoteString$(bank%)
	Local rbank% = PeekInt(bank%, 0)
	UpdateRemoteFile(bank%)
	
	a% = PeekInt(rbank%, 0)
	
	txt$ = ""
	For i = 0 To a% - 1
		txt$ = txt$ + Chr(PeekByte(rbank%, 4 + i))
	Next
	
	ResizeRemoteBank(bank%, BankSize(rbank%) - (4 + a%))
	Return txt$
End Function

Function ReadRemoteInt%(bank%)
	Local rbank% = PeekInt(bank%, 0)
	UpdateRemoteFile(bank%)
	
	a% = PeekInt(rbank%, 0)
	ResizeRemoteBank(bank%, BankSize(rbank%) - 4)
	Return a%
End Function

Function ReadRemoteShort%(bank%)
	Local rbank% = PeekInt(bank%, 0)
	UpdateRemoteFile(bank%)
	
	a% = PeekShort(rbank%, 0)
	ResizeRemoteBank(bank%, BankSize(rbank%) - 2)
	Return a%
End Function

Function ReadRemoteByte%(bank%)
	Local rbank% = PeekInt(bank%, 0)
	UpdateRemoteFile(bank%)
	
	a% = PeekByte(rbank%, 0)
	ResizeRemoteBank(bank%, BankSize(rbank%) - 1)
	Return a%
End Function

Function WriteRemoteBytes(bank%, file%, offset%, count%)
	Local rbank% = PeekInt(bank%, 0)
	UpdateRemoteFile(bank%)
	
	Local N% = WriteBytes(rbank%, file%, offset%, count%)
	ResizeRemoteBank(bank%, BankSize(rbank%) - N%)
	
	Return N%
End Function

Function RemoteReadAvail(bank%)
	UpdateRemoteFile(bank%)
	received% = BankSize(PeekInt(bank%, 0))
	If PeekInt(bank%, 4) Then waiting% = ReadAvail(PeekInt(bank%, 4)) Else waiting% = 0
	Return received% + waiting%
End Function

Function UpdateRemoteFile(bank%)
	Local rbank% = PeekInt(bank%, 0), stream% = PeekInt(bank%, 4)
	
	If stream% And rbank%
		avail% = ReadAvail(stream%)
		offset% = BankSize(rbank%)
		
		ResizeBank(rbank%, offset% + avail%)
		ReadBytes(rbank%, stream%, offset%, avail%)
		
		If Eof(stream%) Then
			CloseTCPStream stream%
			PokeInt bank%, 4, 0
		EndIf
	EndIf
End Function

Function ResizeRemoteBank(bank%, size%)
	Local rbank% = PeekInt(bank%, 0), start% = BankSize(rbank%) - size%
	
	If BankSize(rbank%)
		CopyBank(rbank%, start%, rbank%, 0, size%)
		ResizeBank(rbank%, size%)
	EndIf
End Function

Function Sector$(txt$, separator$, sector%, toend% = False)
	Local result$ = "", occ
	For i = 1 To Len(txt$)
		If Mid$(txt$, i, 1) = separator$
			occ = occ + 1
			If toend% And occ% > sector% Then result$ = result$ + Mid$(txt$, i, 1)
		Else
			If occ => sector Then result$ = result$ + Mid$(txt$, i, 1)
		EndIf
		If Not toend% Then If occ > sector Then Exit
	Next
	Return result$
End Function

Function Sectors%(txt$, needle$)
	occ% = 0
	For i = 1 To Len(txt$) Step 1
		If Instr(txt$, needle$, i)
			occ% = occ% + 1
			i = Instr(txt$, needle$, i)
		Else
			Exit
		EndIf
	Next
	Return occ%
End Function

Andres

(Posted 2006) [#2]

An example how to load TGA files directly from internet:

Include "transporter.bb"

file% = OpenRemoteFile("http://www.oat.ee/andres/oval.tga")
If Not file% Then RuntimeError "Unable to open the file!"

Graphics 800, 600, 32, 2
SetBuffer BackBuffer()

; Read TGA header
IDLenght% = ReadRemoteByte(file%)
ColorMapType% = ReadRemoteByte(file%)
ImageType% = ReadRemoteByte(file%)
ColorMapIndex% = ReadRemoteShort(file%)
ColorMapEntries% = ReadRemoteShort(file%)
ColorMapSize% = ReadRemoteByte(file%)
Xhandle% = ReadRemoteShort(file%)
YHandle% = ReadRemoteShort(file%)
Width% = ReadRemoteShort(file%)
Height% = ReadRemoteShort(file%)
BPP% = ReadRemoteByte(file%)
Attributes% = ReadRemoteByte(file%)

ImageID$ = ""
For i = 1 To IDLenght
	ImageID$ = ImageID$ + Chr(ReadRemoteByte(file%))
Next

; Read TGA body
image% = CreateImage(Width%, Height%)
LockBuffer ImageBuffer(image%)
For y = 0 To Height% - 1
	For x = 0 To Width% - 1
		b% = ReadRemoteByte(file%)
		g% = ReadRemoteByte(file%)
		r% = ReadRemoteByte(file%)
		a% = ReadRemoteByte(file%)
		WritePixelFast x, height% - y - 1, (b% Or (g% Shl 8) Or (r% Shl 16) Or ($FF000000)), ImageBuffer(image%)
	Next
Next
UnlockBuffer ImageBuffer(image%)

CloseRemoteFile(file%)

; Display the received image
While Not KeyHit(1)
	Cls
	DrawImage image%, MouseX(), MouseY()
	Flip
Wend

Beaker

(Posted 2006) [#3]

Looks useful. Thanks.

Andres

(Posted 2006) [#4]

You can read blitzbasic.com's content as logged in by adding cookies:

Include "transporter.bb"

file% = OpenRemoteFile("http://www.blitzbasic.com/Account/_index_.php", 80, "", "Cookie: _i=user_id; _p=user_pass")
If Not file% Then RuntimeError "Unable to open the file!"

While Not KeyHit(1)
	If Not EORF(file%) Then Print ReadRemoteLine$(file%)
Wend

CloseRemoteFile(file%)

For example my user_id is 8421
user_pass is just string of your password (or URL_Encoded if needed)

jfk EO-11110

(Posted 2006) [#5]

Really useful. Could be used to update/patch things.

Andres

(Posted 2006) [#6]

Now supporting FTP protocol and debug mode :)

Andres

(Posted 2006) [#7]

Now:
1. OpenRemoteFile("http://sub.domain.com") will firstly try to connect to sub.domain.com rather than www.domain.com
2. Reads file name and some other info from HTTP header

@rtur

(Posted 2006) [#8]

Cool, thanks!