Lossless UDP engine. Test please?

Community Forums/Showcase/Lossless UDP engine. Test please?

Andres

(Posted 2007) [#1]

UDP based lossless (optional) network engine. I'd appreciate if you could test it.

About server:
Every stream has it's own UDP port. First, client connects to server and then server will automatically redirect it to a new port.

I tested 22'000 packets in a row and it had no errors.

Functions:

server% = CreateUDPServer%([port%])
stream% = AcceptUDPStream%(server%)
stream% = OpenUDPStream%(ip$, port%, [localport%])
inip_from% = ReceiveUDPMsg%(stream%) - 0 if no message has arrived
result% = WriteUDPByte%(stream%, value%)
result% = WriteUDPBytes%(bank%, stream%, offset%, count%)
result% = WriteUDPLine%(stream%, txt$)
result% = WriteUDPInt%(stream%, value%)
result% = WriteUDPString%(stream%, txt$)
value% = ReadUDPByte%(stream%)
value% = ReadUDPBytes%(bank%, stream%, offset%, count%)
value% = ReadUDPLine$(stream%)
value% = ReadUDPInt%(stream%)
value% = ReadUDPString$(stream%)
remote_port% = UDPPort%(stream%)
remote_intip% = UDPIP%(stream%)
byte_count% = UDPReadAvail%(stream%)
RemoveUDPStream%(stream%)

Source:

Const UDPResponseDelay% = 2000, UDPRetries% = 5, LossLessUDP% = True

Function CreateUDPServer%(port% = 0)
	Local server% = CreateUDPStream(port%), bank%
	If server%
		bank% = CreateBank(4 * 3 + 1)
		
		PokeInt bank%, 0 * 4, server%
		PokeInt bank%, 1 * 4, UDPStreamIP(server%)
		PokeInt bank%, 2 * 4, UDPStreamPort(server%)
		PokeByte bank%, 3 * 4, 1
		
		Return bank%
	EndIf
End Function

Function AcceptUDPStream%(bank%)
	Local accepted%
	Local server% = PeekInt(bank%, 0 * 4)
	Local ip% = PeekInt(bank%, 1 * 4)
	Local port% = PeekInt(bank%, 2 * 4)
	
	Local from% = RecvUDPMsg(server%)
	If from%
		clientip% = UDPMsgIP(server%)
		clientport% = UDPMsgPort(server%)
		
		value% = ReadInt(server%)
		If value% => 12345 And value% =< 23456 Then
			; Create stream bank
			bank% = CreateBank(4 * 3 + 1)
			accepted% = CreateUDPStream()
			PokeInt bank%, 0, accepted%
			PokeInt bank%, 4, clientip%
			PokeInt bank%, 8, clientport%
			PokeByte bank%, 12, 0
			
			WriteInt accepted%, UDPStreamPort(accepted%)
			SendUDPMsg accepted%, clientip%, clientport%
			
			Return bank%
		Else
			FreeBank bank%
			CloseUDPStream accepted%
			
			Return False
		EndIf
	EndIf
End Function

Function OpenUDPStream%(ip$, port%, localport% = 0)
	Local stream% = CreateUDPStream(localport%)
	
	Local bank% = CreateBank(4 * 3 + 1)
	PokeInt bank%, 0, stream%
	PokeInt bank%, 4, IntIP(ip$)
	PokeInt bank%, 8, port%
	PokeByte bank%, 12, 1
	
	validation% = Rand(12345, 23456)
	WriteInt stream%, validation%
	SendUDPMsg stream%, PeekInt(bank%, 4), PeekInt(bank%, 8)
	
	For i = 1 To UDPRetries%
		tim% = MilliSecs()
		Repeat
			from% = RecvUDPMsg(stream%)
		Until from% Or MilliSecs() - tim% > UDPResponseDelay%
		If from%
			value% = ReadInt(stream%)
			PokeInt bank%, 8, value%
			
			Return bank%
		EndIf
	Next
	
	; Failed
	FreeBank bank%
	CloseUDPStream stream%
	
	Return False
End Function

Function ReceiveUDPMsg%(bank%)
	Local stream% = PeekInt(bank%, 0)
	Local ip% = PeekInt(bank%, 4)
	Local port% = PeekInt(bank%, 8)
	Local msgindex% = PeekByte(bank%, 12)
	
	from% = RecvUDPMsg(stream%)
	If from%
		If UDPMsgIP(stream%) = ip% And UDPMsgPort(stream%) = port%
			index% = ReadByte(stream%)
			If LossLessUDP%
				If Not index% = msgindex%
					; Response to message
					PokeByte bank%, 12, index%
					
					WriteByte stream%, index%
					SendUDPMsg stream%, ip%, port%
					
					Return from%
				Else
					DebugLog "Invalid message index!"
					
					; Message already received
					WriteByte stream%, index%
					SendUDPMsg stream%, ip%, port%
					
					Return False
				EndIf
			Else
				Return from%
			EndIf
		EndIf
	EndIf
End Function

Function WriteUDPByte%(bank%, value%)
	Local stream% = PeekInt(bank%, 0)
	Local ip% = PeekInt(bank%, 4)
	Local port% = PeekInt(bank%, 8)
	Local msgindex% = PeekByte(bank%, 12)
	
	For i = 1 To UDPRetries%
		; Send message
		WriteByte stream%, msgindex%
		WriteByte stream%, value%
		SendUDPMsg stream%, ip%, port%
		
		If WaitUDPResponse(bank%) Then Return True
	Next
End Function

Function ReadUDPByte%(bank%)
	Local stream% = PeekInt(bank%, 0)
	Local ip% = PeekInt(bank%, 4)
	Local port% = PeekInt(bank%, 8)
	Local msgindex% = PeekByte(bank%, 12)
	
	Return ReadByte(stream%)
End Function

Function WriteUDPBytes%(from%, bank%, offset%, count%)
	Local stream% = PeekInt(bank%, 0)
	Local ip% = PeekInt(bank%, 4)
	Local port% = PeekInt(bank%, 8)
	Local msgindex% = PeekByte(bank%, 12)
	
	For i = 1 To UDPRetries%
		; Send message
		WriteByte stream%, msgindex%
		WriteBytes from%, stream%, offset%, count%
		SendUDPMsg stream%, ip%, port%
		
		If WaitUDPResponse(bank%) Then Return True
	Next
End Function

Function ReadUDPBytes%(from%, bank%, offset%, count%)
	Local stream% = PeekInt(bank%, 0)
	Local ip% = PeekInt(bank%, 4)
	Local port% = PeekInt(bank%, 8)
	Local msgindex% = PeekByte(bank%, 12)
	
	Return ReadBytes(from%, stream%, offset%, count%)
End Function

Function WriteUDPInt%(bank%, value%)
	Local stream% = PeekInt(bank%, 0)
	Local ip% = PeekInt(bank%, 4)
	Local port% = PeekInt(bank%, 8)
	Local msgindex% = PeekByte(bank%, 12)
	
	For i = 1 To UDPRetries%
		; Send message
		WriteByte stream%, msgindex%
		WriteInt stream%, value%
		SendUDPMsg stream%, ip%, port%
		
		If WaitUDPResponse(bank%) Then Return True
	Next
End Function

Function ReadUDPInt%(bank%)
	Local stream% = PeekInt(bank%, 0)
	Local ip% = PeekInt(bank%, 4)
	Local port% = PeekInt(bank%, 8)
	Local msgindex% = PeekByte(bank%, 12)
	
	Return ReadInt(stream%)
End Function

Function WriteUDPString%(bank%, txt$)
	Local stream% = PeekInt(bank%, 0)
	Local ip% = PeekInt(bank%, 4)
	Local port% = PeekInt(bank%, 8)
	Local msgindex% = PeekByte(bank%, 12)
	
	For i = 1 To UDPRetries%
		; Send message
		WriteByte stream%, msgindex%
		WriteString stream%, txt$
		SendUDPMsg stream%, ip%, port%
		
		If WaitUDPResponse(bank%) Then Return True
	Next
End Function

Function ReadUDPString$(bank%)
	Local stream% = PeekInt(bank%, 0)
	Local ip% = PeekInt(bank%, 4)
	Local port% = PeekInt(bank%, 8)
	Local msgindex% = PeekByte(bank%, 12)
	
	Return ReadString(stream%)
End Function

Function WriteUDPLine%(bank%, txt$)
	Local stream% = PeekInt(bank%, 0)
	Local ip% = PeekInt(bank%, 4)
	Local port% = PeekInt(bank%, 8)
	Local msgindex% = PeekByte(bank%, 12)
	
	For i = 1 To UDPRetries%
		; Send message
		WriteByte stream%, msgindex%
		WriteLine stream%, txt$
		SendUDPMsg stream%, ip%, port%
		
		If WaitUDPResponse(bank%) Then Return True
	Next
End Function

Function ReadUDPLine$(bank%)
	Local stream% = PeekInt(bank%, 0)
	Local ip% = PeekInt(bank%, 4)
	Local port% = PeekInt(bank%, 8)
	Local msgindex% = PeekByte(bank%, 12)
	
	Return ReadLine(stream%)
End Function

Function UDPPort%(bank%)
	Return PeekInt(bank%, 2 * 4)
End Function

Function UDPIP%(bank%)
	Return PeekInt(bank%, 1 * 4)
End Function

Function UDPReadAvail%(bank%)
	Return ReadAvail(PeekInt(bank%, 0 * 4))
End Function

Function RemoveUDPStream%(bank%)
	Local stream% = PeekInt(bank%, 0)
	Local ip% = PeekInt(bank%, 4)
	Local port% = PeekInt(bank%, 8)
	
	CloseUDPStream stream%
	FreeBank bank%
End Function

; NON-USER COMMANDS --------------------------------------------------------------------------------

Function WaitUDPResponse%(bank%)
	Local stream% = PeekInt(bank%, 0)
	Local ip% = PeekInt(bank%, 4)
	Local port% = PeekInt(bank%, 8)
	Local msgindex% = PeekByte(bank%, 12)
	
	If LossLessUDP%
		; Wait for replay
		tim% = MilliSecs()
		Repeat
			from% = RecvUDPMsg(stream%)
		Until from% Or MilliSecs() - tim% > UDPResponseDelay%
		If from%
			If UDPMsgIP(stream%) = ip% And UDPMsgPort(stream%) = port%
				; Message from same client
				
				index% = ReadByte(stream%)
				
				If index% = msgindex%
					If msgindex% < 250 Then PokeByte bank%, 12, msgindex% + 1 Else PokeByte bank%, 12, msgindex% - 250
					Return True
				EndIf
			EndIf
		EndIf
	Else
		Return True
	EndIf
End Function

Function IntIP(ip$)
	Local a = UDPSector$(ip$, ".", 0)
	Local b = UDPSector$(ip$, ".", 1)
	Local c = UDPSector$(ip$, ".", 2)
	Local d = UDPSector$(ip$, ".", 3)
	Return  (a Shl 24) + (b Shl 16) + (c Shl 8) + d
End Function

Function UDPSector$(txt$, separator$, sector%, toend% = False)
	Local result$ = "", occ
	For i = 1 To Len(txt$)
		If Mid$(txt$, i, 1) = separator$
			occ = occ + 1
			If toend% And occ% > sector% Then result$ = result$ + Mid$(txt$, i, 1)
		Else
			If occ => sector Then result$ = result$ + Mid$(txt$, i, 1)
		EndIf
		If Not toend% Then If occ > sector Then Exit
	Next
	Return result$
End Function

Function UDPSectors%(txt$, needle$)
	occ% = 0
	For i = 1 To Len(txt$) Step 1
		If Instr(txt$, needle$, i)
			occ% = occ% + 1
			i = Instr(txt$, needle$, i)
		Else
			Exit
		EndIf
	Next
	Return occ%
End Function

Example:

; ------------Server--------------
Include "udp.bb"

Print "Creating UDP server..."
server% = CreateUDPServer(2010)
If server%
	Print "Waiting for client..."
	Repeat
		stream% = AcceptUDPStream(server%)
	Until stream%
	Print "Client (" + DottedIP(UDPIP(stream%)) + ":" + UDPPort(stream%) + ") connected!"
	
	Repeat
		If ReceiveUDPMsg%(stream%)
			avail% = UDPReadAvail(stream%)
			Print avail% + " bytes available!"
			value$ = ReadUDPLine$(stream%)
			Print "Message: " + value$
		EndIf
	Forever
Else
	Print "Failed to create server!"
EndIf
WaitKey
; ------------Client--------------
Include "udp.bb"

Print "Connecting to server..."
stream% = OpenUDPStream("127.0.0.1", 2010)
If stream%
	Print "Received access to " + DottedIP(UDPIP(stream%)) + ":" + UDPPort(stream%)
	
	For i = 1 To 1000
		msg$ = "Hello message #" + i + "!"
		If WriteUDPLine(stream%, msg$)
			Print "Message: " + msg$
		EndIf
	Next
Else
	Print "Failed!"
EndIf
WaitKey

Koriolis

(Posted 2007) [#2]

I haven't tested it (no time), but I have just one question. Did you make any benchmark to compare the speed with raw htpp streams?

Andres

(Posted 2007) [#3]

Nope, i haven't benchmarked it yet.

Stevie G

(Posted 2007) [#4]

Just gave this a go and program froze without opening a server. Is there something else I need to do for a router?

Andres

(Posted 2007) [#5]

Don't think so. I created it in BP so maybe B3D needs some changes in the example.

CreateUDPServer isn't anything more than:

Function CreateUDPServer%(port% = 0)
	Local server% = CreateUDPStream(port%), bank%
	If server%
		bank% = CreateBank(4 * 3 + 1)
		
		PokeInt bank%, 0 * 4, server%
		PokeInt bank%, 1 * 4, UDPStreamIP(server%)
		PokeInt bank%, 2 * 4, UDPStreamPort(server%)
		PokeByte bank%, 3 * 4, 1
		
		Return bank%
	EndIf
End Function

basically only two functions: CreateUDPStream() and CreateBank()

Vertex

(Posted 2007) [#6]

Sounds nice, but I don't know:
Can packages change there order? For example, a package with index 5 comes before the package with index 4.

Is the using of banks faster than types ?

cu olli

Gabriel

(Posted 2007) [#7]

Stevie G: you'll probably need to enable port forwarding for the port the example uses to open a server. I'm not big on net stuff, but I'm pretty sure I remember Surreal telling me that UDP communications don't get through a NAT firewall the way that TCP ones might.

I guess it wouldn't really do for a firewall to allow programs to start a server at will.

Andres

(Posted 2007) [#8]

Vertex: No! Packages are sent in order if LossLessUDP% is true.

I used for loop in the test and 22'000 packages were in order and none of them got lost.

Yes, IMO banks are faster, but if i'd request lost packet if a bigger index has been received it would be even faster.

jfk EO-11110

(Posted 2007) [#9]

I didn't test it, just some suggestions, in case you did not already add these features: there WILL be lost packets with UDP, or even packets with altered content. So if you did not already, you should implement an itegrity check of packets at a very low layer. EG: every packet is using a checksum in the header. So the receiver can see if the packet is integer and valid. If it isn't there must be some kind of reordering system. Something like "received packet number 1234 with corrupt data, please send again", as well as a basic handshake protocol to make the sender know if a packet was received successfully, so he can erase it from the stack of outgoing data.
Everything other than Acknowledge packets should then be acknowledged. (Cause acknowledging acknowledges would result in a endless loop of course)

To test this inegrity check and error correction etc. (also antispoofing btw.) you should use a utility that simulates lost and altered packets. This may be a little function in the test app, that will randomly "destroy" things.

Andres

(Posted 2007) [#10]

I've read that the packages are always correct (if they are received) the only thing is that they might be received in wrong order or not at all.

jfk EO-11110

(Posted 2007) [#11]

Not sure, UDP has no error detection, so in theory there could be some wrong bits, probably only on defect hardware/connections. Using a checksum will also make sure packets cannot be manipulated by a third party. A more or less secret checksum system could be used.
Wrong order may be caused by various routes of certain packets. That's why each packet header should also contain the sequence number of its stream.

You may even detect a missing packet by the sequence numbers, eg. when packet 1 and 3 were received, but packet 2 is still missing after some time.

These are of course only some thoughts, hope you don't mind.

Andres

(Posted 2007) [#12]

No, not at all.
Currently it waits for response and then returns either true or false. These 22'000 packets were all non-corrupted.
I tested packets with LosslessUDP=false few times and it sent about 700 packets correctly and then about five of them got lost so only 1 packet doesn't disappear usually.

jfk EO-11110

(Posted 2007) [#13]

What clent/server connection did you use to test it?

Andres

(Posted 2007) [#14]

Sorry for so late post.

What do you mean by client/server connection?
You mean my connection speeds? I have 1MB/s down and 80KB/s up.

While testing i even ran emule (50KB/s up, ~100KB/s down) and ftp client (20KB/s up) and it still gave no errors though it was extremely slow.

jfk EO-11110

(Posted 2007) [#15]

I meant, when you used to test it, did you use a real internet connection to an other machine, or 2 machines in a lan, or client and server on the same machine?

Andres

(Posted 2007) [#16]

No not in lan, tested it twice, without running emule and ftp client the machine's distance was about 2KM, but while running them i used a machine's that's distance was about 200KMs

slenkar

(Posted 2007) [#17]

thanks for the source