NewsForumsCodeLogsGallerySpecsSearch

BRL.Reflection: Pointer support

BlitzMax Forums/BlitzMax Module Tweaks/BRL.Reflection: Pointer support

grable(Posted 2009) [#1]
snip


grable(Posted 2009) [#2]
UPDATED: September 3 2016 - version 1.28

!!! Contains a potentially breaking change to how function return types are handled. !!!
!!! Specificly TMethod and TFunction requires calling TypeId().ReturnType() instead of just TypeId() !!!
!!! Or one can call TMethod.ReturnType() and TFunction.ReturnType() directly !!!

!! Only mainline module that uses this is BRL.MaxLUA, so line 77 in maxlua.bmx would need to me modified to use this modified reflection !!
If this change is unacceptable, see post #82 for a copy of reflection.bmx with original semantics. (untested)

!! Win32 and Linux targets use assembly for more arguments (30), MacOS still uses old way (8).

see history in source below for more info.

Download precompiled: brl.reflection.mod_grb_v1.28.7z
Or use the sources below...

mod/brl.mod/reflection.mod/reflection.bmx


mod/brl.mod/reflection.mod/reflection.cpp


mod/brl.mod/reflection.mod/callmethod.win32.x86.s


mod/brl.mod/reflection.mod/callmethod.linux.x86.s


mod/brl.mod/reflection.mod/callmethod.macos.x86.s



grable(Posted 2009) [#3]
snip


Warpy(Posted 2009) [#4]
Would it be possible to create something like EnumFunctions() for functions which don't belong to a type?

I can create a TFunction manually for a function I already know about, proving that functions work the same wherever they are:
Function a(s$)
	Print s
End Function

f:tfunction=New tfunction.init("a",IntTypeId.functiontype([StringTypeId]),"",IntTypeId,a)
f.invoke(["hi"])


but I don't know enough about how the debugger works internally to be able to find arbitrary functions.

How I imagine it working is something like: f:TFunction = TFunction.ForFunction(a)


grable(Posted 2009) [#5]
It is possible. But only in Debug mode, as no meta-data is stored for functions in Release mode :/

The meta-data is related to the Debug Scope, so its in its own table.
Decoding it shouldnt pose a problem though, its getting access to it.

Any ideas?


spacerat(Posted 2010) [#6]
This code doesn't seem to work any more. When I call TTypeID.ForName("class name") for any class, I usually get an EXCEPTION_ACCESS_VIOLATION, and the debugger points to line 1139:
	_functions.AddLast New TFunction.Init(id, tt, meta, Self, Byte Ptr(Int Ptr(Self._class + p[3])[0]) )


Before finding this thread I'd been using the code I stuck together at the end of this http://www.blitzmax.com/Community/posts.php?topic=84420#956203 thread, which crashes in the same way on the equivalent line. I suppose some internal change to blitz broke it; can anyone spread any light on what changed or how to fix it?


grable(Posted 2010) [#7]
The original BRL.Reflection hasnt changed though, so im guessing the meta-data hasnt changed either.
I just tried some tests with 1.39 and i cant get it to crash on any of my tests.

Can you give me an example?

Btw, you say it "usually" crashes. Does this mean it works some of the time?
If so it might not be the reflection code, but some other memory related error.


n-Halbleiter(Posted 2010) [#8]
Another thing I just found out: You forgot to add 
	_functions=New TList

to the Method "Init" of TTypeID. Adding this prevents your program to crash sometimes, if you tried to look up an unexistent function, because ObjectTypeID does not have a list "_functions".

So the (slightly) modified Method "Init" could look like this:
	Method Init:TTypeId( name$,size,class=0,supor:TTypeId=Null )
		_name=name
		_size=size
		_class=class
		_super=supor
		_fields=New TList
		_methods=New TList
		_functions=New TList
		_nameMap.Insert _name.ToLower(),Self
		If class _classMap.Insert New TClass.SetClass( class ),Self
		Return Self
	End Method


Just wanted to mention. :)

EDIT: Corrected typing mistake...


n-Halbleiter(Posted 2011) [#9]
Just one thing I just noticed:

The modification seems not to work with BMax version 1.42. If I have a Function inside a type and when I try to do _anything_ with reflection it crashes in the line spacerat already mentioned. Can anyone confirm this?

Thanks in advance


grable(Posted 2013) [#10]
bump

top post has been updated with new BRL.Reflection.


grable(Posted 2013) [#11]
bug fix update. new version 1.10, see top post.

fixed a stupid bug in FindConstant and made ForName able to parse function types with spaces between elements.
allso added a new convenience method TField.FieldPtr() for a direct pointer to the field of and instance.

On a side note, I really wonder why globals within a type arent emitted like the rest of the elements.. even constants are emitted, just not exposed via the default reflection implementation.

It would be handy to have type globals as well i think :)


grable(Posted 2013) [#12]
version 1.11:
refixed method overrides, allso did the same for functions (sorting by name and removing duplicates)
cant believe i missed this one for so long :p


FabriceW(Posted 2013) [#13]
I wrote a little dependency injection module for blitzmax using reflection. It is working great with your modifications.

You can find it on GitHub: here


Brucey(Posted 2015) [#14]
Hallo. Here's a list of issues I found whilst applying these changes for bmx-ng :

* TypeTagForId() doesn't work properly for pointers.
* _Push and _Assign can crash for Pointer/Function types if value is null.


grable(Posted 2015) [#15]
Thanks brucey, they fail in vanilla bmx too.. Some wrong assumptions on my part ;)

Pointer types now use _elementType to properly store its base.
And _Push/_Assign check for Null and sets 0 for pointers and NullFunctionError for function pointers.

EDIT: Left some bad code in last update, fixed now.


Derron(Posted 2016) [#16]
Any ideas why the "updated revisions" (>1.05 or so) do no longer work with LUA objects)?

I expose objects to lua, so eg. a lua script could call "list.Count()". With the current revision, this is no longer possible (it is not exposed).

When accessing an object my lua engine does something in the likes of:
Local obj:Object = lua_unboxobject(getLuaState(), 1)
Local typeId:TTypeId = TTypeId.ForObject(obj)
Local ident:String = lua_tostring(getLuaState(), 2)
print "ident: "+ident +"  typeId: "+typeId.name()


so for aboves example this outputs "ident: Count typeId: TList" - when running on an ancient version of the code (1.05 or so - at least it contains some of the changes you did, but I did not have pointer / functionpointers...).
Ok so when I updated the code, it now outputs "ident: Count typeId: String".
Of course invokation of "Count()" is then no longer possible (as it sees it as a property then).

I assume it has to do with the changes done in the helper functions (ForTag etc.) and that it is coming from an external source (lua_unboxobject).

lua_unboxobject() could be found in "maxlua.mod/lua_object.c".


I also tried to replace some changed functions with other ones, but up to now without success.


Edit:

Samplecode:
Strict

Type TMyObject
	Field myList:TList = CreateList()

	Method GetList:TList()
		Local list:TList = CreateList()
		list.AddLast("1")
		Return list
	End Method
End Type


Global myObject:TMyObject = New TMyObject
LuaRegisterObject(myobject,"myobject")


Global source:string = "function Run()~n print(~qmylist:~q .. myobject.myList.Count() .. ~q  GetList:~q .. myobject.GetList().Count())~nend~n"
Global luaClass:TLuaClass = TLuaClass.Create(source)
Local luaInstance:TLuaObject = TLuaObject.Create(luaClass, Null)

luaInstance.Invoke("Run",Null)


This returns "mylist:0 GetList:1" when using old reflection code (which could be found HERE).
When using the extended reflection in it's current incarnation, it exits with an lua error: "[string "function Run()..."]:2: attempt to call field 'Count' (a nil value)".
To check it on your own you will have to adjust either brl.mod/reflection.mod - or adjust brl.mod/maxlua.mod to use your custom reflectionXYZ.mod.


Does someone have "inbetween" releases of the code, so I could check at which point of development it stopped working (for me) ?


bye
Ron


Derron(Posted 2016) [#17]
It also fails when calling some "extended objects" Super.Init()-calls (which call their Super.Init() and so on) ... I got errors in "TagForType()" with some null-properties.

Am not able to produce a small code snippet for now - but it seems to be something aside of the LUA connection.

bye
Ron


grable(Posted 2016) [#18]
Yeah, sorry about that. I made the TypeId of functions, methods and function pointers be the same (before, its TypeId was only its return type).
Essentially allowing for Function TypeIds, this in turn now requires to call ReturnType() to get at the return type.

There was a bug in my BRL.Reflection version too which didnt account for this when calling methods, added a fixed version above now.

For MaxLUA and any other code that uses a TMethod or TFunction and calling TypeId() on it to deduce its return type now has to use TypeId().ReturnType() or TMethod.ReturnType() instead...
I know its a breaking change :(

On line 77 in bmxlua.bmx change:
Select meth.TypeId()
to
Select meth.TypeId().ReturnType()
or
Select meth.ReturnType()
And it should work again.


EDIT: If this change really bothers some people or messes up a lot of code, let me know. I could change TMethod.TypeId() and TFunction.TypeId() to return ONLY the return type directly and add an aditional .FunctionType() and/or .MethodType() to get at the full type (even though this would brake from the convention of .TypeId() returning the full type of any member)


Derron(Posted 2016) [#19]
Your new code - and some small adjustments, let my lua-connection does its duty again. Thanks.


Regarding "ReturnType()" and inconsistency: I understand your thoughts on this. I think the reason for a "TypeID" returning the call-result's type is, something similar to "consistency".

You do not need to know what special instance a "TMember"-extending object is, you just need to call memberOrExtendedMember.TypeID() to get the desired result.

The "member.TypeId().ReturnType()" is indeed a worthy replacement - if you know the existence of ReturnType() :-).



@functionType/methodType
I do not know what is more important: "backwards compatibility" (keeping other modules unchanged) or consistency?

I just searched mods/brl.mod for "TypeID(" and only maxlua.bmx seems to use it for now.

What about "TypeID()" returning the member typeID (TFunction, TConstant...) and "ValueTypeID()" returning the typeID for whatever is returned (for functions this is the result type, for constants the TypeID().

Keeping it backwards compatible would lead to "MemberTypeID()" returning TConstant, TField, TFunction, .. and "TypeID()" returning the return value "type".
Hmpf, somehow I cannot come up with something really helpful now, maybe others will come to help regarding that issue.


bye
Ron


Derron(Posted 2016) [#20]
Edit: maybe you want to incorporate Brucey's fix regarding Null-Arrays:

https://github.com/maxmods/brl.mod/commit/fbbdbc8d222d115a11d0f2eb7ebe5556c29347f4


Else I would wait some days and modify the maxmods-reflection.mod to reflect your current changes (and then adjust maxlua.mod if needed).


bye
Ron


grable(Posted 2016) [#21]
Thanks for letting me know about that one Derron :)
Updated top post, and i changed it a little so one doesnt need mingw to recompile.


Derron(Posted 2016) [#22]
Doesn't your adjustment (regarding Brucey's bbRefArrayNull) remove the possibility to check if the returned value is a NullArray?

In Brucey's code the "bbRefArrayNull" is always the same (the pointer to a special object is returned ?) in yours there is a fresh object created everytime.

Or did I miss something - what is the difference between your approach and Brucey's ? I just want to avoid that we miss something Brucey added by intention...


bye
Ron


Derron(Posted 2016) [#23]
Needed to add this to reflection.cpp

BBArray * bbRefArrayNull() {
	return &bbEmptyArray;
}


and to reflection.bmx:

in externs:
Function bbRefArrayNull:Object()


and replace
		If value
			Local c=typeId._class
			Local t=bbRefGetObjectClass( value )
			While t And t<>c
				t=bbRefGetSuperClass( t )
			Wend
			If Not t Throw "ERROR"
		EndIf
		bbRefAssignObject p,value


with
		If value
			Local c=typeId._class
			Local t=bbRefGetObjectClass( value )
			While t And t<>c
				t=bbRefGetSuperClass( t )
			Wend
			If Not t Throw "ERROR"
		Else
			If typeId.Name().Endswith("]") Then
				value = bbRefArrayNull()
			EndIf
		EndIf
		bbRefAssignObject p,value

(as done by Brucey).


Without that change I get segfaults in a sample code of my framework.
Function TypeTagForId$( id:TTypeId )
	If id.ExtendsType( ArrayTypeId )
		Return "[]"+TypeTagForId( id.ElementType() )
	EndIf
[...]

Debugger says, "id" is null. The problem is a nulled array which I use. So instead of "field obj:myobjects[5]" I have "field obj:myobjects[]" ("uninitialized").


So this means your "extends(ArrayTypeId)" does not trigger in that case (dunno why) and the result is a null-access (segfault).

bye
Ron


Derron(Posted 2016) [#24]
Some further questions:

---
in _CallMethod you convert pointers from Byte Ptr, in _CallFunction from Int

		If retTypeId.ExtendsType(PointerTypeId) Or retTypeId.ExtendsType(FunctionTypeId) Then
			Local f:Byte Ptr(p0,p1,p2,p3,p4,p5,p6,p7)=p
			Return String.FromInt( Int f( q[0],q[1],q[2],q[3],q[4],q[5],q[6],q[7] ) )		


'=== VS ===
			If retTypeId.ExtendsType(PointerTypeId) Or retTypeId.ExtendsType(FunctionTypeId) Then
				Local f:Int(p0, p1, p2, p3, p4, p5, p6, p7) = funcp
				Return String.FromInt( f( q[0],q[1],q[2],q[3],q[4],q[5],q[6],q[7] ) )


Is this _intention_ or is this something you forgot to unify?


---
In "Function TypeTagForId$( id:TTypeId )" you take care of "PointerTypeId" and "FunctionTypeId" twice (once "simple" and another time by checking if it extends from it).


---
Within TMethod you have "Method FunctionPtr..." - which is not utilized. The "else"-case of this method differs to the "else" part of the Invoke() method. Is this by intention?

- by further checks, it seems as if you just do unneeded code there (as in "Init()" the _fptr is already set to BytePtr(index). Think we could shape some code off here.

So 
	Method Invoke:Object( obj:Object,args:Object[] )
		If _index<65536
			Return _CallMethod( bbRefMethodPtr( obj,_index ),_typeId.ReturnType(),obj,args,_typeId._argTypes )
		EndIf
		Return _CallMethod( Byte Ptr(_index),_typeId,obj,args,_typeId._argTypes )
	End Method


becomes
	Method Invoke:Object( obj:Object,args:Object[] )
		Return _CallMethod( FunctionPtr(obj), _typeId.ReturnType(), obj, args, _typeId._argTypes )
	End Method


If aboves int/byte ptr difference was unintented, you could unify "_callMethod" and "_callFunction" and shorten it even more (only difference then is the pushing of the instance in a method call).


Edit:
This is the current version of your code including my adjustments:
https://raw.githubusercontent.com/GWRon/brl.mod-vanilla/ae348595e4e8778e7a8d8a53639e72c48c5e6a39/reflection.mod/reflection.bmx

Changes (yours + mine) could be followed there:
https://github.com/GWRon/brl.mod-vanilla/commits/feat_extendedReflection2/reflection.mod/reflection.bmx

With mine being:
https://github.com/GWRon/brl.mod-vanilla/commit/ae348595e4e8778e7a8d8a53639e72c48c5e6a39


bye
Ron


grable(Posted 2016) [#25]
Sorry for being so slow to answer, i forget to check my email often enough and dont frequenctly check the module sub forum.

In Brucey's code the "bbRefArrayNull" is always the same (the pointer to a special object is returned ?) in yours there is a fresh object created everytime.
It should be the same really, and with my extremely limited tests it was.

Without that change I get segfaults in a sample code of my framework.
Hmm.. there might be a reason Brucey chose to test the name of the type after all. Its just that it should not be needed if everything is as it should...

SuperStrict

Local nullobject:Object
Local nullarray:Object[]
Local nullstring:String

Local t1:TTypeId = TTypeId.ForObject(nullobject)
Local t2:TTypeId = TTypeId.ForObject(nullarray)
Local t3:TTypeId = TTypeId.ForObject(nullstring)

If t1 Then Print "t1: " + t1.Name()
If t2 Then Print "t2: " + t2.Name()
If t3 Then Print "t3: " + t3.Name()
Notice that t1 in the above example is Null, which is in line with the original.
Is there a need for a Null type akin to the Null[] perhaps? Or is the Null[] type not needed either?
Its been a while since i coded this so the reasoning behind it escapes me :/


About _CallMethod and _CallFunction differences: Forgetfullness :p

About TypeTagForId: More forgetfullness, specificly that ExtendsType() checks for equality directly.

About TMethod.FunctionPtr() not being used: Perhaps premature optimization by limiting the amount of method calls. I intended it to be a user level method really.


Could you test your framework with bbArrayNull() and using ExtendsType(ArrayTypeId) instead of the name check? The purist in me doesnt like it one bit ;)
And double check that it really does get different Null arrays, as it shouldnt do that.

Id really like to keep it able to compile without mingw for those who still dont use it, though maybe at this point there is no need?

But all in all i accept your changes :)
If you agree i can include them in the copy here.. But id like to double check the null array stuff first.


Derron(Posted 2016) [#26]
Hmm, I somehow cannot replicate the segfault now ... neither without your nullArray nor Brucey's variant.

regarding MinGW: brl.reflection does contain the .cpp-file already, so MinGW is used regardless of the adjustment Brucey's fix uses.


So it seems, as we could mix both additions to something new:
		ElseIf typeId.ExtendsType(ArrayTypeId) Then
			If Not value Then
				value = bbRefArrayNull()
			EndIf
		EndIf


bbRefArrayNull() is the better decision (imho) as it uses the "internal" representation - maybe "object[]" does that too, but am not sure about this.

I understand your scepticism to things like Find("]") :-). If array recognition works flawless, there would be no need for this.


Regarding your sample:
_assign is only used when using reflection to manipulate values - so in my case this is most probably a Clone-Method for cloning instances (which might contain uninitialized arrays).
As there were other issues (already tackled but: dimensions were not correct for some (copied) arrays - happens on Mac and windows) things might have come together without us knowing the real issues.



---
I just searched my emails for the correspondence with Brucey about this subject and found an issue for this, but he only adjusted it for "bmx-ng/brl.mod":
Brucey:
In some cases, bbArrayCastFromObject will return a &bbNullObject.
It should probably always return an array of some kind - at least a &bbEmptyArray.

https://github.com/bmx-ng/brl.mod/issues/7



@different null arrays
Seems I did not express myself well then. What I am was talking about is, that "nullArray" is a freshly created local variable - each time. the bbRefArrayNull() is returning the same variable each time (but an function call of course - or is the "ref" different each time?).
I assume "local nullarray:object[]" is returning the very same thing (uninitialized).



BTW:


I assumed "myB" would have a length of 0 - but it throws out that myB is not an arrayType...



Conclusion:
Your code (nullarray) should work as expected now (until I get it to crash again :-)). Nonetheless I would prefer doing it similar to other object types and return the "bb***" type (as Brucey did) - of course then, with your "Extends-variant".


The most up to date variant is now:
https://github.com/GWRon/brl.mod-vanilla/commits/feat_extendedReflection2/reflection.mod/reflection.bmx

@agree
No problem, most of codes are zlib/libpng - or in this case they are just the same as the original code, so feel free to adopt/copy/print out ... :-)


bye
Ron


grable(Posted 2016) [#27]
bbRefArrayNull() is the better decision (imho) as it uses the "internal" representation - maybe "object[]" does that too, but am not sure about this.
Any array declaration without a size will be set to bbEmptyArray no matter what.
Otherwise you could not do .Length on it without it returning bogus values or crashing.

brl.reflection does contain the .cpp-file already, so MinGW is used regardless of the adjustment Brucey's fix uses.
Not if it has already been compiled. So long as the CPP file itself hasent changed it will just import the object file unmodified.

But since both you and Brucey seem to like it better il cave ;)


The reason it fails for the second array in your test is because it expects TTypeId._ElementType to be filled, the Null[] type didnt do this.. So a bug :)

Stick this after the definition of NullTypeId to fix it:
' finish setup of ArrayTypeId
ArrayTypeId._ElementType = NullTypeId


Updated top post with new version.


Derron(Posted 2016) [#28]
So for my sample this now responds with:
myB: Null[] length: 0

While "myB" is of type int[] - so a null-sized integer-array.

"So a bug :)" :-)


@precompiled cpp
You are right with that - nonetheless people will already have MinGW installed if they compiled BlitzMax from github.com (as it is an open source project now :-).
*teeth grinding* you are right - but at least I tried to exculpate.


bye
Ron


grable(Posted 2016) [#29]
While "myB" is of type int[] - so a null-sized integer-array.
Even though the type of the field is Int[] there is no way for TTypeId.ForObject() to know that. All it sees is bbEmptyArray, hence the Null[].

*teeth grinding* you are right - but at least I tried to exculpate.
Sorry for being a pedant :p


Derron(Posted 2016) [#30]
Right, but for "Fields" we could find out the type by asking the "parent".


SuperStrict
Framework Brl.StandardIO
Import Brl.Reflection 'the modded one!

Type TMyType
	Field B:Int[0]
End Type
global my:TMyType = new TMyType

print "Definition: TMyType.B type="+TTypeID.ForObject(my).FindField("B").TypeID().name()
print "Content:            B type="+TTypeID.ForObject(my.B).name()


Definition: TMyType.B type=Int[]
Content:            B type=Null[]



But returning "Null[]" for all possibilities is surely more consistent.
Else we could return the definition-TTypeId for empty arrays when possible. But for this, "ForObject(my.B)" should know about the object "my", which it doesn't (could be referenced by multiple objects).
Hmm, seems there is no 100% working way, except adding this data right on compilation (with BCC, kind of "meta data"). Things should be kept up to date then as BlitzMax allows assigning 0-sized arrays ("bla = New Int[0]").


Ideas how this could get solved?

Also ideas on how to access globals are very welcome.


BTW:
' finnish setup of array type

Might be better than a Danish setup :-)


bye
Ron


grable(Posted 2016) [#31]
But for this, "ForObject(my.B)" should know about the object "my", which it doesn't (could be referenced by multiple objects).
You just cant. Not because of some limitation of reflection, but because its an expression returning a value, and that value is Null.

An evil example to further the point.
Type TTest
   Field X:Float[]
EndType
Local x:TTest = New TTest
Local a:Int[] = [ 1 ]
Byte Ptr Ptr(Varptr x.X)[0] = Byte Ptr Ptr(Varptr a)[0]
Print TTypeId.ForObject(x.X).Name()

except adding this data right on compilation (with BCC, kind of "meta data")
Its not impossible, but it would blow up the meta-data, as every level of every object expression would have to be added. And its not something one really needs anyway.

BlitzMax allows assigning 0-sized arrays ("bla = New Int[0]").
True, but it still returns bbEmptyArray for them, so no type is assigned.
You could make bbEmptyIntArray and friends but that would break a whole lot of code expecting only 3 null types (object, array, string) and would probably make things slower when checking for Null.

Also ideas on how to access globals are very welcome.
Not unless mark decides to add them to the meta-data.
Though i guess now that BlitzMax is opensource it wouldnt be to difficult, i just prefer to use vanilla myself.

Might be better than a Danish setup :-)
Damn, i could swear i fixed that! Thats how it goes when one juggles 4 editor instances of the same file lol


Derron(Posted 2016) [#32]
You could make bbEmptyIntArray and friends but that would break a whole lot of code expecting only 3 null types (object, array, string) and would probably make things slower when checking for Null.


couldnt bbEmptyIntArray inherit bbEmptyArray so "old code" is still able to checkwhether something is an empty array at all?

Problem stays, that empty "MyObject[]"-arrays would need their "MyObjectEmptyArray" too - which is not existing. So we only would add exceptions for the inbuilt-simple-"types" (int, float, ...).


@other stuff
Thanks for the elaboration. Disregarding this we advanced with the code and got rid of some bugs. Benefits for all (and for free ;-)).

Appended the last revision to my repository.

bye
Ron


grable(Posted 2016) [#33]
couldnt bbEmptyIntArray inherit bbEmptyArray so "old code" is still able to checkwhether something is an empty array at all?
It may work like that with reflection, but blitz itself does simple pointer compares so it would have to check a loooong list of stuff.
And it would require compiler help to do it, not something i would advocate at all!

In fact there are too many null types already, it would be best to coalesce the 3 into a single one. Less confusion for the runtime and compiler.


Always nice to get rid of bugs :D


Derron(Posted 2016) [#34]
When using The DX10/11 modules (https://github.com/SRSSoftware/srs.mod) the resulting binary crashes at the "Throw" in "Function TypeTagForId$( id:TTypeId )".

But it then crashes in one of my types, which I cannot explain. As soon as I remove the connection to that modules, everything works nicely again.

Ideas?


bye
Ron


grable(Posted 2016) [#35]
Could you change that Throw into:
Throw "~q" + id.Name() + "~q was unexpected at this time"
To narrow it down a bit?

Im just guessing here, but it might happen upon an extern type and getting confused...


Derron(Posted 2016) [#36]
Thrown (as expected):
"Null" was unexpected at this time.









The odd thing is, I did not even use that TBatchImage, I just imported the module. And as you see on the pictures, the source of the throw is in "base.util.helper" - no function given, which means it troubled within two globals I define there:

	Global ListTypeID:TTypeId= TTypeId.ForObject(new TList)
	Global MapTypeID:TTypeId= TTypeId.ForObject(new TMap)

(I use them to check in a clone method whether an object extends from a Tlist or a tmap - as they are often used containers)

Disabling the "ForObject" call there, delays the error for some milliseconds to trigger at something else trying to use some reflection. So I assume it was coincidence to crash on these globals ... they are just the first things to use reflection (right upon start).


As I am not able to "shrink it down" to something useable.

Download my complete game (code + assets), copy the SRS.mod to your modules, import it from the "TVTower.bmx" file (the project main file) and it should "break".

https://github.com/GWRon/TVTower
https://github.com/SRSSoftware/srs.mod


bye
Ron


Derron(Posted 2016) [#37]
Thrown (as expected):
"Null" was unexpected at this time.









The odd thing is, I did not even use that TBatchImage, I just imported the module. And as you see on the pictures, the source of the throw is in "base.util.helper" - no function given, which means it troubled within two globals I define there:

	Global ListTypeID:TTypeId= TTypeId.ForObject(new TList)
	Global MapTypeID:TTypeId= TTypeId.ForObject(new TMap)

(I use them to check in a clone method whether an object extends from a Tlist or a tmap - as they are often used containers)

Disabling the "ForObject" call there, delays the error for some milliseconds to trigger at something else trying to use some reflection. So I assume it was coincidence to crash on these globals ... they are just the first things to use reflection (right upon start).


As I am not able to "shrink it down" to something useable for now:
Download my complete game (code + assets), copy the SRS.mod to your modules, import it from the "TVTower.bmx" file (the project main file) and it should "break".

https://github.com/GWRon/TVTower
https://github.com/SRSSoftware/srs.mod


bye
Ron


grable(Posted 2016) [#38]
I downloaded the source and tried to run, but all i get is this weird error:
Building TVTower
error loading "C:/TVTower/source/main.bmx"
Process complete
Never run into something like this before :(

I suspect my BMK is broken for some reason, but everything else seems to work so im stumped...


Derron(Posted 2016) [#39]
Ah ok... think you use the vanilla BMK ? It has no support for some additional things done on compilation (creation of a "source/version.txt" containing the compilation date)

So it might then stumble over the missing incbin or so?


bye
Ron


grable(Posted 2016) [#40]
Yeah i use vanilla bmx.

And its not the version.txt it bugs on, it just doesnt like reading from UTF-16-LE files. At all.
Meaning it fails to read even the first byte.

This is surely weird.

EDIT: It manages to read the BOM atleast, and figures out the correct format. But fails to read the next byte.


Derron(Posted 2016) [#41]
They should be UTF8 with BOM ... as BlitzMax else even is not able to print "äüö" (Umlauts) - which are present in the ANSI chars (246 is "ö").

$ file -i main.bmx
main.bmx: text/plain; charset=utf-8

(i also run for f in `ls *.bmx`; do echo "$f" ' -- ' `file -bi "$f"` ; done and only got us-ascii and utf-8 files)
Saving the file as UTF16-LE resulted in the correct recognition as utf-16le.


Only alternative to UTF8 with BOM was to extract every string to text files and then importing them... which I only did for localized files. But one my developer buddies uses German comments - resulting in the usage of umlauts...



I compile the very same code within my Windows XP VM using the default BlitzMax 1.50 (plus its MaxIDE) and MinGW 4.6.1. I updated BMK to use Brucey's (as this allows for some pre/post-compilation scripts).


bye
Ron


grable(Posted 2016) [#42]
Yeah, i see now the original is UTF8 with BOM, must have been maxide that saved it as UTF16LE for some reason.
But it still fails to read it :( And opening the file in Scite i can see it has highlighted umlauts in Red, signaling that they may be incorrect.. or something.

Il try to remove all occurances of that and see what happens.


Derron(Posted 2016) [#43]
Yes... editing these files in MaxIDE is not recommended ... same for blIDE (that dev used blIDE [full version]) as this also destroyed these characters as it tried to save it in some odd encoding mix.


Even if it destroys the character, the error you got is more than misleading: the source file is there, it just might run into problems compiling them... very odd.

PS: Brucey did compile the game more than once and never mentioned such an error, so it might be something with your BlitzMax-installation (1.50 - as some older versions stalled when processing some comments in my code).


bye
Ron


grable(Posted 2016) [#44]
For even more weirdness, compiling BMK without "Framework BRL.Blitz" and it works as expected!
Scratch that, i forgot i replaced LoadText() with LoadString() to test that ;)

The odd thing too, is that i can use LoadText() to load that same file without trouble. Its just my BMK that is borked for whatever reason :(


Derron(Posted 2016) [#45]
I am using "Framework BRL.StandardIO" :-)

... but I cannot explain why removing the framework-command resolves such an issue...

Edit: registered your "scratch this"

If LoadText was the thing failing, then the incbin might be problematic - albeit it fails differently here, when the to-incbin-file is not existing.

Edit2:
Do you use the vanilla bmk or a custom built one ?
Maybe you are able to try Bruceys BMK (do not forget to copy core.bmk + make.bmk to the bin-dir). It will really speed up compilation. Newer incarnations of his BMK look for BlitzMax/MinGW32 to find a custom MinGW and else fall back to the default one.

bye
Ron


grable(Posted 2016) [#46]
I use vanilla BMK, just modified to work with GCC 5.1.0. So nothing special, just some added flags to various parts.
Nothing that should mess with TTextStream.

I have tried Bruceys BMK before, but never gotten it to work sadly.

By removing the BOM it does compile though, which is weird to say the least.


Derron(Posted 2016) [#47]
Ok ... then - if the game is running now on your side (hopefully it does :-)) the next step would be to download the DX10-module and place it in your mods-folder. The import it within the code (main.bmx or so - I used source/Dig/base.util.graphicsmanager.win32.bmx - as it would be used there) and after compilation it should error when executing (because of the reflection).

Your reflection code is there:
source/Dig/external/reflectionExtended/reflection.bmx


Edit: if you miss maxmod2.mod then you could find a "lite" version of it in source/Dig/external/maxmod2_lite.mod.zip

If you do not want to have to use another module (and then fall back to "FreeAudio"), open up
source/Dig/base.sfx.soundmanager.bmx
and use the "nortaudio" variant for your platform (I use this for a user who runs Knoppix/Linux and has issues with PulseAudio...).

bye
Ron


grable(Posted 2016) [#48]
It will take some time, i have to compile until it fails on a file. Remove the BOM on that file. And then recompile until it hits another file with the same error.
Surprisingly its not all of the files, just a few of them.


Derron(Posted 2016) [#49]
Sorry to read about that tedious job :-)

Meanwhile you could check my post before yours ...as I edited it (maxmod2.mod).

bye
Ron


grable(Posted 2016) [#50]
Got it to compile, but it now fails in the linker:
C:/TVTower/source/.bmx/zip.c.release.win32.x86.o:zip.c:(.text+0x197b): undefined reference to `_time32'
C:/TVTower/source/Dig/external/libxml/src/.bmx/nanohttp.c.release.win32.x86.o:nanohttp.c:(.text+0x7d2): undefined reference to `WspiapiGetAddrInfo@16'
C:/TVTower/source/Dig/external/libxml/src/.bmx/nanohttp.c.release.win32.x86.o:nanohttp.c:(.text+0x847): undefined reference to `WspiapiFreeAddrInfo@4'
C:/TVTower/source/Dig/external/libxml/src/.bmx/nanohttp.c.release.win32.x86.o:nanohttp.c:(.text+0x864): undefined reference to `WspiapiFreeAddrInfo@4'
C:/TVTower/source/Dig/external/libxml/src/.bmx/nanohttp.c.release.win32.x86.o:nanohttp.c:(.text+0x8bf): undefined reference to `WspiapiFreeAddrInfo@4'

Nothing i do manages to fix it either.
I guess our build systems are just too different :(

Sorry, but im giving up now. Unless you manage to narrow the reflection bug down to a smaller sample.

Ive updated top post with a possible fix. (added NullTypeId check)


Derron(Posted 2016) [#51]
Sorry to hear (but i understand ...)

Dunno why you get this error, people on the internet suggest to import "-lws2_32" ... but this is not needed here. Assume this has to do with our minGW versions.


Will try your "possible fix" tomorrow when sober and awake :-)

Maybe I can come up with a smaller example then.

bye
Ron


Derron(Posted 2016) [#52]
SuperStrict

Import SRS.D3D11Max2D
Import "../TVTower.WorkingCopy/TVTower.git/source/Dig/external/reflectionExtended/reflection.bmx"

Global ListTypeID:TTypeId=TTypeId.ForObject(New TList)

Print "done"


This errors out here... of course you need to adjust the path to the reflection file.


bye
Ron


grable(Posted 2016) [#53]
Hmmm.. It works here :/

Even when loading only that version of reflection.
Framework BRL.StandardIO
SuperStrict

Import SRS.D3D11Max2D
Import "c:/TVTower/source/Dig/external/reflectionExtended/reflection.bmx"

Global ListTypeID:TTypeId=TTypeId.ForObject(New TList)

Print "done"



Derron(Posted 2016) [#54]
That's very odd then...

Hmm.. potential source of the error?
I mean ... I understand that it should not "error" as I do not use the imported source, so reflection does not know about it... some "framework" thingy ?


bye
Ron


grable(Posted 2016) [#55]
I just thought of something that might cause problems even if no types or reflection is used directly, types with the same name in different modules.

And i got this to error out:
SuperStrict

Import SRS.D3D11Max2D

Local o:Object = New TList
Global ListTypeID:TTypeId=TTypeId.ForObject(o)
Print ListTypeID.Name()
Print ListTypeID.FindMethod("ToString").Invoke(o, Null).ToString()

Print "done"

Type TList
	Method ToString:String()
		Return "new TList"
	EndMethod
EndType
Comment out the D3D11 module and it works like it should again.

Im going to bed now but il look into it more tomorrow. At least i have something to work with now :)


grable(Posted 2016) [#56]
I couldnt sleep, so did some tests and figured out what was causing the error.
Var parameters ;) Seems they get converted to pointers in the metadata.
Well it was the cause, but the bug in reflection code was in TTypeId.PointerType() which caused it to recurse over its root PointerTypeId when it shouldnt.

Top post updated with fix.

I hope this works for you Derron, and thanks for being patient :)

EDIT: Heres the test i ended up using:
SuperStrict

Type TSample
	Method Update(pos:Float[] Var)
	EndMethod
EndType

Local o:Object = New TSample
Global t:TTypeId=TTypeId.ForObject(o)
Print t.Name()
Local m:TMethod = t.FindMethod("Update")
For Local a:TTypeId = EachIn m.ArgTypes()
	Print a.Name()
Next

Print "done"



Derron(Posted 2016) [#57]
'finnish
-> finish :-)

Type TMethod
Method Invoke:Object( obj:Object,args:Object[] )
-> Method Invoke:Object( obj:Object,args:Object[] = Null )

TFunction and TField already use "= Null", so adding it is nice for consistency.

=== SNIP ===
Now to the important: it works (does not crash).


Do you have an explanation why it did not crash without that module? Why is it using some reflection stuff of that module even if I do not use it?


Thanks for "not being able to sleep" (and solving another issue) :-)


bye
Ron


grable(Posted 2016) [#58]
Do you have an explanation why it did not crash without that module? Why is it using some reflection stuff of that module even if I do not use it?
Because when reflection is first used, it goes through ALL registered types and adds them internally to a TMap.
At which point it happened upon a Type with a Method with Var parameters.

I guess Var parameters arent used much on Methods elsewhere, but SRS.D3D11Max2D has one in TBatchImage with 4 of them ;)

Fixed the typo (again! lol) and added Null as suggested.

Thanks for "not being able to sleep" (and solving another issue) :-)
Hehe, no problem :)


grable(Posted 2016) [#59]
Version 1.21 - fixed _Push not setting bbEmptyArray for Null arrays.


Derron(Posted 2016) [#60]
Updated my Pull Request (Brucey's Maxmod: github.com/maxmods/brl.mod) to incorporate 1.21.


bye
Ron


grable(Posted 2016) [#61]
Version 1.22 - fixed _Call not working with Long/Double return values.

Weird that i (or anyone else) havent noticed this before, so to be sure. Run this before trying 1.22 to verify that it really is bugged:



grable(Posted 2016) [#62]
Version 1.23:
Added _bbCallMethod() asm function for calling with proper number of arguments.
Increased maximum argument count to 30.

I was inspired by this post to finally add proper calling :)


Derron(Posted 2016) [#63]
@Test "long/double":
segfaulting with current implementation (v1.21)


Updated reflection code (all 2 files + 1 new file).
./bmk makeapp -t console -a -r -x "/testcodes/reflectionExtendedTest.bmx"
[  6%] Compiling:reflection.cpp
[ 74%] Archiving:reflection.release.linux.x86.a
[ 99%] Processing:reflectionExtendedTest.bmx
ar: creating /BlitzMax/mod/brl.mod/reflection.mod/reflection.release.linux.x86.a
[ 99%] Compiling:reflectionExtendedTest.bmx.console.release.linux.x86.s
flat assembler  version 1.68  (32768 kilobytes memory)
3 passes, 5544 bytes.
[100%] Linking:reflectionExtendedTest
/BlitzMax/mod/brl.mod/reflection.mod/reflection.release.linux.x86.a(reflection.bmx.release.linux.x86.o): In function `_brl_reflection_TClass_SetClass':
(code+0xb7f): undefined reference to `bbCallMethod'
/BlitzMax/mod/brl.mod/reflection.mod/reflection.release.linux.x86.a(reflection.bmx.release.linux.x86.o): In function `_brl_reflection_TClass_SetClass':
(code+0xbaa): undefined reference to `bbCallMethod'
/BlitzMax/mod/brl.mod/reflection.mod/reflection.release.linux.x86.a(reflection.bmx.release.linux.x86.o): In function `_brl_reflection_TClass_SetClass':
(code+0xbcc): undefined reference to `bbCallMethod'
/BlitzMax/mod/brl.mod/reflection.mod/reflection.release.linux.x86.a(reflection.bmx.release.linux.x86.o): In function `_brl_reflection_TClass_SetClass':
(code+0xbf7): undefined reference to `bbCallMethod'
/BlitzMax/mod/brl.mod/reflection.mod/reflection.release.linux.x86.a(reflection.bmx.release.linux.x86.o): In function `_brl_reflection_TClass_SetClass':
(code+0xc21): undefined reference to `bbCallMethod'
/BlitzMax/mod/brl.mod/reflection.mod/reflection.release.linux.x86.a(reflection.bmx.release.linux.x86.o):(code+0xc4e): more undefined references to `bbCallMethod' follow
collect2: error: ld returned 1 exit status
Build Error: Failed to link /testcodes/reflectionExtendedTest


Did I miss something?


bye
Ron


grable(Posted 2016) [#64]
Hmm thats a bit weird. TClass is virtually empty, no references to bbCallMethod there. Only used in _Call 6 times.

The log doesnt show callmethod.s either, maybe it isnt picked up by bmk and then not compiled in?

To double check that, see if reflection.bmx.release.linux.x86.s has _bbCallMethod as an extrn at the top somewhere.

Other than that, maybe try a recompile of the mod alone.


Derron(Posted 2016) [#65]
reflection.bmx.release.linux.x86.s:
	format	ELF
	extrn	__bb_blitz_blitz
	extrn	__bb_linkedlist_linkedlist
	extrn	__bb_map_map
	extrn	bbArrayNew1D
	extrn	bbArraySlice
	extrn	bbCallMethod
	extrn	bbEmptyArray
...


compiled the module via:
./bmk makemods -a brl.reflection

it compiled flawlessly


removed all .bmx stuff and i/a files.
re-compiled your sample (which should rebuild everything if needed):
./bmk makeapp -t console -a -r -x "/testcodes/reflectionExtendedTest.bmx"
[  6%] Compiling:callmethod.s
flat assembler  version 1.68  (32768 kilobytes memory)
1 passes, 170 bytes.
[  6%] Compiling:reflection.cpp
[ 69%] Processing:reflection.bmx
[ 72%] Compiling:reflection.bmx.release.linux.x86.s
flat assembler  version 1.68  (32768 kilobytes memory)
5 passes, 38112 bytes.
[ 74%] Archiving:reflection.release.linux.x86.a
[ 99%] Processing:reflectionExtendedTest.bmx
ar: creating /BlitzMax/mod/brl.mod/reflection.mod/reflection.release.linux.x86.a
[ 99%] Compiling:reflectionExtendedTest.bmx.console.release.linux.x86.s
flat assembler  version 1.68  (32768 kilobytes memory)
3 passes, 5544 bytes.
[100%] Linking:reflectionExtendedTest
/BlitzMax/mod/brl.mod/reflection.mod/reflection.release.linux.x86.a(reflection.bmx.release.linux.x86.o): In function `_brl_reflection_TClass_SetClass':
(code+0xb7f): undefined reference to `bbCallMethod'
/BlitzMax/mod/brl.mod/reflection.mod/reflection.release.linux.x86.a(reflection.bmx.release.linux.x86.o): In function `_brl_reflection_TClass_SetClass':
(code+0xbaa): undefined reference to `bbCallMethod'
/BlitzMax/mod/brl.mod/reflection.mod/reflection.release.linux.x86.a(reflection.bmx.release.linux.x86.o): In function `_brl_reflection_TClass_SetClass':
(code+0xbcc): undefined reference to `bbCallMethod'
/BlitzMax/mod/brl.mod/reflection.mod/reflection.release.linux.x86.a(reflection.bmx.release.linux.x86.o): In function `_brl_reflection_TClass_SetClass':
(code+0xbf7): undefined reference to `bbCallMethod'
/BlitzMax/mod/brl.mod/reflection.mod/reflection.release.linux.x86.a(reflection.bmx.release.linux.x86.o): In function `_brl_reflection_TClass_SetClass':
(code+0xc21): undefined reference to `bbCallMethod'
/BlitzMax/mod/brl.mod/reflection.mod/reflection.release.linux.x86.a(reflection.bmx.release.linux.x86.o):(code+0xc4e): more undefined references to `bbCallMethod' follow
collect2: error: ld returned 1 exit status
Build Error: Failed to link /testcodes/reflectionExtendedTest



bye
Ron


grable(Posted 2016) [#66]
Aha, i forgot about ELF ;) The .s file is using "MS COFF".
Im surprised fasm and bmk doesnt complain about it though, they really should!

Anyway, try version 1.24 with own files for linux and win32.


Derron(Posted 2016) [#67]
What's up with MacOS - special handling needed too?


Linux still fails:
./bmk makeapp -t console -r -x "/testcodes/reflectionExtendedTest.bmx"
[  6%] Compiling:callmethod.linux.x86.s
flat assembler  version 1.68  (32768 kilobytes memory)
/BlitzMax/mod/brl.mod/reflection.mod/callmethod.linux.x86.s [6]:
section	"code" code
error: invalid argument.
Build Error: Failed to assemble /BlitzMax/mod/brl.mod/reflection.mod/callmethod.linux.x86.s



bye
Ron


grable(Posted 2016) [#68]
Damn fasm is fickle ;) Yeah, il add one for macos as well. Even though i have to use AT&T syntax :(

Updated top post with version 1.25, hopefully this is it :p


Derron(Posted 2016) [#69]
Back to the error in #65

(removed module's .i/.a files, .bmx directory, and precompilates of the test sample before compiling it again)


bye
Ron


grable(Posted 2016) [#70]
Hmm seems linux doesnt like underscores or something, or it adds them itself.
Anyway, remove the underscores from all the labels (including _memcpy) and it should compile.

Had to install linux in a vm to test ;) MacOS is still untested.

Updated top post.


Derron(Posted 2016) [#71]
Works now.
Seems you were right with the underscore.


@Mac
There are options to install a Mac OS X in a VM too..


bye
Ron


Derron(Posted 2016) [#72]
The versions starting with 1.14/1.15 lead to a bugged BMK (Brucey's BMK).

This BMK calls some Lua files (.bmk) - but these now fail with "arg0" being invalid data - so passing things to lua is bit of broken.

Do not know what exactly, as my game's lua-scripts work as intented (think so at least).


bye
Ron


grable(Posted 2016) [#73]
I assume its the one from here https://github.com/bmx-ng/bmk ..

I am unable to reproduce it though, arg0 contains valid data for assemble, compileBMX and compileC (arg0 in those contains the entire commandline).

In what instances does this happen, and what platforms?


Derron(Posted 2016) [#74]
Using Brl.mod from
https://github.com/maxmods/brl.mod
and Pub.mod from
https://github.com/maxmods/pub.mod

Copied your Brl.reflection-code from the first post (reflection.bmx, reflection.cpp, callmethod.linux.x68.s). Recompiled modules via "bmk makemods brl" (recompiled a bunch of modules, as brl.blitz is recompiled too (base for many things).


Then I recompiled BMK with vanilla BCC and the modified Brl.reflection.
Put BMK into another BlitzMax-Directory (BlitzMaxNG-overload) and when trying to execute a compilation (eg. modules) I get this:

$ ./bmk makemods
ERROR
[string "function bmk_source_type(...)..."]:5: attempt to concatenate local 'arg0' (a userdata value)


This errors is based on a .bmk-file executed before compilation - saying "arg0" is invalid data.


OS: Linux 64Bit.


As "callmethod***" is introduced after it started getting "borked", I assume it must be something different.


bye
Ron


grable(Posted 2016) [#75]
There is only 4 places where it spits out "ERROR", in _Push(), _Assign() and two in _Call().
Im guessing its one of them in _Call(), both of them are thrown when the number of arguments is exceeded.
Though with a maximum of 30 arguments that is a bit weird...

The reason for the invalid data in arg0 is probably because it throws that exception and skips some init code related to the lua call.

Could you number the throws in reflection.bmx to narrow it down perhaps?

Il try to narrow it down in any case, but it might take me a while to reproduce.


Derron(Posted 2016) [#76]
I tried that already (Error1 - Error4) but it keeps saying "ERROR" (no number).

I rebuilt all modules afterwards (and manually deleted the modules precompilates), rebuild BMK completely ...

- removed brl.mod completely, unzipped a new one, pasted reflection code into it, adjusted "Error" to "errorNumber" and recompiled BMK ... message is "ERROR".

Maybe "Error" is the LUA message, not the message from reflection.

I think this is the case and the error is, that "arg0" is kind of "nil" (Lua-"Null") which disallows access to it.


My game's Lua-code might run without trouble as I do not use MaxLua but a custom implementation with some extensions/differences.


Edit: maxlua.bmx:
Function LuaDumpErr()
	WriteStdout "ERROR~n"
	WriteStdout lua_tostring( LuaState(),-1 )
End Function


This is where the ERROR comes from.


bye
Ron


grable(Posted 2016) [#77]
Ah, my bad :p

I did the same as in your recipe, and got an access violation when calling RunCommand (in bmk)
And then remembered that i had to modify brl.maxlua for the changed ReturnType() thing, and after that everything went as normal.

This is on Windows though, il try the same dance in Linux too.


Derron(Posted 2016) [#78]
So would you mind sharing what you have changed in maxlua? :-)


I assume this:
Function Invoke( L:Byte Ptr )
	Local t:Object=meth.Invoke( obj,args )
	Select meth.TypeId()
	' becomes
	Select meth.ReturnType()
...


This explains why my lua-engine still works - I use this code line:
Local typeId:TTypeId = funcOrMeth.TypeID().ReturnType()


bye
Ron


grable(Posted 2016) [#79]
Its in the top post.. Line 77 of maxlua.bmx, same change as yours.

EDIT: Just tried it on linux, still not able to reproduce :(


Derron(Posted 2016) [#80]
As soon as I adjusted that maxlua line, compilation with the newly compiled BMK was possible again (no LUA-error anylonger).

Thanks for your help.


bye
Ron


grable(Posted 2016) [#81]
Ah, good to know, no problem dude :)


Derron(Posted 2016) [#82]
Not a bug, but a question:

Do you see any chance to make that "ReturnType()" portion compatible to the original mod?

When trying out things with Bruceys NG (and the NG-modules) I used the brl.reflection of NG as it contained Method/Function/Field/Const already (and your code contained some things not compatible with NG/overload).

It compiled fine but the issue was, that my lua-engine relied on

Local typeId:TTypeId = funcOrMeth.TypeID().ReturnType()

which resulted in different types in your code file and Bruceys brl.reflection.


So the question is based on the idea of keeping things "compatible" as much as possible. It would also get rid of the needed modification of brl.maxlua.


Thoughts?

bye
Ron


grable(Posted 2016) [#83]
Well, since you have control over your own lua glue you could use the ?bmx-ng directive and revert to original behavior inside that.

There is a reason for the change though, because of Function Types.
Notice that TypeId() is implemented in TMember, and it returns the full typeid.
So a TField can contain a FunctionType, and that type can have a return type which is also a FunctionType etc.

I see it as a limitation of the original BRL.Reflection being incomplete in a way, not only missing types but also overloading the TypeId() method to mean different things. It really should have had a ReturnType() instead :p

I dont want to override it for TMethod and TFunction to be honest, as that way you cant treat a TMember and its TypeId() uniformly.

But since this modification is more or less complete (barring any new bugs) overriding them yourself isnt that much of a chore..

All you would need is adding these two methods to TMethod and TFunction.
	Method TypeId:TTypeId()
		Return ReturnType()
	End Method

	Method FunctionTypeId:TTypeId()
		Return _typeId
	End Method
Note that i have not tested this ;)

reflection.bmx with original semantics.



Derron(Posted 2016) [#84]
Missed your reply - so excuse the delay.

I understand your reasons on why you did it that way.
Of course I am using "?bmxng"-conditionals, but the problem is, that once, bmxng adjusts its behaviour to "yours", then the application needs manual adjustment (if you miss removing that conditional adjustments, it breaks logic/bugs out).

Hmpf.

Thanks again for your time.


bye
Ron


Derron(Posted 2016) [#85]
Just recognized, that invoking functions does not work as expected:

SuperStrict

Framework Brl.StandardIO

Import Brl.Reflection 'extended variant


Type Test
	Function Func:Int()
	End Function
End Type


Local tID:TTypeId = TTypeId.ForName("Test")

Local func:TFunction = tID.FindFunction("Func")

'segfaults in "invoke"
func.Invoke(Null)

Print "invoked"



When adding a "debugstop" and then going into all called functions it segfaults when calling the external function "bbRefMethodPtr" (because "obj" is null)

	Method FunctionPtr:Byte Ptr( obj:Object)
		If _fptr Then Return _fptr
		If _index < 65536 Then
~~~~>		_fptr = bbRefMethodPtr( obj ,_index)
		EndIf
		Return _fptr
	End Method


So the problem is that I pass "null" to invoke (as the obj-param) ... but am I really supposed to pass an instance of the type there? Functions are meant to be callable also via "TMyClass.FunctionName()".

Replacing that "func.Invoke(null)" with an existing type-instance as param, gets rid of the segfault...

'works
Local t:Test = New Test
func.Invoke(t)



Invoking a method works.

bye
Ron


grable(Posted 2016) [#86]
Thanks, its fixed in v0.26 by using the Self TypeId class directly on Null arguments.


Derron(Posted 2016) [#87]
Thanks for the update.


bye
Ron


Derron(Posted 2016) [#88]
Your code does not compile with my Mac OS X...

the *.x86.s contains invalid characters - also things like ".global" do not exist (you need to use ".globl") - and ".extern" seems to be not needed.


bye
Ron


grable(Posted 2016) [#89]
Thanks for the heads up, without a mac its hard to test these things.

The invalid characters might be CRLF? Since i develop on windows, my editors use that as EOL, and its not UTF-8 either, which both linux and macosx might expect.

But that AT&T syntax really sucks :( Some docs say ".global" other say ".globl", other say they are aliased. But apparently its up the the platform??
And I cant get it to assemble no matter what i do either...
Usually one uses size suffixes, but it complains when i do. And when i dont it also complains, so i am at a loss here.
Stupid me forgets i have 64bit mingw now ;)

Does it work just with changing to .globl for you?

In any case, i updated it with your change at least. But im leaving .extern in to make it similar to the others, and just because its the right thing to do ;)

EDIT: Oh and btw, what version of AS are you using? Since 2.10 they added .intel_syntax directive so i can probably use that instead.
EDIT2: I re-uploaded the archive with linux and macosx files in UTF-8 and with only LF chars.


Derron(Posted 2016) [#90]
I use a Mac in a VM ... so it is outdated (but used to work).


as -v
Apple Inc version cctools-750-70, GNU assembler version 1.38



Stored with LineEnding "CR" ("LF" is Linux, "CR/LF" is Windows) and it compiled with

.globl _bbCallMethod
_memcpy

.text
	
_bbCallMethod:
	push %ebp
	mov %esp, %ebp
	sub 16(%ebp), %esp
	push 16(%ebp)
	push 12(%ebp)
	push %esp
	call _memcpy
	add 4, %esp
	call *8(%ebp)
	mov %ebp, %esp
	pop %ebp
	ret


Then on linking:
Undefined symbols:
  "_bbCallMethod", referenced from:
      _814 in reflection.bmx.release.macos.x86.o
      _816 in reflection.bmx.release.macos.x86.o
      _808 in reflection.bmx.release.macos.x86.o
      _809 in reflection.bmx.release.macos.x86.o
      _810 in reflection.bmx.release.macos.x86.o
      _811 in reflection.bmx.release.macos.x86.o
ld: symbol(s) not found



PS: Excuse my request without a solution - was prepared to release a new version of my open source game ... and then it did not compile for Mac ... fiddling with little hotfixes for 3 hrs now ...arrrgh ;-)


bye
Ron


grable(Posted 2016) [#91]
Apple Inc version cctools-750-70, GNU assembler version 1.38

Ah yes, the things ive heard about seriously outdated gnu tools on mac isnt an exaggeration i suppose.

Im surprised it compiled with only CR, so far as i know OSX follows its BSD roots and uses LF like linux does.

Excuse my request without a solution - was prepared to release a new version of my open source game ... and then it did not compile for Mac ... fiddling with little hotfixes for 3 hrs now ...arrrgh ;-)

No problem dude. Its my baby so i have to keep it well fed :p
But it seems your the only one using it though hehe


Derron(Posted 2016) [#92]
Any ideas about the linking issue?

Is there a real need for the ASM files at all? I mean: could that get achieved with c/bmx only?

Bye
Ron


grable(Posted 2016) [#93]
Any ideas about the linking issue?

Oh, sorry. I missed that.

Its probably the number of underscores before the symbol. Windows tends to always have one, the others not so much.

You can look at the generated S files on mac and see what bmx spits out, or try removing/adding underscores.

Allso, after trying it on here it crashes. Which is probably because i didnt use size suffixes so it inferred the wrong size.
I disassembled the win32 code, and saw that it really did use them, so i did too and it now works here (on Windows, but still)
So try using this snippet until it compiles.
.globl bbCallMethod
.extern _memcpy

.text

bbCallMethod:
	push   %ebp
	mov    %esp,%ebp
	sub    0x10(%ebp),%esp
	pushl  0x10(%ebp)
	pushl  0xc(%ebp)
	push   %esp
	call   _memcpy
	add    $0x4,%esp
	call   *0x8(%ebp)
	mov    %ebp,%esp
	pop    %ebp
	ret
Note the removed underscore, this wont compile on windows.

When you figure out the underscore thing il update the top post :)

EDIT:
Is there a real need for the ASM files at all? I mean: could that get achieved with c/bmx only?

Well, sure. The original did just that, but it was limited by number of arguments and it was probably a bit slower. Though i havent benched it.
ASM is needed in this case to support unlimited arguments.


Derron(Posted 2016) [#94]
I tried your code... and it compiled ... but the linker error kept the same.

So while I removed the underscore from "bbCallMethod" (two times) and _REBUILT_ my application (so no cache usage) the error kept saying

Undefined symbols:
  "_bbCallMethod", referenced from:
      _814 in reflection.bmx.release.macos.x86.o
      _816 in reflection.bmx.release.macos.x86.o
      _808 in reflection.bmx.release.macos.x86.o
      _809 in reflection.bmx.release.macos.x86.o
      _810 in reflection.bmx.release.macos.x86.o
      _811 in reflection.bmx.release.macos.x86.o
ld: symbol(s) not found

(same line numbers than before)


reflection.bmx.release.linux.x86.s only contains "bbCallMethod", no underscore.



bye
Ron


grable(Posted 2016) [#95]
Hmm..

After looking over how BRL.Blitz does it, it uses underscore for macos so i was wrong on that one.
What i did notice though was that Blitz.bmx always uses the extern name with no underscores, so maybe that will work.

Change line 125 of reflection.bmx to:
Function bbCallMethod:Int( p:Byte Ptr, args:Byte Ptr, sz:Int) = "bbCallMethod"
EDIT: And add the underscore back in to the one i posted above..


Derron(Posted 2016) [#96]
That did not do it.

So I build up a small example for the reflection stuff (my game just takes too long to compile in the Mac VM).


- switching back to LF-LineEnding (Unix-Style) (required, CR does not work)
- removed the "_memcpy" (invalid character '_' in mnemonic)
- added back underscores

-> compiled and executed


I think ".extern _memcpy" is not needed, as "_memcpy" is automatically "externed".



.globl _bbCallMethod

.text

_bbCallMethod:
	push %ebp
	mov %esp, %ebp
	sub 16(%ebp), %esp
	push 16(%ebp)
	push 12(%ebp)
	push %esp
	call _memcpy
	add 4, %esp
	call *8(%ebp)
	mov %ebp, %esp
	pop %ebp
	ret



reflection.bmx kept as before:
Function bbCallMethod:Int( p:Byte Ptr, args:Byte Ptr, sz:Int)
Function bbCallMethod_Float:Float( p:Byte Ptr, args:Byte Ptr, sz:Int) = "bbCallMethod"
Function bbCallMethod_Object:Object( p:Byte Ptr, args:Byte Ptr, sz:Int) = "bbCallMethod"
Function bbCallMethod_Double:Double( p:Byte Ptr, args:Byte Ptr, sz:Int) = "bbCallMethod"



Edit: My Game now crashes when going into a specific situation. Will check in the evening if it has to do with the new reflection stuff (debug build surely takes some time ...time I do not have now).


bye
Ron


grable(Posted 2016) [#97]
Weird... At least it compiles now :)

But you need to use the one with size suffixes, or it will read/write bytes instead of longs.

.globl _bbCallMethod

.text

_bbCallMethod:
	pushl   %ebp
	movl    %esp,%ebp
	subl    0x10(%ebp),%esp
	pushl  0x10(%ebp)
	pushl  0xc(%ebp)
	pushl   %esp
	calll   _memcpy
	addl    $0x4,%esp
	calll   *0x8(%ebp)
	movl    %ebp,%esp
	popl    %ebp
	ret


Updated top post with the new code as well.

EDIT:
I think ".extern _memcpy" is not needed, as "_memcpy" is automatically "externed".

Its a GCC builtin, which is probably why you dont have to define it. Still its the right thing to do ;) declaring intent and all.


Derron(Posted 2016) [#98]
@ extern not needed
It is not "allowed" in that ancient build chain. So I did not "optionally" remove it, but was forced to do so.


Recompiled with your adjustments (in "debug mode"):

"TVTower.debug wurde unerwartet beendet." - think it is telling something in the likes of "unconditionally exited the application". So even the debugger did not "hop in".


Running it via "GDB" saays

Program received signal EXC_BADACCESS, Could not access memory.
Reason: KERN_PROTECTION_FAILURE at address: ...

the backtrace then contains
"in std::list<IMaxModChannel*, std::allocator<IMaxModChannel*> >:: ..."

So it somehow crashes when doing something with MaxMod2 (audio streaming module). But I doubt it has to do something - did not change that code portions the last days.


Ok, so I replaced the reflection stuff with a variant "before ASM" ... compiled and run the game without that crash.


bye
Ron


grable(Posted 2016) [#99]
Well thats a bummer. The code is identical for all 3 platforms, but there may be some stack layout or other im not aware of on MacOSX (or Linux for that matter).

Hmm weird that it should propagate into there, but if there is some stack misalignment that would do it.
Il have to look into this...

At least you only have to replace _Call() with an earlier version.

Are you compiling this on Linux as well btw?


grable(Posted 2016) [#100]
You can try the code below. Which aligns the arguments and the stack on 16 a byte boundary (with some waste of course) for MacOSX only.

.globl _bbCallMethod

.text

_bbCallMethod:
	pushl %ebp
	movl %esp, %ebp
	andl $0xFFFFFFF0, %esp
	subl 0x10(%ebp), %esp
	pushl 0x10(%ebp)
	pushl 0xc(%ebp)
	pushl %esp
	calll _memcpy
	addl $0x4, %esp
	calll *0x8(%ebp)
	movl %ebp, %esp
	popl %ebp
	ret



Derron(Posted 2016) [#101]
Compiled with your adjustments. Compilation successful, game starts ... crashes on a specific action (going to next screen)

It must be some "specific thing". I also tried to removed "play sound"-parts but it still crashes.


@ Linux
Yes, my main development takes place on a Linux Mint 64 bit installation.
Windows is compiled via MicroXP-VM and Mac via a Mac OS X VM.
(Windows then could be tested on winXP, 7, 8 and 10 - for now I only had trouble with win10 and some graphics thingies)



Edit: I also tried the "demoapp"-sample of my framework Dig
(http://github.com/GWRon/Dig.git) and when replacing the "external/reflectionExtended"-portion with yours - aka "updating your reflection stuff") it crashes too - as soon as I close the modal dialogue (some gui stuff).
So it seems to be not connected to sound stuff.



bye
Ron


grable(Posted 2016) [#102]
Hmm, if that didnt do it i dont know what will. I havent found any other "rules" for asm on macs either.
At least its only macs that are fussy, the rest are ok.

So for now, im going to revert back to the old way without assembly for macs.


Derron(Posted 2016) [#103]
Excuse me for not being able to solve that puzzle with you.

ASM and memory/internal stuff is way over my head.


bye
Ron


grable(Posted 2016) [#104]
Hehe no problem dude :)

I will look into it some more though, whenever i get osx running in virtualbox.
Tried the 2 latest versions last year, but they both failed. Maybe itl work now, who knows :p


grable(Posted 2016) [#105]
I wasnt aligning the stack properly, it needs to be aligned before each call apparently.
If you still want a go at this, replace the code in v1.27 with the one below.
.globl _bbCallMethod

.text

_bbCallMethod:
	push   %ebp
	mov    %esp,%ebp
	mov    %esp,%eax
	sub    0x10(%ebp),%eax
	and    $0xfffffff0,%eax
	mov    %eax,%esp
	push   $0x0
	pushl  0x10(%ebp)
	pushl  0xc(%ebp)
	push   %eax
	call   _memcpy
	add    $0x10,%esp
	call   *0x8(%ebp)
	mov    %ebp,%esp
	pop    %ebp
	ret