SCCM collection Query do not have particular exe installed

Use Subselected query to get the results:

select *  from  SMS_R_System where SMS_R_System.NetbiosName like "%rts%" and SMS_R_System.ResourceId not in (select SMS_R_System.ResourceID  from  SMS_R_System inner join SMS_G_System_SoftwareFile on SMS_G_System_SoftwareFile.ResourceId = SMS_R_System.ResourceId where SMS_G_System_SoftwareFile.FileName like "rtvscan.exe" AND SMS_R_System.Client = 1)