Energy Trading Technology

Saturday, February 29, 2020

Hifax: Hierarchically Flexible Access Control Scheme

1. Introduction

Any non-trivial computer system needs to control the access to the functionality and the services it provides and has to be able to regulate what its users can or cannot do with and within the system. Controlling users access to a system — through careful management of access rights given to them — is generally known as access control. Any design or methodology how this can actually be achieved, therefore, can be called as an access control scheme.

A good access control scheme should be very flexible both in terms of features(i.e. capabilities) it supports and also the ease with which it can be deployed and used. Features flexibility refers to the scheme allowing its administrators to manage permissions (i.e. what can or cannot be done) by a specific user or a group of users at a sufficiently detailed level. Though features flexibility is obviously a desired attribute in an access control system, greater flexibility may bring greater complexity and operational risk if the additional flexibility is something the users of the system cannot opt out easily — assuming they choose to do so — and are forced to adapt a "straight jacket" solution.

We refer to the ability to "pick-and-choose" the set of features of an access control scheme — such that the complexity and the cost of an implementation can be tuned by the set of advanced features actually being used — as deployment flexibility. A good access control scheme, therefore, is one which possesses both features and deployment flexibilities. It should allow detailed control over various aspects of the system if needed (e.g. user and group management, permissioning, etc.,) but it should also be possible to be used in a relatively simple fashion — for example just out of box deployment — by applications that don’t require fine access control.

In this paper, we describe the design of an access control scheme called Hifax (Hierarchically Flexible Access Control Scheme). Hifax aims to be very flexible both in terms of features it supports and the ease with which such features can be deployed and used. The design of Hifax is such that it gives system administrators fine control over managing user access permissions while at the same time allowing them to deploy it relatively easily and effortlessly in non-sophisticated deployment scenarios. By employing techniques such as hierarchical grouping and inheritance, the design of Hifax allows fine control over the entitlement management process (managing groups, roles, permissions, resources, etc.,) and ensures that the deployment complexity increases gradually only as more and more features are used.

2. Concepts

In this section, we describe the fundamental concepts and abstractions used in the design of Hifax. These concepts manifest themselves as the domain objects in the design of Hifax and directly follow from the observations from the problem domain (i.e. access control management)

2.1. User

A user is an actor that is going to "use" the system. A user could either be a human operator or a non-human entity (e.g. a bot, a process, etc.). Information about users is usually kept in an enterprise-wide directory system like Active Directory.

2.2. Group

Users can be members of the various groups. This helps with managing users who should be treated in the same way for a specific purpose like access control, system login expiry management, etc.,

Groups are hierarchically structured as follows:

Subgroups: A group is composed of zero or more users and zero or more other groups called subgroups. In other words, groups are hierarchical in nature. There is a parent-child relationship between a group and its subgroups where a subgroup is considered as a parent of the group that includes it. Conversely, the group that includes subgroups is considered as a child of them.
Sharing: A group could be a subgroup of many different groups. In other words, subgroups are shared, not owned, by the child group.
Recursive Inheritance: A group which includes another group indirectly (i.e. thorough its subgroups and their subgroups and so on) is said to be that group’s grandchild. The included group is said to be the grandparent of the including group.
Non-cyclicalness: A group cannot be its own grandparent. In other words, a group cannot be included in any of its subgroups as a child. This prevents cycles in group definition.

Banning: Just like users are added to a group, they could also be banned from it. Banning results in a user being removed from a group it is somehow — directly or indirectly through group inheritance — included in. Banning allows fine tuning of the group user membership.

Group User Graph: There is a special group called RootGroup which is included by any other groups directly or indirectly. Users and groups form a directed acyclic graph (DAG) originating from RootGroup called Group User Graph (GUG).
GUG Nodes: Each node of GUG is either a group or a user. User nodes are the leaf nodes in the graph and they are where any traversal from a root node will ultimately end at.

GEUS: The final set of users in a given root group — considering all the users in its subgroups recursively — is called group effective user set (GEUS).

2.3. Resource

A resource is any part of a system over which we want to impose some level of access control. A resource could be anything of significance in a system. Some examples of typical resources in a given computer system are:

Functions or segments of the code to execute in the system
GUI elements like menus to access, forms to fill in, buttons to click, etc.
System-wide operations like login, print, networking, etc.
Persistent entities like deal information, market data, user information, etc.
Commands or functions in the system (e.g. API elements)

Level of granularity used to designate resources depend entirely on the level of access control required. Finer granularity will enable finer access control but will come at the expense of increased complexity and higher cost of access control implementation.

The list of resources defined in a system forms an inventory as far the access control is concerned. Once the resources are defined, the rest of the system operates to control how and by whom those resources can be accessed.

Resources are named with a hierarchical naming convention to help with managing access to resource hierarchies using pattern matching techniques. This way, for example, the resource identifier API.Sales.* could refer to the entire set of API functions provided for the sales module of a business application.

2.3.1. Resource Parameters

Each resource is associated with a set of name-value pairs called resource parameters. Some of the resource parameters are static (i.e. known at development time). Such parameters will be specified by hard-coding them in the source code. An example for static parameters is resource name: every resource has a name and a resource’s name is known at development time.

There could be other parameters of a resource that are only known at runtime. Such parameters are called dynamic parameters. As an example, consider a resource which corresponds to a data record representing customer information in a sales management module of a business application. The values of various fields like enrollment_date, company_name, geographical_region, etc., will only be known once the record is loaded at runtime. Notice that although we know that the resource has fields such as a company_name at development time, we cannot know the values any particular customer record can take for those fields and hence if we want to a have an access control scheme where we impose restrictions based on the value of those fields, we have to rely on the dynamic parameters features of the resources.

2.4. Operation

Operations refer to the way a given resource is used in the system. There are five operation types:

Create: Operations involving adding a new item in the system (e.g. a new record, new definition, etc.).
Read: Operations involving accessing, reading and listing data entities.
Update: Operations involving changing data and entities in the system.
Delete: Operations involving deleting data, removing objects, etc.
Execute: Any operations involving some sort of execution (running functions, accessing APIs, etc.)

These operations are known as CRUDE. You will notice that the first four of them refer to persistence operations over database records or files, whereas the last one refers to executing code/functions in a system and accessing (viewing) user interfaces.

2.5. Permission

A permission defines the set of operations that can be carried out on either a specific resource or a family of resources as described a by resource regex. Permissions act like the link between resources and operations to be carried out on them.

Note that the access enforcing code (e.g. an API resource) does not know anything about permissions. Permissions are access control management abstractions. Enforcing code will only know about the resources it is guarding and the operations that are going to be attempted on them in the code that immediately follows, both of them being very related to and confined to the part of a system that is needed to be access controlled. A corollary to this fact is that permissions can be managed — i.e. new permissions being added, existing ones modified to change their resource and operations scopes, etc. — independently of the actual system that is access controlled. This enables a good degree of operational flexibility as far as access control is concerned.

A permission could either be given to (i.e. granted) or taken from (i.e. revoked) a user or a group.

Permissions are defined by specifying the following:

Name: Each permission is uniquely identified by a name string. Just like resources, permissions should be named using a hierarchical structure so that pattern matching techniques could be used to refer to a collection of them in one go.
Operations: A combination of C/R/U/D/E operations are that to be carried out on a resource. Operation types can be represented by different bits in a byte and hence can be combined into a byte to signal multiple operation types. For example, suppose we represent operation types as follows:
- int OP_CREATE = 0x01
- int OP_READ = 0x02
- int OP_UPDATE = 0x04
- int OP_DELETE = 0x10
- int OP_EXECUTE = 0x20
  Some of the composite operations that can be specified using the above encoding are:
  - 3 ⇒ CR operations (OP_CREATE || OP_READ)
  - 15 ⇒ CRUD operations (OP_CREATE || OP_READ || OP_UPDATE || OP_DELETE)
  - 31 ⇒ CRUDE operations (OP_CREATE || OP_READ || OP_UPDATE || OP_DELETE || OP_EXECUTE)
Resource Regex: The regular expression string that corresponds to zero or more resources over which a permission will be granted or revoked. Recall that resources are named using hierarchical, URL like strings. By combining hierarchical nature of the resource name with the flexibility provided by regular expressions, we can refer to an entire family of resources with one string!
Control Script: This is an optional string composed of an expression which can be evaluated to True or False . Hifax will assume that it evaluates to True if it is not provided, hence making it effectively optional. Control script will have access to contextual objects like current user, resource, etc.,

2.5.1. Permission Context

Any resource access is controlled by checking that the user has the necessary permissions to the resource in the question. These permission checks can be carried out using an object called permission context that provides the following information to the access control enforcing code:

Principal: The user who tries to access the resource. It is likely to be the logged-in user in the system.
Resource: Resource to be accessed. The checking code will be able to access all the parameters of the resource (both static and dynamic ones)
Operations: Operations to be carried out. This will be a combination of C/R/U/D/E operations on a given resource.

Above three pieces of key contextual information — principal(p)/resource(r)/operation(o) — is known as PRO.

2.5.2. Control Script

Control script of a permission allows fine tuning of access control process mechanism based on dynamic parameters of resources to be controlled. For example, suppose we want only users with the role "IBXTraders" to access deal data whose counterparty is "IBXBank". This can be expressed as the following using the following hypothetical control script: HasRole(p.username, "IBXTraders") and (r.counterparty == "IBXBank").

Without delving into too much detail, it is important to observe few things in the example above:

Context objects such as principal (p) and resource (r) have attributes that can be used in permission checking code
Complex expressions can be composed of using Boolean operators (and, or, xor, etc.) and parenthesis.
Some built-in functions such as HasRole is provided by Hifax runtime that could be useful in permission checking code

2.6. Role

A role is just a set of permissions. However, like groups, roles are hierarchically structured and include other roles as subroles.

Roles and permissions form a directed acyclic graph called RPG (Role Permission Graph) whose root node is a special role called RootRole.

2.7. User/Group and Permission/Role Isomorphism

Hierarchical structuring that governs the relationship between users and groups — as discussed in the section Group — also applies to the relationship between permissions and roles. In other words, roles are just hierarchically organized collections of permissions and supports grouping (i.e. subroles), sharing, recursive inheritance (i.e. ancestral roles), non-cyclicalness and banning (i.e. disallowing a permission).

The final set of permissions in a given role including all the subroles recursively is called role effective permission set (REPS). Notice how REPS is defined in a similar vein as its dual GEUS.

3. Principles

Having discussed the core concepts in the access control domain, we are ready to state the founding principles of Hifax. There are four fundamental architectural pillars over which Hifax stand on:

Flexible user management: Deals with the users (i.e. actors)
Flexible resource management: Deals with the resources (i.e. objects)
Flexible permission management: Deals with the permissions (i.e. actions)
Declarative access control enforcement: Deals with enforcing access control on the resources in a straight-forward way

We will explain these principles within the context of a concrete example. We will use a fictional business system called X1Sys. X1Sys is a business application consisting of two modules (i.e. areas of functionality):

Sales: Deals with customers and selling products to them
Accounting: Keeps tracks of business transactions.

We have created some fictional data to use to illustrate the implementation of Hifax for a basic system like X1Sys.

3.1. Flexible User Management

Hifax supports a very flexible and highly scalable user management thanks to leveraging the group user graph to manage users and their groups.

We will start with showing the list of users we have for X1Sys:

Figure 1. Users of X1Sys

We have users from three major departments in the organization: Accounting, Sales and IT. We can place these users into hierarchical groups to better manage them for access control — among other — purposes. Below you will find such a group organization:

Figure 2. Group User Graph of X1Sys

This diagram consists of two parts: implicit and explicit display of group/user membership for X1Sys. On the left side, you see the definition of groups and on the right hand side you see the effective user list of each group. Notice the following about this diagram:

Each group has a name and this is shown at the top of the group box in bold e.g. IT_Admins.
Each group could have directly added or removed user members. When a user is added, this is shown by prefixing the user’s username with + sign, whereas when it is removed (i.e. banned) this is shown with - prefix.
A group can inherit users from zero or more parent groups. Conversely, a group itself could be a parent of zero or more groups. Parent-child relationship is also known as subgroup-group relation. In the figure, IT_Admins is a subgroup of both Acct_Admins and Sales_Admins groups. In other words, IT_Admins is a parent of those two groups. It is also said to be a grandparent of Sales_Users.
On the right hand side of the figure, you see the effective user sets of each group. These are the list of users that are composed after taking group inheritance and direct membership precedence rules in consideration. The algorithm for this is given in the section Group Effective User Set (GEUS) Computation.

3.2. Flexible Resource Management

Anything could be a resource in a Hifax implementation and resources are named using hierarchically structured URL-like strings. This allows referring entire families of resources using regular expressions.

There are three major types of resources in X1Sys: database entities, UI screens and API functions. These define the families of resources during the resource definition process and influence the hierarchical naming convention when assigning resource identifiers to each resource. Below we show the resources defined for X1Sys:

Figure 3. Resources of X1Sys

As noted in the concept section Resource, the level of granularity used to designate resources is entirely arbitrary and determines the level of access control desired in the system. In this case, we have chosen a reasonably fine granular approach of controlling access at screens, API functions and database entities levels.

3.3. Flexible Permission Management

Hifax provides a very flexible and highly scalable permissioning thanks to leveraging the role permission graph to provide a powerful role-based security model. This model is based on the ability to assign both specific permissions and roles to either groups or users directly.

Just like users can be organized into groups, permissions can also be organized into hierarchical roles. An example of such an organization is shown in the figure below for X1Sys:

Figure 4. Role Permission Graph of X1Sys

You will see the implicit and explicit (i.e. exploded) views of a role permission graph (RPG) defined for X1Sys. Structural similarities between this picture and one shown for users/groups earlier are unmistakable. This is what we meant when we were talking about isomorphism in the section User/Group and Permission/Role Isomorphism.

Notice that we have shown the inherited (i.e. indirect) permissions in a role with a italic red font in the above diagram. Directly defined permissions are shown with normal, black font.

3.3.1. Group Roles/Permissions Enrollment (GRPE)

Both specific permissions and roles (i.e. a packaging of permissions) can be either given to or taken from (revoked) from a group. We refer to this process as group roles/permission enrollment and it works in a recursive fashion.

Group role enrollment information of X1Sys is shown below:

Figure 5. Group Role/Permission Enrollment for X1Sys

You will notice that there is a strong similarity between group names and the roles usually assigned to them (e.g. Sales_Admins group vs Sales_Admin role). Though following such a convention is usually an useful and convenient thing to do — as practiced heavily in functionally less limited access control schemes — there is absolutely no requirements — as far as Hifax is concerned — to do so. In fact you see some examples of not following the said scheme in the diagram (e.g. IT_Admins group is assigned SalesAcct_Admin role)

3.3.2. User Roles/Permissions Enrollment (URPE)

Both specific permissions and roles (i.e. a packaging of permissions) can be either given to or taken from (revoked) from a user. We refer to this process as user roles/permission enrollment and it works in a recursive fashion.

Moreover, a user will also get an effective set of permissions from the groups it belongs to. An example of this is shown in the following figure:

Figure 6. User Role/Permission Enrollment for X1Sys

Notice how user mary3 has been given both the role SalesAcct_PowerUser and the permission DB_ADMIN_SALES. This is to address the fact that SalesAcct_PowerUser role has been specifically created not to have that permission — as you can see in X1 RPG Graph with the explicit revoke operation — but we nevertheless want to give the user the said permission.

3.4. Declarative Access Control Enforcement

Any part of a system that needs to be access controlled can do so by simply expressing the essential contextual information — the resource to be guarded in question and the operations that are going to be carried out on them — to Hifax runtime. This contextual information is very specific to the part of the said system and does not require knowing anything else — especially the access control related operational aspects of the system (i.e. its users and groups, permissions, roles, etc.) — to function properly. This achieves a very good balance in terms of externalizing such operational aspects of the system as much as possible so that they can be administered externally in a policy-driven fashion.

To illustrate how declarative access control enforcement works, let’s consider a specific part of X1Sys that we designate as needing access control: the API function that does the accounting period end. We have earlier identified it as a resource with the name API.Accounting.EndPeriod in X1Sys resource specification. Imagine we have a function called AcctEndPeriod() in X1Sys codebase that provides the implementation of this API. It will have an outline like the following:


  1
2
3
4

   def AcctEndPeriod (*args, **kwargs):                    (1)
    CheckAccess("API.Accounting.EndPeriod", OP_EXECUTE)  (2)

    DoAcctPeriodEnd()                                    (3)

API function (i.e. the resource) to be access controlled
Declarative access control enforcement, stating only the resource and operation to be carried out
Rest of the function implementation

Note how clean is the access control enforcement code above. AcctEndPeriod implementation just uses a major function provided by Hifax runtime — CheckAccess() — and tells that function about what needs to be guarded (i.e. combination of resource and operation). Everything else in the complex machinery of access control is handled by Hifax runtime in a data-driven and policy-controlled manner within an operational environment externally managed by the relevant operational staff.

4. Algorithms

In this section, we will provide the details of the algorithms such as GEUS, REPS, etc., which produce flattened-out (i.e. effective) sets of access control objects (e.g. users, permissions, etc.,). The use of these effective sets not only allows checking permissions correctly by means of ensuring all the contributing elements in a hierarchy are incorporated in the evaluation process but also enables this process to happen as efficiently as possible thanks to using methods such as caching.

In the following discussions, we will frequently be adding/removing items to/from sets. A set does not have duplicates and trying to add an element that already exists in a set is a no-op. Similarly, trying to remove an element that does not exist from a set is also a no-op. We will not repeat these points to keep the descriptions of our algorithms concise.

A important feature of the hierarchy we are dealing with in the access control domain is that we want lower levels to have overriding effects over upper levels (i.e. grand parents). This way, user and role hierarchies can be set up with generic parameters at the top, safe with the knowledge that they can be overridden in specific settings. To that end, you will see the frequent use of a set called SeenNodes in the implementation of our algorithms. The algorithms we provide are recursively defined and they start at the most specific node (level) in a particular graph (e.g. GUG or RPG) and works their way up to the top of the graph. If we have already "seen" a particular node in the graph (e.g. a role being revoked), that information should be reflected in the result of the algorithm even if that node is encountered again in higher levels (e.g. said role was granted earlier). We achieve this by keeping track of each unique node we process in SeenNodes set and ignore a given node if it is already in that set (i.e. we have already processed it at a lower level of the graph).

A rather peculiar feature of the graphs we are dealing with is that we support both adding nodes to and removing nodes from our graphs. Moreover, we support the feature of overriding inheritance i.e. the lower levels of the graphs having the ability to determine the effective member set by virtue of either adding a new node or specifying the removal of a node (effectively banning it).

4.1. Group Effective User Set (GEUS) Computation

Problem: Given a group G, find all the users that are children and grandchildren (recursively) of that group.

Solution:

Initialize the sets GEUS and SeenNodes to be empty.
For each direct child C of G do the following:
- if C is in SeenNodes continue with the next child.
- Otherwise add it to SeenNodes
  - If C is being added, add it to GEUS
For each parent node P of G do the following:
- If P is RootGroup stop
- Otherwise recursively go to Step#2 with P replacing G

4.2. Role Effective Permission Set (REPS) Computation

Problem: Given a role R, find all the permissions included in that role directly or indirectly (i.e. in a recursive fashion)

Solution:

Initialize the sets REPS and SeenNodes to be empty.
For each direct permission M of R do the following:
- if M is in SeenNodes continue with the next permission.
- Otherwise add it to SeenNodes
  - If M is being granted (i.e. added), add it to REPS
For each parent node P of R do the following:
- If P is RootRole stop
- Otherwise recursively go to Step#2 with P replacing R

4.3. Group Effective Permission Set (GEPS) Computation

Problem: Given a group G, find all the permissions given to it directly or indirectly (i.e. in a recursive fashion)

Solution:

Initialize the sets GEPS and SeenNodes to be empty.
For each direct permission M of G do the following:
- if M is in SeenNodes continue with the next permission.
- Otherwise add it to SeenNodes
  - If M is being granted (i.e. added), add it to GEPS
For each role R directly assigned to G do the following:
- Let be S be the REPS for R
- For each permission M in S do the following:
  - if M is in SeenNodes continue with the next permission.
  - Otherwise add it to SeenNodes
    
    If M is being granted (i.e. added), add it to GEPS
For each parent node P of G do the following:
- If P is RootGroup stop
- Otherwise recursively go to Step#2 with P replacing G

4.4. User Effective Permission Set (UEPS) Computation

Problem: Given an user U, find all the permissions given to it directly or indirectly (i.e. in a recursive fashion, considering all group memberships)

Solution:

Initialize the sets UEPS and SeenNodes to be empty.
For each direct permission M of U do the following:
- if M is in SeenNodes continue with the next permission.
- Otherwise add it to SeenNodes
  - If M is being granted (i.e. added), add it to UEPS
For each role R directly assigned to U do the following:
- Let be S be the REPS for R
- For each permission M in S do the following:
  - if M is in SeenNodes continue with the next permission.
  - Otherwise add it to SeenNodes
    
    If M is being granted (i.e. added), add it to UEPS
For each parent group G of U do the following:
- Let be T be the GEPS for G
- For each permission M in T do the following:
  - if M is in SeenNodes continue with the next permission.
  - Otherwise add it to SeenNodes
    
    If M is being granted (i.e. added), add it to UEPS

4.5. Permission Checking

Permission checking involves checking whether a given user has access to a particular resource to perform a specific operation. The user in question is usually the current user — known as the principal — and any permission checking is done against the permission set — known as principal effective permission set (PEPS) — he/she/it is entitled to. PEPS is simply the UEPS computed for the current user (i.e. logged on user).

Permission checking is initiated and driven by the code — i.e. parts of the client system that implement Hifax — that needs to enforce access control and has the following general outline:

Context Setup: Client code has to create a PermissionContext object which will have all the contextual parameters needed in permission checking:
- Principal: Details about the current (logged in) user who tries to access the resource. This will be known by Hifax — usually using the underlying operating system APIs to determine the logged on user — so it does not need to be explicitly set by the client code.
- Resource: Details about the resource to be accessed. It has to be explicitly set by the client code as Hifax has no way to know which resource being accessed by the principal. It will include all static and dynamic attributes of the resource in question. Dynamic attributes such as persisted record fields have to be loaded by a system process before any access control could be performed with the principal’s credentials.
- Operations: The types of operations — i.e. a combination of the C/R/U/D/E operations — to be carried out. Like the resource, this has to be explicitly set by the client code.
PEPS Creation: Hifax will compute the effective set of all permissions the principal is entitled to (i.e. PEPS). PEPS is relatively expensive to create and is unlikely to change during a typical user session. Therefore it is ideal for pre-computing and caching as a part of login process.
Permission Check: The client code will pass the PermissionContext object pc to the Hifax function CheckAccess(context). Hifax will go through each permission in the PEPS and try to see if it satisfies the access operation requested on the resource as specified in pc. This might require evaluating the control script — if the permission has such a script — of each permission in PEPS. If this process succeeds, CheckAccess(…) will simply end silently, resulting in the client code that invoked it as its first statement to proceed with the rest of its control flow as usual. Otherwise, it will throw an exception with the details of permission that failed to be satisfied and causes the normal control flow of the client code to be diverted — possibly to the client error reporting code that then can use the error message generated by Hifax for logging and user notification purposes — and ultimately causes the client code to fail.

5. Conclusion

We have presented an access control scheme that is structured around clearly defined abstractions and whose design is guided by well-founded principles. Architecture of Hifax allows great flexibility in terms of user and permission management thanks to the use of hierarchical inheritance schemes utilizing the power of directed acyclic graphs. We have provided fundamental algorithms that utilize those graphs to provide very powerful and flexible access control functionality.

We have also presented a simple deployment use case of how Hifax could be implemented in the context of a basic business application. As this use case shows — to some extent — , Hifax has the power to address the most complicated access control needs in an operationally efficient and clean way, while also making it possible to be deployed to less demanding environments in a cost effective way.

Appendix A: Acronyms

DAG: Directed Acyclic Graph
EPS: Effective Permission Set
GEPS: Group Effective Permission Set
GEUS: Group Effective User Set
GRPE: Group Role Permission Enrollment
GUG: Group User Graph
REPS: Role Effective Permission Set
RPG: Role Permission Graph
RPE: Role Permission Enrollment
PEPS: Principal Effective Permission Set
UEPS: User Effective Permission Set
URPE: User Role Permission Enrollment
PRO: Principal/Resource/Operation
CRUDE: Create/Read/Update/Delete/Execute

Glossary of Terms

CRUDE: Create/Read/Update/Delete/Execute operations
DAG: Directed Acyclic Graph
Group: A set of users. Groups are hierarchically structured and have parent and child groups.
GUG: Group User Graph. A DAG formed from users and groups.
Operation: Basic actions that can be carried out on a resource by a user. It refers to C/R/U/D/E operations.
Permission: A combination of a resource identifier (could refer to more than one resource) together with operations to be carried out on said resource(s)
Principal: Currently logged on user
Resource: Any entity in a computer system which needs to be access controlled e.g. GUI objects, database tables, API functions, etc.
Role: A collection of permissions. Roles are hierarchically organized (i.e. they have parent and child roles).
RPG: Role Permission Graph. A DAG formed from permissions and roles.
User: Any actor in a computer system including both human and non-human actors

Amazon ReMARS 2019 Conference: Personal Impressions

1. Executive Summary

I have had the fortune to attend to the Amazon ReMARS 2019 conference focused on cutting edge AI/ML, Robotics, IoT and Automation technologies. I have tried to attend as many sessions as I could from Tuesday through Thursday. You will find a summary of my observations of those sessions in the following sections.

There were several keynote speeches some of which were quite entertaining. The main message was that Machine Learning (ML) technology is now mainstream and very accessible to everyday developers thanks to the availability of open source libraries that package cutting edge algorithms and products such as Amazon SageMaker that makes it very easy to consume such libraries to build applications that can be easily deployed to the production hardware on Amazon cloud.

There were lots of emphasis on the automation and robotics and how there have been significant developments in the robot vision, dexterity (handling) and autonomy/movement. Such developments are obviously happened on the back of recent developments seen in the ML technology.

Although there is still a lot of hype around some of those technologies, I have seen that with appropriate focus and sound understanding of the real underlying technologies, truly innovative solutions could be provided. This is what the team at the Australian energy company Woodside did: they married the good old system and software engineering with emerging technologies such as IoT, ML, computer vision and the cloud computing to build a completely automated, intelligent remote natural gas site that is very easy to operate.

2. Tuesday Workshop Sessions

2.1. W07: Voice Control for Any Thing

This session was about using an Arduino IoT kit to build a simple IoT device which consisted of a set of flashing LED lights arranged on a breadboard and connected and controlled through a Arduino-based micro computer running an embedded Linux operating system. The microcomputer had all the basic interfaces you would expect from a modern computer: USB ports, HDMI port, Ethernet port to connect to internet and usual PC monitor and keyboard.

As interesting as it was to have a micro computer of the size of your palm running a complete Linux system with live connection to the Internet, the real excitement was to control the LED IoT device through the internet connection of the microcontroller from the AWS cloud through a set of AWS services. Such services could be integrated to the Amazon Alexa, hence augmenting Alexa’s capabilities with custom hardware in the form of an IoT device.

This was a very hands-on session and all that breadboard, connections and devices reminded me of my electrical engineering days!

2.2. W05: Finding Martians with AWS RoboMaker and the JPL Open Source Rover

This session was about using the open source robot integration framework RoboMaker to build a simple rover application on AWS and to simulate and control the rover through the tools that are available through AWS RoboMaker product.

We learned the basic concepts of RoboMaker framework which is a scalable, node-based processing architecture that allows robot developers to build sophisticated robotics systems from a set of building-block components (e.g. a camera node to provide vision, a radar node to handle radio signals, etc.) and connect them through a message processing architecture (similar to how Windows operating systems works with handling window messages).

AWS RoboMaker makes creating, building, deploying, running (for simulation or processing) and scaling robotic applications as easy as possible. I likened the whole process to wizards you see in development IDEs like Visual Studio. You specify what kind of stack you want and AWS RoboMaker generates the whole software project on the cloud for you and it deals with all the hardware infrastructure you will need to run the code on the fly (IP network, compute and storage nodes, service entry points, etc.). It was almost too easy! As a developer you just focus on the core logic of your robot application and let AWS’s framework to handle the rest. I quite liked the concept and seems like a very useful product to quickly prototype robot-based applications.

3. Wednesday Sessions

3.1. M10 - Infor Coleman: Streamlining Enterprise ML Complexities

This session was about how data analytics and Machine Learning technology being used by one of the major ERP vendors in mid-market segment, Infor Coleman, to enhance the functionality of the traditional ERP applications: inventory management, accounting, production planning, sales and distribution, etc. The benefit of the use of visualization backed data analytics is obvious in such applications. What was interesting was their use of ML techniques to augment/enhance their MRP (Material Resource Planning) algorithms to make them more dynamic and flexible with respect to the changing market and operating conditions thanks to making them data-driven.

One of the major challenges mentioned by the speaker when it came to the use of advanced ML technology like deep learning was the model explainability. In tightly controlled environments such as where usual ERP software is deployed, you cannot just act based on the signals from a black-box. Decisions have to be possible to be traced back to the logic/drivers that generated them. This is the case with basic ML models like decision trees, random forests etc. but not the case with the ones based on neural nets even though the latter seem to generate better results overall. So there is still more work to be done in that space for production deployment of such models.

3.2. M20 - From Business Intelligence to Artificial Intelligence

This session was about the journey of a large chemical company, INVISTA, in building and operating a data analytics platform. Like any large organization, they were familiar with data analytics through data warehousing and business intelligence applications. They wanted to move from that to a more integrated and automated architecture where more advanced data analytics could be delivered in a more-or-less predictable way.

One of the most important lessons they learned was to form mixed-skill teams (e.g. a data scientist, one or two domain experts, few programmers, etc. all part of the same team) instead of forming teams along skillset lines (e.g. data scientists, programmers, domain experts, etc.). This allowed them to do agile style quick iteration cycles and develop their platform quickly and organically without losing the trust and confidence of their users.

3.3. R11 - The Open-source Bionic Leg: Constructing and Controlling a Prosthesis Driven by Artificial Intelligence

This session demonstrated a real-life bionic leg which was built by an academic team initially through basic finite state-based model of a leg. When you think about it, a leg could be in certain number of states: standing, walking, climbing up or down, etc. These states are important because they determine the sort of forces and torque the leg hardware has to exert to allow safe and stable operations. In fact, we were told that the correct state determination and smooth state transition was the most challenging part of building a robotic leg for humans.

The team developed a simple Finite State Machine (FSM) model of the leg first. Later on, they used the data coming from the sensors to train Convolutional Neural Network (CNN) based models to fine tune this FSM model to make state transitions smoother. The interesting thing was they did not give up their original FSM model because that model is simple enough to understand and to visualize the operation of the leg to get a good intuition behind the more complex CNN models.

3.4. A29 - Sooner Than You Think: Neural Interfaces are Finally Here

I was not quite sure about what I would learn in this session as I always found the idea of tapping into the brain signals as a hopelessly futile sci-fi fantasy. I was pleased to find out the presenters of this session were of the same opinion and they approached the problem of finding out what is going on in the brain from a completely different and novel angle. Instead of trying to understand and reverse engineer something as complex as brain by looking at brain electromagnetic activity — as many other researchers seem to be doing — they accepted the brain as a black-box system with very well-known and understood IO ports: nerves that connect brain to the muscles through the spinal cord. Such nerves enable motor actions/reactions in the muscles and it is much more easier to both measure and pinpoint/separate specific signals coming from the brain.

In the demo, they have shown how a user can play a simple computer game by just moving his fingers around. There was no computer vision etc. involved. The user would wear a device which measures the electromagnetic signals detected on the skin resulting from muscle motor actions and those signals could be mapped to the certain parts of the hand (e.g. middle finger). The device will send that signal to the receptor (computer in this case) through a radio signal and that way it could be actioned upon.

I found this technology quite exciting and think that it could be even used to detect the speech signals (if you place some sensors on the tongue and throat to detect muscle movements associated with speech synthesis process)

3.5. R08 - Creating the Intelligent Asset: Fusing IoT, Robotics, and Artificial Intelligence

This was one of the most interesting and enjoyable sessions I attended to at the conference.

The presenter was from Australian energy company Woodside and he demonstrated a very concrete use of technologies that were subject of the conference in a specific case of remote gas plant management operations.

Woodside’s problem was to manage a remote natural gas plant remotely with minimal(i.e. no on-site) human involvement in an effective and cost efficient way. They deployed various IoT devices around the plant --some of which they had to design and develop themselves for cost related reasons — to take measurements and to send/receive signals so that they can be remotely controlled. This far it is standard SCADA we are already familiar with. What they did innovatively was to create a 3D model of the entire plant and build an software control application that allowed operators to walk through this 3D model (like in Google earth/street view) and be able to see and operate each and every part of the plant remotely!

Each part of the CAD model they built is reactive to the user interaction through the software (e.g. click to see most recent measurements etc.). Moreover, they have integrated a rich data analytics platform on the parts of this model where an operator not only can see the current measurements but also the history of them together with trends in the measurements, associated graphs and all related data in the same application view. It almost made the job of a plant operator as that of someone playing the Sim City. It seemed like an amazing application.

I asked few questions how they developed the whole system and it turns out that they used a cross functional team of 10-12 people (developers, data scientists, domain experts, etc.) over a period of 2 years.

4. Thursday Sessions

4.1. M39 - Scaling Photo-realistic 3D Content Creation with Machine Learning

This was another very interesting session. The presenter is the development manager of Amazon imaging technology who are tasked with generating 3D CAD images of the products you see on the Amazon.com. 3D models of the product products have lots of advantages over the photos: they can be looked at from different angles/perspectives, they can be “teleported” into virtual settings to let customer know how they will look in different contexts (e.g. sofa in a living room).

The labor intensive way of 3D model generation is to have 3D artists and let them create the models. This is not only very expensive but — more crucially — does not scale in Amazon’s product range scale. They simply do not have that many 3D artists to create the models for every one of the millions of the products they sell. The solution they have in mind is to find a way to automatically generate the 3D models from product descriptions.

I have to say the most of the session was about the problem definition rather than the actual solution. As far the solution is concerned, the presenter stated that he believed the solution lies along the spectrum of machine vision and machine learning models. He did not get into the details at all, perhaps it is still unsolved problem and/or it is Amazon’s R&D secret.

I asked him about whether structural, ontology-driven product description could be used as well and he gave affirmative answer as they are looking into that approach as well.

So, in the absence of Amazon’s actual solution, I took the liberty to come up with my arm chair engineering design for the benefit of the reader:

Have a structural description of products and parts. For example, have a concept of chair which consists of legs and a base and optionally a back stand. This part-whole relationship is important to compose complex CAD models from constituent parts.
Use machine vision algorithms to recognize objects from their photos/pictures and map them to the items in the product inventory. Use the ontological whole/part description of the product to further recognize individual parts of the product.
Build a repository of CAD models for some of the parts of products we come across. This will be a labor intensive exercise.
Use Machine Learning models to map the images of the part of a new product to a CAD model. This ML model will be trained using the data in the repository described in the third step above. There are some open research problems of generating highly structured mesh data — that is characteristics of 3D CAD models — as an output of a ML model.
Build a 3D CAD synthesis model that can build a complicated model from simpler parts just like you build a structure from LEGO parts. This should be straightforward. This process will use the structural/ontological description of the product mentioned in the first step above.

4.2. M30-L - Investing in Technology Breakthroughs

This was a fun session by a big shot venture capitalist who was talking fast enough that if you had momentarily lost your interest you would miss several complete sentences. Fortunately what he was talking about was very attention grabbing laced with jokes and good humor I believe I did not miss much.

One of main take away for me was how advanced, innovative technologies are linked to each other. He was stating the widely known fact that no one can for sure will know which technology will be successful and which startup will prevail over a 10 year investment horizon of a typical VC investment cycle. However, by virtue of investing in some of those companies and taking interest in their products and technologies you can find hidden opportunities and links allowing jumping to the other areas/products/technologies you never even thought about. This is bit like some of the most successful startups like Slack starting as companies with completely different goals and finding their successful products (Messaging app for Slack) almost by accident. His point was by investing in cutting edge companies and taking an interest in their technologies and connecting "the dots", one can significantly increase the chances of coming across these "accidents".

4.3. A03-L - Follow the Money: What We Can Learn from Venture Capital Investment in Automation

This was a session presented by a veteran of American Robotics Association who has been in the robotics industry since 70s and have witnessed many hype and disillusionment cycles over the years.

One of his interesting messages was that the increase in robotic automation in US is positively correlated with the employment (in other words, negatively correlated with unemployment). He was stating this empirical fact to claim that some of the worries around automation taking away jobs were overblown. Personally, I am not sure how that correlation will hold over the next 40 years.

Another of his main talking points was the observation that even though high profile robotics products/applications such as Boston Dynamics robots take the headlines as if these sort of developments suddenly took place, the progress in the industrial robotics has been steady since 70s with gradual improvements. There is certainly some acceleration thanks to the use of big data, advanced ML models and cheap and powerful computing power to endow robots with vision, dexterity and autonomy.

4.4. A22 - A Hype-free and Cutting-edge Discussion on Autonomous Driving

This was a historical overview of the development of self-driving car technology from one of his veterans who got involved in the field from mid 90s through the academia. One of most striking observations the presenter made was they had the self-driving car demo — the sort who gets people excited nowadays — back in late 90s, perhaps less presentable with big sensors sticking out but functionally almost equivalent.

Improvements in the sensor technology and the increase in the compute power through GPUs have been the two major technical enablers of the self-driving car technology. The main challenge of safety remains the same, especially in dense urban areas where self-driving cars depend on the availability and accuracy (i.e. maintenance) of high definition maps to allow vehicles to have precise control at the accuracy of 1/1.5 meters.

One of the most interesting insights from this talk was that it is most likely to find self-driving vehicle technology to be deployed first in hybrid settings and in small physical scales rather than in fully automated large scale self-driving scenarios:

An example for the hybrid setting would be self-driving trucks being autonomous on the highways — where the driving is more predictable and hence safe and GPS and camera/LIDAR guidance is sufficient — but being driven by humans in and out of city centers to truck ports outside city boundaries.
An example for the small physical scale vehicles would be small carrier/console like vehicles (e.g. grocery shop carrier) which would pose no harm to humans because of their the low operating speed and weight.

4.5. R14-L - Every Tool’s A Hammer: Life is What You Make It

I was expecting this to be a maker’s how to session but it turned out to be more of social/philosophical/leadership session where the speaker — a famous TV presenter active in makers show scene — shared his philosophy about how to design things, get them built and share them with the community.

It was full of leadership and life lessons and I did not regret that he did not talk about his latest gizmos or maker projects. One of the interesting things he talked about was how he learned to delegate tasks and let others have a go at the some of the work that needs to be a done in a big project. Another person is very unlikely to do the things exactly how you would do but if you have the right perspective about this, you can view this as an opportunity to perhaps learn something new or look at a problem from a completely different angle, both of which could be very rewarding for the receptive minds.