Traditionally, most complex intelligence architectures are extremely non-convex, which could not be well performed by convex optimization. However, this paper decomposes complex structures into three types of nodes: operators, algorithms and functions. Further, iteratively propagating from node to node along edge, we prove that "regarding the neural graph without triangles, it is nearly convex in each variable, when the other variables are fixed." In fact, the non-convex properties stem from triangles and functions, which could be transformed to be convex with our proposed \textit{\textbf{convexification inequality}}. In conclusion, we generally depict the landscape for the objective of neural graph and propose the methodology to convexify neural graph.
View on arXiv